forked from animatedread/Warrior_EA
f64e0f8b67
User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2093 lines
130 KiB
MQL5
2093 lines
130 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Warrior_EA |
|
|
//| AnimateDread |
|
|
//| |
|
|
//| Filter-based indicator auto-tuner (mutual information scoring). |
|
|
//| |
|
|
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
|
|
//| This holds CExpertSignalAIBase method BODIES only. The class |
|
|
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
|
|
//| #includes this file at the bottom, after the declaration. Do not |
|
|
//| include it anywhere else and do not compile it on its own. |
|
|
//| |
|
|
//| Split out purely to make the 8216-line original navigable; the |
|
|
//| code inside was moved verbatim, not rewritten. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_AIBASE_AUTOTUNE_MQH
|
|
#define WARRIOR_AIBASE_AUTOTUNE_MQH
|
|
//--- ONCE-PER-CHART gate for the MI diagnostic suite on a multi-member ensemble. The suite (headline
|
|
//--- MI, positive control, alignment scan, lag profile, barrier-geometry scan + winner test) is a
|
|
//--- pure MEASUREMENT of this chart's features and labels - identical inputs for every ensemble
|
|
//--- member, so running it four times quadruples the slowest part of "getting ready" (the permuted
|
|
//--- nulls are ~200 draws per line) to report the same numbers four times. The first member to reach
|
|
//--- it runs it; the rest log one line and skip. CAVEAT, documented not hidden: the geometry scan can
|
|
//--- in principle ADOPT its winner under the family-wise gate (it never has on any instrument -
|
|
//--- "Change nothing" every run); if it ever clears, this gate makes that adoption donor-only and
|
|
//--- must be revisited. Solo charts are untouched.
|
|
bool g_ensembleChartMiReportDone = false;
|
|
//--- ONCE-PER-CHART share of the indicator auto-tune SWEEP on a multi-member ensemble, same doctrine as
|
|
//--- the MI gate above: the sweep scores candidate indicator settings by feature/label MI, and every
|
|
//--- ensemble member holds identical indicators, identical cached features and identical labels, so all
|
|
//--- N sweeps are the same deterministic calculation (verified 2026-08-16 on SP500 H4: four members,
|
|
//--- byte-identical scores, spans and selection p). Worse, the sweep ends in the full MI diagnostic
|
|
//--- suite (ReportFeatureLabelInformation at its tail), which the MI gate above never intercepts on the
|
|
//--- sweep path - so each duplicate sweep also duplicated the ~200-draw permutation nulls, the slowest
|
|
//--- single block of "getting ready". The first member runs the sweep and publishes its outcome here;
|
|
//--- the rest apply the outcome (install the winner, or keep the configured settings the sweep restored)
|
|
//--- and skip both the sweep and the report. Same caveat as the MI gate: any winner ADOPTION is made by
|
|
//--- the donor and applied to every member via the flattened settings below, which is the consistent
|
|
//--- choice - members training on divergent feature vectors would not be an ensemble. Solo charts are
|
|
//--- untouched.
|
|
bool g_ensembleChartTuneDone = false;
|
|
bool g_ensembleChartTuneInstalled = false; // did the donor's sweep clear the family-wise gate and install?
|
|
double g_ensembleChartTuneSettings[]; // CADIndicatorTuner::Flatten() of the donor's final settings
|
|
//--- THE SAME DOCTRINE, APPLIED TO THE BARRIER GEOMETRY - and it was missing, which broke the ensemble.
|
|
//--- The geometry scan lives at the end of the MI chain, and that chain runs ONCE PER CHART
|
|
//--- (g_ensembleChartMiReportDone): whichever member reaches it first measures, and the other three skip
|
|
//--- the whole thing. That was harmless while the scan only PRINTED. Once it became authoritative
|
|
//--- (62a719f) the donor started relabelling itself and nobody else, so a single chart's members trained
|
|
//--- on DIFFERENT TARGETS - and the orchestrator averages their votes and the joint gate certifies the
|
|
//--- combined vote as if they answered one question.
|
|
//--- MEASURED 2026-08-17 19:06 on USDJPY: LSTM adopted 2:10 (horizon 256, break-even 16.7%, mean label
|
|
//--- lifespan 81.4 bars) while PAI/CONV/HYB stayed at 1.61:3.21 (horizon 64, break-even 33.3%, lifespan
|
|
//--- 19.2). A "Buy" from LSTM and a "Buy" from PAI were not the same claim, and averaging them is not a
|
|
//--- vote. g_DerivedSlAtrMult - which places the LIVE order - is one global, so the stop actually sent
|
|
//--- was whichever member wrote last.
|
|
bool g_ensembleChartGeomAdopted = false; // did the donor's scan adopt a pairing the siblings must take?
|
|
double g_ensembleChartGeomSl = 0.0; // the DERIVED pair (the one authority - see BarrierMultiples)
|
|
double g_ensembleChartGeomTp = 0.0;
|
|
int g_ensembleChartGeomSlMode = 0; // legacy mode ints, kept in step for the fallback/fingerprint
|
|
int g_ensembleChartGeomTpMode = 0;
|
|
#ifdef WARRIOR_EXPORT_FEATURES
|
|
//+------------------------------------------------------------------+
|
|
//| RESEARCH BUILD ONLY - see the declaration comment. |
|
|
//| |
|
|
//| Every research question so far has cost a compile, a deploy, an |
|
|
//| attach and a log read - minutes each, and the answer arrives one |
|
|
//| hypothesis at a time. That loop, not the modelling, is what has |
|
|
//| made this slow. Exporting the feature matrix ONCE moves the whole |
|
|
//| question offline, where a hypothesis costs seconds and real tools |
|
|
//| (joint mutual information, gradient boosting, proper walk-forward |
|
|
//| cross-validation) are available - none of which can be written in |
|
|
//| MQL5 in reasonable time. |
|
|
//| |
|
|
//| Exports the RAW BARS next to the features deliberately: with OHLC |
|
|
//| and ATR offline, every barrier geometry, every horizon and every |
|
|
//| in-trade target can be recomputed without touching MetaTrader |
|
|
//| again. The bar TIME goes out too, which makes session, hour and |
|
|
//| day-of-week features derivable for free - and those are the only |
|
|
//| inputs in play that are NOT a transform of the same OHLCV series. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ExportFeatureMatrix(void)
|
|
{
|
|
if(MQLInfoInteger(MQL_OPTIMIZATION))
|
|
return;
|
|
int barsNow = Bars(m_symbol.Name(), PERIOD_CURRENT);
|
|
//--- Clamp BEFORE the emptiness test, so a fully-capped symbol reports the depth it can actually
|
|
//--- export rather than the price-series depth it cannot. Exporting the uncapped count is how this
|
|
//--- path used to fail: ResizeBuffers/RefreshData "succeed" (Refresh() is void), then every MA read
|
|
//--- comes back EMPTY_VALUE and the CSV is written with a dead feature block - a research artefact
|
|
//--- that looks complete and is silently wrong, which is the whole failure class this guards.
|
|
barsNow = ServableBars(barsNow, "feature export");
|
|
if(barsNow <= m_historyBars + 2)
|
|
{
|
|
Print(ID + ": EXPORT - only " + IntegerToString(barsNow) + " bars available, nothing to write");
|
|
return;
|
|
}
|
|
if(!ResizeBuffers(barsNow) || !RefreshData())
|
|
{
|
|
Print(ID + ": EXPORT - buffers not ready (" + IntegerToString(barsNow) + " bars), aborting");
|
|
return;
|
|
}
|
|
EnsureBarCachesCapacity(barsNow);
|
|
EnsureBarrierHorizon(barsNow);
|
|
string dir = eaName + "\\Research\\";
|
|
string fn = dir + m_symbol.Name() + "_" + IntegerToString(_Period) + "_features.csv";
|
|
int h = FileOpen(fn, FILE_COMMON | FILE_WRITE | FILE_CSV | FILE_ANSI, ',');
|
|
if(h == INVALID_HANDLE)
|
|
{
|
|
Print(ID + ": EXPORT - cannot open " + fn + ", error " + IntegerToString(GetLastError()));
|
|
return;
|
|
}
|
|
string header = "idx,time,open,high,low,close,atr";
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
header += ",f" + IntegerToString(f);
|
|
FileWrite(h, header);
|
|
//--- Oldest first. The loop walks DOWN the series index, which is forward in time (higher index =
|
|
//--- older), so the file reads chronologically and Python can treat row order as time order.
|
|
int written = 0, skipped = 0;
|
|
uint t0 = GetTickCount();
|
|
for(int i = barsNow - 1; i >= 0; i--)
|
|
{
|
|
TempData.Clear();
|
|
if(!BufferTempData(i) || TempData.Total() < m_neuronsCount)
|
|
{
|
|
skipped++;
|
|
continue;
|
|
}
|
|
double atr = m_ATR.Main(i);
|
|
string row = IntegerToString(i) + "," + IntegerToString((long)m_Time.GetData(i)) + "," +
|
|
DoubleToString(m_Open.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_High.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_Low.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_Close.GetData(i), _Digits) + "," +
|
|
DoubleToString(MathIsValidNumber(atr) ? atr : 0.0, _Digits);
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
row += "," + DoubleToString(TempData.At(f), 8);
|
|
FileWrite(h, row);
|
|
written++;
|
|
}
|
|
TempData.Clear();
|
|
FileClose(h);
|
|
Print(ID + StringFormat(": EXPORT COMPLETE - %d rows x %d features -> Common\\Files\\%s "
|
|
"(%d bars skipped for missing features, %.1fs, horizon %d, spread %d points)",
|
|
written, m_neuronsCount, fn, skipped, (GetTickCount() - t0) / 1000.0,
|
|
m_barrierHorizonBars, (int)m_symbol.Spread()));
|
|
ExportRawRates();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| RESEARCH BUILD ONLY. Raw OHLCV for a GRID of symbols/timeframes, |
|
|
//| not just this chart's. |
|
|
//| |
|
|
//| The 26 engineered features can only be produced for the chart the |
|
|
//| EA is attached to - the indicator handles are bound to |
|
|
//| PERIOD_CURRENT. Raw rates are not: CopyRates serves any symbol |
|
|
//| and any timeframe from a single chart. So one attach yields the |
|
|
//| whole research grid, and every question that does not require the |
|
|
//| EXISTING feature set - a different horizon, a different barrier, |
|
|
//| session/time-of-day effects, features this EA does not have yet - |
|
|
//| can then be answered offline without MetaTrader in the loop at |
|
|
//| all. That is what turns a per-hypothesis cost of minutes into |
|
|
//| seconds, which has been the real bottleneck all along. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ExportRawRates(void)
|
|
{
|
|
string symbols[] = { "SP500", "USDJPY", "XAUUSD", "EURUSD", "GBPUSD", "US30", "NAS100", "BTCUSD" };
|
|
ENUM_TIMEFRAMES tfs[] = { PERIOD_M5, PERIOD_M15, PERIOD_H1, PERIOD_H4, PERIOD_D1 };
|
|
string dir = eaName + "\\Research\\";
|
|
int cells = 0, rowsTotal = 0;
|
|
for(int s = 0; s < ArraySize(symbols); s++)
|
|
{
|
|
//--- Skip silently rather than warn: the grid is deliberately broader than any one broker's symbol
|
|
//--- list, so an absent instrument is expected, not an error.
|
|
if(!SymbolSelect(symbols[s], true))
|
|
continue;
|
|
for(int p = 0; p < ArraySize(tfs); p++)
|
|
{
|
|
MqlRates r[];
|
|
ArraySetAsSeries(r, false); // oldest first, so file order is time order
|
|
int got = CopyRates(symbols[s], tfs[p], 0, 200000, r);
|
|
if(got <= 100)
|
|
continue;
|
|
string fn = dir + symbols[s] + "_" + IntegerToString((int)tfs[p]) + "_rates.csv";
|
|
int h = FileOpen(fn, FILE_COMMON | FILE_WRITE | FILE_CSV | FILE_ANSI, ',');
|
|
if(h == INVALID_HANDLE)
|
|
continue;
|
|
int dg = (int)SymbolInfoInteger(symbols[s], SYMBOL_DIGITS);
|
|
FileWrite(h, "time,open,high,low,close,tickvol,spread");
|
|
for(int i = 0; i < got; i++)
|
|
FileWrite(h, IntegerToString((long)r[i].time) + "," +
|
|
DoubleToString(r[i].open, dg) + "," + DoubleToString(r[i].high, dg) + "," +
|
|
DoubleToString(r[i].low, dg) + "," + DoubleToString(r[i].close, dg) + "," +
|
|
IntegerToString((long)r[i].tick_volume) + "," + IntegerToString(r[i].spread));
|
|
FileClose(h);
|
|
cells++;
|
|
rowsTotal += got;
|
|
Print(ID + StringFormat(": EXPORT rates - %s %s: %d bars", symbols[s],
|
|
EnumToString(tfs[p]), got));
|
|
}
|
|
}
|
|
Print(ID + StringFormat(": EXPORT RATES COMPLETE - %d cells, %d bars total, under Common\\Files\\%s",
|
|
cells, rowsTotal, dir));
|
|
}
|
|
#endif
|
|
//--- The genetic + successive-halving helpers that used to live here (GaRungEras, GaExtract, GaStore,
|
|
//--- GaMutate, GaRandomCandidate, GaBlockCrossover, GaSortAliveByScoreDesc, GaBreedNextGeneration) were
|
|
//--- deleted on 2026-08-01 together with the search they served. See TuneIndicatorsByFilter() below for
|
|
//--- the measured cost that retired them and what replaced it.
|
|
//+------------------------------------------------------------------+
|
|
//| MUTUAL INFORMATION between one cached feature column and the |
|
|
//| triple-barrier label, in nats, over a sample of in-sample bars. |
|
|
//| |
|
|
//| I(X;Y) = sum p(x,y) log( p(x,y) / (p(x) p(y)) ), with the feature |
|
|
//| discretised into MI_BINS EQUAL-FREQUENCY bins. Equal-frequency |
|
|
//| rather than equal-width because these features are ATR-normalised |
|
|
//| and heavy-tailed: fixed-width bins put nearly everything in one |
|
|
//| bucket and report ~0 information for a genuinely useful feature. |
|
|
//| |
|
|
//| Rank-based binning gives equal frequency for free - sort a copy of |
|
|
//| the column, then a value's bin is its rank scaled into MI_BINS. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::FeatureColumnMI(const double &vals[], const int &labels[], int n)
|
|
{
|
|
if(n < MI_MIN_SAMPLES)
|
|
return 0.0;
|
|
double sorted[];
|
|
ArrayResize(sorted, n);
|
|
ArrayCopy(sorted, vals, 0, 0, n);
|
|
ArraySort(sorted);
|
|
//--- A column that never varies carries no information; short-circuit so the log below is never
|
|
//--- reached with a degenerate single-bin histogram.
|
|
if(sorted[0] == sorted[n - 1])
|
|
return 0.0;
|
|
int joint[]; ArrayResize(joint, MI_BINS * 3); ArrayInitialize(joint, 0);
|
|
int px[]; ArrayResize(px, MI_BINS); ArrayInitialize(px, 0);
|
|
int py[]; ArrayResize(py, 3); ArrayInitialize(py, 0);
|
|
for(int i = 0; i < n; i++)
|
|
{
|
|
//--- rank via binary search on the sorted copy; ties land in the same bin, which is correct
|
|
int lo = 0, hi = n - 1, rank = 0;
|
|
while(lo <= hi)
|
|
{
|
|
int mid = (lo + hi) / 2;
|
|
if(sorted[mid] < vals[i])
|
|
{
|
|
rank = mid + 1;
|
|
lo = mid + 1;
|
|
}
|
|
else
|
|
hi = mid - 1;
|
|
}
|
|
int bx = (int)((double)rank * MI_BINS / n);
|
|
if(bx >= MI_BINS)
|
|
bx = MI_BINS - 1;
|
|
int by = labels[i];
|
|
if(by < 0 || by > 2)
|
|
continue;
|
|
joint[bx * 3 + by]++;
|
|
px[bx]++;
|
|
py[by]++;
|
|
}
|
|
double mi = 0.0;
|
|
for(int b = 0; b < MI_BINS; b++)
|
|
{
|
|
if(px[b] <= 0)
|
|
continue;
|
|
for(int c = 0; c < 3; c++)
|
|
{
|
|
int j = joint[b * 3 + c];
|
|
if(j <= 0 || py[c] <= 0)
|
|
continue;
|
|
double pxy = (double)j / n;
|
|
mi += pxy * MathLog(pxy / (((double)px[b] / n) * ((double)py[c] / n)));
|
|
}
|
|
}
|
|
return (mi > 0.0) ? mi : 0.0;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Scores the CURRENT indicator parameters by how much the resulting |
|
|
//| feature vector tells us about the label - the mean per-column |
|
|
//| mutual information over a stratified sample of in-sample bars. |
|
|
//| |
|
|
//| Deliberately scores EVERY column, not just the ones belonging to |
|
|
//| the parameter being swept. Columns the sweep did not touch |
|
|
//| contribute the SAME amount to every candidate, so they shift the |
|
|
//| mean by a constant and cannot change which candidate wins - while |
|
|
//| avoiding any need for this code to know the feature-vector layout, |
|
|
//| which is exactly the kind of coupling that rots. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::BuildMiSample(double &cols[], int &labels[], int labelBarOffset = 0,
|
|
int featureBarOffset = 0, int target = MI_TARGET_BARRIER)
|
|
{
|
|
//--- Continuous targets are collected raw here and discretised after the loop, because equal-frequency
|
|
//--- binning needs the whole sample's distribution before any one row can be assigned a bin.
|
|
double raw[];
|
|
bool continuousTarget = (target != MI_TARGET_BARRIER);
|
|
int bars = m_labelCacheBars;
|
|
if(bars <= 0 || m_neuronsCount <= 0)
|
|
return -1;
|
|
//--- Sample the IS region only. The OOS window must not influence which indicator settings ship, or
|
|
//--- the holdout has been used for selection and stops being a holdout at all.
|
|
int oosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0
|
|
* MathMax(bars - MathMax(m_historyBars, 0), 0));
|
|
int lo = MathMax(oosCutoff, MathMax(m_barrierHorizonBars, 1) + 1);
|
|
int hi = bars - MathMax(m_historyBars, 0) - 1;
|
|
//--- Keep the OFFSET label lookup inside the same bounds as the features, so a shifted scan measures a
|
|
//--- shift and not an edge effect. Widened symmetrically rather than clamping per bar, which would pile
|
|
//--- several sample rows onto the same clamped label and manufacture association out of nothing.
|
|
//--- THE PAD IS FIXED, NOT |labelBarOffset|. Two builds are only comparable row by row if they enumerate
|
|
//--- the same bars with the same stride, and both `lo` and `stride` below are derived from this range -
|
|
//--- so padding by the requested offset would move every row of the offset build. That is exactly what
|
|
//--- broke the positive control: it paired row k of an unshifted build with row k of a build starting
|
|
//--- `offset` bars later, whose label was then shifted a further `offset`, giving a pair 2*offset apart.
|
|
//--- The measured consequence was a control that reported the MI of labels 48 bars apart while claiming
|
|
//--- 24, failed its 5x gate, and voided every MI figure the EA printed.
|
|
int shiftPad = MiShiftPad();
|
|
if(MathAbs(labelBarOffset) > shiftPad || MathAbs(featureBarOffset) > shiftPad)
|
|
return -1; // caller asked for a shift the pad does not cover
|
|
lo += shiftPad;
|
|
hi -= shiftPad;
|
|
if(hi - lo < MI_MIN_SAMPLES)
|
|
return -1;
|
|
int stride = (int)MathMax(1, (hi - lo) / MI_SAMPLE_BARS);
|
|
//--- Published so the positive control can say how many BARS apart two sample rows are without
|
|
//--- recomputing this arithmetic at the call site, where it would silently drift out of agreement.
|
|
m_miStrideBars = stride;
|
|
int cap = (hi - lo) / stride + 1;
|
|
ArrayResize(cols, cap * m_neuronsCount);
|
|
ArrayResize(labels, cap);
|
|
if(continuousTarget)
|
|
ArrayResize(raw, cap);
|
|
int n = 0;
|
|
for(int i = lo; i < hi && n < cap; i += stride)
|
|
{
|
|
//--- Features come from bar i; the LABEL may be taken from a neighbouring bar (labelBarOffset != 0)
|
|
//--- so the caller can scan for a feature/label misalignment - see the alignment scan in
|
|
//--- ReportFeatureLabelInformation(). Both bars must carry a valid label for the row to count.
|
|
int li = i + labelBarOffset;
|
|
if(i >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[i])
|
|
continue;
|
|
//--- The geometry scan asks "what WOULD this label be under a different barrier?", which by
|
|
//--- definition is not in the cache. Compute it on the spot instead - the cache belongs to the
|
|
//--- configured geometry and a scan must never write to it.
|
|
if(!m_barrierScanLiveLabels && (li < 0 || li >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[li]))
|
|
continue;
|
|
if(m_barrierScanLiveLabels && (li < MathMax(m_barrierHorizonBars, 1) || li >= bars))
|
|
continue;
|
|
//--- BufferTempData(), NOT BufferTempDataCompute(). The Compute variant APPENDS the bar's features
|
|
//--- to TempData and never touches m_featureCache - only the caching wrapper writes that array. The
|
|
//--- first version of this function called Compute and then read m_featureCache, which
|
|
//--- ReInitADIndicators had just invalidated, so every column read back constant, FeatureColumnMI
|
|
//--- returned 0 for all of them, and all 17 candidates scored exactly 0.0000 nats. The tuner ran for
|
|
//--- 139 s per chart and always reported "no improvement" - a silent no-op that looked like a
|
|
//--- measurement. Read the values back out of TempData, which is where they actually land.
|
|
//--- FEATURE-side shift, distinct from labelBarOffset and not interchangeable with it. Shifting the
|
|
//--- LABEL changes which trade is being predicted, so at any non-zero offset the features sit INSIDE
|
|
//--- the labelled window and the score is lookahead - which is exactly what the alignment scan
|
|
//--- measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the
|
|
//--- FEATURES instead keeps the label pinned to the entry bar and asks the honest question: does the
|
|
//--- state k bars BEFORE the entry still carry information about that entry's outcome? Positive k is
|
|
//--- strictly older (higher series index), so every row stays causal.
|
|
TempData.Clear();
|
|
if(!BufferTempData(i + featureBarOffset) || TempData.Total() < m_neuronsCount)
|
|
continue;
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
cols[n * m_neuronsCount + f] = TempData.At(f);
|
|
if(continuousTarget)
|
|
{
|
|
//--- Excursions come from the cache only. The geometry scan's live-relabel path deliberately
|
|
//--- does not feed them: excursions do not depend on SL/TP at all (see the accumulators in
|
|
//--- TripleBarrierLabel), so re-deriving them per candidate geometry would compute the same
|
|
//--- number repeatedly and invite the impression that it varies with the barrier.
|
|
if(li >= ArraySize(m_excUpCache))
|
|
continue;
|
|
double up = m_excUpCache[li];
|
|
double dn = m_excDownCache[li];
|
|
if(!MathIsValidNumber(up) || !MathIsValidNumber(dn))
|
|
continue;
|
|
//--- A bar that TripleBarrierLabel() could not resolve (no valid ATR or close, typically the
|
|
//--- oldest bars) is still flagged as having a label, but its excursions were cleared to zero
|
|
//--- rather than measured. Price cannot genuinely travel zero in BOTH directions over a whole
|
|
//--- horizon, so this is an unambiguous "not measured" marker. Dropping those rows matters more
|
|
//--- than it looks: under EQUAL-FREQUENCY binning a block of identical zeros drags the lowest
|
|
//--- cut point onto zero, and a third of the sample then lands in one bin carrying no
|
|
//--- information - which would show up as a depressed score and read as "not predictable".
|
|
if(up <= 0.0 && dn <= 0.0)
|
|
continue;
|
|
if(target == MI_TARGET_EXC_UP)
|
|
raw[n] = up;
|
|
else
|
|
if(target == MI_TARGET_EXC_DOWN)
|
|
raw[n] = dn;
|
|
else
|
|
if(target == MI_TARGET_EXC_RANGE)
|
|
raw[n] = up + dn;
|
|
else
|
|
if(target == MI_TARGET_EXC_ASYM)
|
|
raw[n] = up - dn;
|
|
else
|
|
{
|
|
//--- Scale-free asymmetry. The denominator is > 0 here because rows with both
|
|
//--- excursions zero were dropped above, so no guard is needed beyond that.
|
|
raw[n] = (up - dn) / (up + dn); // MI_TARGET_EXC_ASYM_NORM
|
|
}
|
|
labels[n] = 0; // assigned below, once the distribution is known
|
|
}
|
|
else
|
|
if(m_barrierScanLiveLabels)
|
|
{
|
|
ENUM_SIGNAL v = TripleBarrierLabel(li);
|
|
if(v == Neutral && m_lastBarrierTimedOut)
|
|
m_barrierScanTimeouts++;
|
|
labels[n] = (v == Buy) ? 0 : ((v == Sell) ? 1 : 2);
|
|
}
|
|
else
|
|
labels[n] = m_labelCacheBuy[li] ? 0 : (m_labelCacheSell[li] ? 1 : 2);
|
|
n++;
|
|
}
|
|
TempData.Clear();
|
|
//--- EQUAL-FREQUENCY DISCRETISATION into the same 3 classes FeatureColumnMI's joint table expects, so
|
|
//--- every downstream piece - the block permutation, the null, the p-value, the lag profile - works on
|
|
//--- a continuous target with no change at all. Equal-frequency rather than equal-width because these
|
|
//--- distributions are fat-tailed (MFE especially): fixed-width bins would put almost every row in the
|
|
//--- first bin and measure nothing. It also fixes H(Y) at ln(3) = 1.099 nats for all four excursion
|
|
//--- targets, which makes their scores directly comparable to each other AND to the barrier label's
|
|
//--- ~1.02 - a comparison that would otherwise be confounded by class balance.
|
|
if(continuousTarget && n > 0)
|
|
{
|
|
double sorted[];
|
|
ArrayResize(sorted, n);
|
|
ArrayCopy(sorted, raw, 0, 0, n);
|
|
ArraySort(sorted);
|
|
double cut1 = sorted[n / 3];
|
|
double cut2 = sorted[(2 * n) / 3];
|
|
//--- A degenerate target (every value identical, e.g. a cache that never filled) would land every
|
|
//--- row in one class and score a flat zero. Say so rather than reporting the zero as a finding.
|
|
if(cut1 == cut2 && sorted[0] == sorted[n - 1])
|
|
{
|
|
Print(ID + ": MI excursion target " + IntegerToString(target) + " is CONSTANT across all "
|
|
+ IntegerToString(n) + " sampled bars - the excursion cache did not fill. Treating as "
|
|
"unusable rather than reporting its zero score as a measurement.");
|
|
return -1;
|
|
}
|
|
for(int q = 0; q < n; q++)
|
|
labels[q] = (raw[q] <= cut1) ? 0 : ((raw[q] <= cut2) ? 1 : 2);
|
|
}
|
|
return n;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Score an already-extracted sample. Split out from the extraction |
|
|
//| above so the permutation test can reuse ONE sample across every |
|
|
//| draw: feature extraction dominates the cost, and re-running it |
|
|
//| per shuffle is what would have made a few hundred permutations |
|
|
//| unaffordable. The shuffle is in place and destructive, which is |
|
|
//| harmless - composing permutations still yields a uniform |
|
|
//| permutation, so successive draws stay independent - but it does |
|
|
//| mean the OBSERVED (unshuffled) statistic must be taken first. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::ScoreMiSample(const double &cols[], int &labels[], int n, bool shuffleLabels)
|
|
{
|
|
if(n < MI_MIN_SAMPLES)
|
|
return -1.0;
|
|
//--- PERMUTATION BASELINE. Mutual information estimated from finite samples is biased UPWARD - with
|
|
//--- MI_BINS bins and 3 classes the bias is roughly (bins-1)(classes-1)/(2n) nats, which at these
|
|
//--- sample sizes is the same order as any real edge in this domain. So a raw MI figure is
|
|
//--- uninterpretable on its own: 0.004 nats could be a genuine weak signal or could be pure noise.
|
|
//--- Shuffling the labels destroys every real association while leaving the sample size, the binning
|
|
//--- and the class proportions untouched, so the score it produces IS this dataset's noise floor,
|
|
//--- measured rather than approximated. Reporting the two together turns "0.0042 nats" into either
|
|
//--- "0.0042 against a 0.0041 floor" (nothing) or "0.0042 against a 0.0009 floor" (something).
|
|
//--- BLOCK permutation, not a free one, and the difference is the whole validity of the test.
|
|
//--- Triple-barrier labels OVERLAP: two sample rows less than m_barrierHorizonBars apart share most of
|
|
//--- their outcome window, so their labels are strongly dependent. A free Fisher-Yates shuffle destroys
|
|
//--- that dependence as well as the feature/label association, which makes the null distribution far
|
|
//--- NARROWER than the truth and hands out significance that isn't there. The 2026-08-01 symbol sweep
|
|
//--- showed it in the raw: excess tracked the sampling STRIDE almost monotonically, and the three D1
|
|
//--- cells - where the stride had collapsed to 1-5 bars against a 128-bar horizon, i.e. ~99% window
|
|
//--- overlap - returned 5-9x the "signal" of every H1 cell at p=0.005. That was label autocorrelation
|
|
//--- leaking through an independence assumption, not an edge. It is Lopez de Prado ch. 4's non-IID
|
|
//--- problem arriving through the back door of the significance test.
|
|
//--- Permuting whole CONTIGUOUS BLOCKS at least one horizon long preserves the autocorrelation inside a
|
|
//--- block while destroying any feature/label association across blocks - so the null keeps the
|
|
//--- dependence structure and the p-value means what it says. It also degrades honestly: when overlap is
|
|
//--- severe there are few blocks, the null is correspondingly wide, and nothing reaches significance,
|
|
//--- which is the correct answer rather than a flattering one.
|
|
if(shuffleLabels)
|
|
{
|
|
int blockRows = (m_miStrideBars > 0)
|
|
? (int)MathCeil((double)MathMax(m_barrierHorizonBars, 1) / m_miStrideBars) : 1;
|
|
if(blockRows < 1)
|
|
blockRows = 1;
|
|
if(blockRows > n)
|
|
blockRows = n;
|
|
int blocks = (n + blockRows - 1) / blockRows;
|
|
m_miNullBlocks = blocks;
|
|
//--- Fisher-Yates over BLOCK ORDER; within-block order is left untouched, which is what preserves
|
|
//--- the local dependence. Copied out rather than swapped in place because blocks are not
|
|
//--- interchangeable in size - the last one is short whenever blockRows does not divide n.
|
|
int order[];
|
|
ArrayResize(order, blocks);
|
|
for(int b = 0; b < blocks; b++)
|
|
order[b] = b;
|
|
for(int b = blocks - 1; b > 0; b--)
|
|
{
|
|
//--- ShuffleRandomIndex, not MathRand()%: with blockRows == 1 the block count equals the row
|
|
//--- count, which can exceed MathRand()'s 15-bit range - same bias as the pass-2 queue shuffle.
|
|
int j = ShuffleRandomIndex(b + 1);
|
|
int t = order[b];
|
|
order[b] = order[j];
|
|
order[j] = t;
|
|
}
|
|
int shuffled[];
|
|
ArrayResize(shuffled, n);
|
|
int w = 0;
|
|
for(int b = 0; b < blocks && w < n; b++)
|
|
{
|
|
int src = order[b] * blockRows;
|
|
for(int q = 0; q < blockRows && w < n; q++)
|
|
{
|
|
int s = src + q;
|
|
shuffled[w++] = (s < n) ? labels[s] : labels[n - 1];
|
|
}
|
|
}
|
|
for(int i = 0; i < n; i++)
|
|
labels[i] = shuffled[i];
|
|
}
|
|
//--- H(Y) over the sampled labels, so the caller can express MI as a fraction of the information the
|
|
//--- label actually contains. Computed AFTER any shuffle, which leaves it unchanged by construction
|
|
//--- (a permutation preserves the class counts) - that invariance is itself a check on the shuffle.
|
|
int classCount[3] = {0, 0, 0};
|
|
for(int k = 0; k < n; k++)
|
|
classCount[labels[k]]++;
|
|
m_miLabelEntropy = 0.0;
|
|
for(int c = 0; c < 3; c++)
|
|
{
|
|
if(classCount[c] <= 0)
|
|
continue;
|
|
double pc = (double)classCount[c] / n;
|
|
m_miLabelEntropy -= pc * MathLog(pc);
|
|
}
|
|
double colVals[];
|
|
ArrayResize(colVals, n);
|
|
double total = 0.0;
|
|
m_miBestColumn = 0.0;
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
{
|
|
for(int k = 0; k < n; k++)
|
|
colVals[k] = cols[k * m_neuronsCount + f];
|
|
double mi = FeatureColumnMI(colVals, labels, n);
|
|
total += mi;
|
|
if(mi > m_miBestColumn)
|
|
m_miBestColumn = mi;
|
|
}
|
|
return total / m_neuronsCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Extract + score in one call - the form the coordinate sweep uses, |
|
|
//| where each candidate genuinely needs a fresh extraction because |
|
|
//| the indicator settings (and therefore the features) just changed. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::ScoreCurrentParamsByMI(bool shuffleLabels = false)
|
|
{
|
|
double cols[];
|
|
int labels[];
|
|
//--- MI_TUNE_TARGET, not the barrier label - see the define's comment: the tuner selects
|
|
//--- indicator settings for the channel with measured signal (realised RANGE), not the one
|
|
//--- measured at the noise floor (direction).
|
|
int n = BuildMiSample(cols, labels, 0, 0, MI_TUNE_TARGET);
|
|
if(n < MI_MIN_SAMPLES)
|
|
return -1.0;
|
|
return ScoreMiSample(cols, labels, n, shuffleLabels);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| FILTER-BASED indicator tuning. Replaced the genetic + successive- |
|
|
//| halving search on 2026-08-01. |
|
|
//| |
|
|
//| WHY THE GA HAD TO GO - measured, not assumed. Its cost was |
|
|
//| population x generations x rungs x seeds x eras-per-rung: |
|
|
//| rung 0: 8 cand x 3 seeds x 3 eras = 72 eras |
|
|
//| rung 1: 4 cand x 3 seeds x 8 eras = 96 |
|
|
//| rung 2: 2 cand x 3 seeds x 20 eras = 120 |
|
|
//| = 288 eras per generation x 4 generations = 1152 eras |
|
|
//| BEFORE the winner's real training started. Measured on SP500 H1: |
|
|
//| 9.3 h for the perceptron, 13.2 h for conv, ~48 h for LSTM and |
|
|
//| hybrid. Two days to tune is not a first-run experience. |
|
|
//| |
|
|
//| And it bought nothing. The space here is 90 points (10 MA periods |
|
|
//| x 9 MA types), so 1152 evaluations revisited each point ~13 times; |
|
|
//| meanwhile rungs of 3 and 8 eras cannot separate two MA periods at |
|
|
//| all - the 2026-08-01 run's finalists all scored 25.0-25.9% |
|
|
//| balanced accuracy, i.e. indistinguishable noise, and it then |
|
|
//| deployed the "winner" of that. |
|
|
//| |
|
|
//| THE REAL ERROR was using a full training run as the scoring |
|
|
//| function for a feature's period. The reference book does not: ch. |
|
|
//| 3.3 selects inputs by measuring each candidate indicator's |
|
|
//| CORRELATION with the target and dropping the ones with none, with |
|
|
//| no network involved. Mutual information is the same idea without |
|
|
//| the linearity assumption, which matters here because the label is |
|
|
//| 3-class categorical and the features are not monotonically related |
|
|
//| to it. Scoring is then arithmetic over cached features: seconds, |
|
|
//| not hours, and it scales with the number of enabled features |
|
|
//| rather than with topology cost - so LSTM tunes as fast as the MLP. |
|
|
//| |
|
|
//| COORDINATE SWEEP, not a product sweep: each parameter is optimised |
|
|
//| against the others' current values, one at a time. Cost is the SUM |
|
|
//| of the per-parameter candidate counts, not their product, so |
|
|
//| enabling every indicator stays affordable. Two passes, because the |
|
|
//| second can exploit what the first learned about the others; it |
|
|
//| stops early the moment a pass changes nothing. |
|
|
//| |
|
|
//| HONEST LIMIT, stated because it is the price of the trade: MI is a |
|
|
//| MARGINAL measure. It scores each feature column on its own, so a |
|
|
//| parameter that only pays off in combination with another can be |
|
|
//| missed. That is the standard filter-vs-wrapper tradeoff (Guyon & |
|
|
//| Elisseeff 2003). Given the wrapper here was ranking pure noise at |
|
|
//| 48 h a run, a fast marginal score is strictly the better deal. |
|
|
//| |
|
|
//| OBJECTIVE RE-POINTED 2026-08-16: candidates are scored against the |
|
|
//| excursion RANGE target (MI_TUNE_TARGET), not the barrier label - |
|
|
//| see the define's comment in ExpertSignalAIBase.mqh. Direction is |
|
|
//| measured noise, so the old objective gave the sweep a flat |
|
|
//| landscape and the gate rightly rejected every "winner"; RANGE is |
|
|
//| the channel with measured signal and the one the excursion head |
|
|
//| actually consumes these features for. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::TuneIndicatorsByFilter(void)
|
|
{
|
|
double best[];
|
|
m_indicatorTuner.Flatten(best);
|
|
double bestScore = ScoreCurrentParamsByMI();
|
|
if(bestScore < 0.0)
|
|
{
|
|
Print(ID + ": auto-tune skipped - not enough labelled in-sample bars to score indicator settings");
|
|
return;
|
|
}
|
|
double startScore = bestScore;
|
|
int evaluated = 0;
|
|
uint t0 = GetTickCount();
|
|
//--- SPREAD OF THE CANDIDATE SCORES. Without it "no improvement" is ambiguous between two readings
|
|
//--- that want opposite responses: INERT (trial scores identical to the incumbent because the
|
|
//--- parameter change never reaches the scored features, so `sc > bestScore` can never fire) versus
|
|
//--- LIVE and genuinely finding nothing. A spread of exactly zero says the first; a spread near the
|
|
//--- estimator's own noise says the second - and then the winner needs the family-wise gate the
|
|
//--- geometry scan and lag profile now carry, because installing a winner CHANGES THE FEATURE VECTOR
|
|
//--- and forces a fresh topology, a far heavier consequence than a printed row.
|
|
//---
|
|
//--- This measures the distinction directly, which is the point: the run-to-run evidence cannot settle
|
|
//--- it. "No improvement" on four consecutive runs (2026-08-05/06, 17 candidates) looks damning if the
|
|
//--- runs are treated as independent trials, but they are NOT - the scorer is deterministic and the
|
|
//--- runs cover nearly the same bars, so an incumbent that is the maximum on this data is the maximum
|
|
//--- on every run. That is one ~1-in-18 observation with three correlated repeats, not four of them.
|
|
//--- Note also that the INERT failure has already happened once here in a different form and was
|
|
//--- fixed (see the BufferTempData note in BuildMiSample: every candidate scored exactly 0.0000).
|
|
//--- Non-zero scores now mean that particular fault is gone.
|
|
double candMin = DBL_MAX, candMax = -DBL_MAX;
|
|
int readyMin = INT_MAX;
|
|
//--- The configured settings, kept so a winner that fails the gate below can be handed back. best[] is
|
|
//--- mutated in place by the descent, so it cannot serve as the restore point.
|
|
double configured[];
|
|
ArrayCopy(configured, best);
|
|
for(int pass = 0; pass < MI_TUNE_PASSES; pass++)
|
|
{
|
|
bool improvedThisPass = false;
|
|
for(int p = 0; p < AD_TUNE_PARAM_COUNT; p++)
|
|
{
|
|
//--- skip parameters whose indicator is switched off - they cannot affect the feature vector
|
|
int owner = m_indicatorTuner.ParamOwner(p);
|
|
bool on = (owner == 0 && m_useADCumulativeDelta) || (owner == 1 && m_useADShorteningOfThrust) ||
|
|
(owner == 2 && m_useADWyckoffEventStream) || (owner == 3 && m_useADWyckoffFailedStructure) ||
|
|
(owner == 4 && m_useADWyckoffSignificantBarInversion) || (owner == 5 && m_useMA) ||
|
|
(owner == 6 && m_useRSI) || (owner == 7 && m_useMACD) || (owner == 8 && m_useIchimoku);
|
|
if(!on)
|
|
continue;
|
|
double cands[];
|
|
int nc = m_indicatorTuner.ParamCandidates(p, cands);
|
|
double keep = best[p];
|
|
for(int c = 0; c < nc; c++)
|
|
{
|
|
//--- The longest uninterruptible stretch in the EA: every candidate re-creates handles,
|
|
//--- refreshes, and scores a full MI sample. Asked per candidate so a stop request costs at
|
|
//--- most one candidate rather than the rest of the descent - see ShutdownRequested().
|
|
if(ShutdownRequested())
|
|
{
|
|
//--- Hand the OPERATOR's settings back before leaving. best[] is mutated in place by the
|
|
//--- descent and the tuner object currently carries the LAST TRIAL's parameters, which
|
|
//--- nothing chose and which the .cfg would otherwise persist as if it had been selected.
|
|
//--- Deliberately no ReInitADIndicators/RefreshData here - those build indicator handles
|
|
//--- for a program that is being torn down, and spending the teardown budget on them is
|
|
//--- exactly what strands the chart objects.
|
|
m_indicatorTuner.Unflatten(configured);
|
|
PrintFormat("%s: auto-tune ABANDONED after %d candidates - stop requested. Configured"
|
|
" indicator settings restored; nothing installed.", ID, evaluated);
|
|
return;
|
|
}
|
|
if(cands[c] == keep)
|
|
continue; // already scored as the incumbent
|
|
double trial[];
|
|
ArrayCopy(trial, best);
|
|
trial[p] = cands[c];
|
|
m_indicatorTuner.Unflatten(trial);
|
|
ReInitADIndicators(m_indicatorsPtr); // also invalidates the feature cache (params changed)
|
|
//--- REFRESH, or the re-init changes nothing that the scorer can see. ReInitADIndicators
|
|
//--- creates a NEW handle carrying the new parameters and flags the feature cache stale, so
|
|
//--- features are genuinely recomputed - but BufferTempDataCompute() reads the CIndicatorBuffer
|
|
//--- objects, and only Refresh() copies data out of a handle into those. Without this the
|
|
//--- buffers still hold values copied from the PREVIOUS handle, so every candidate is scored on
|
|
//--- identical features. Measured on SP500 H1 2026-08-07: all 17 candidates returned exactly
|
|
//--- 0.00359 nats, a candidate-score span of 0.00000.
|
|
RefreshData();
|
|
int ready = TunableBarsCalculated();
|
|
if(ready >= 0)
|
|
readyMin = (int)MathMin(readyMin, ready);
|
|
double sc = ScoreCurrentParamsByMI();
|
|
evaluated++;
|
|
if(sc >= 0.0)
|
|
{
|
|
candMin = MathMin(candMin, sc);
|
|
candMax = MathMax(candMax, sc);
|
|
}
|
|
if(sc > bestScore)
|
|
{
|
|
bestScore = sc;
|
|
keep = cands[c];
|
|
improvedThisPass = true;
|
|
}
|
|
}
|
|
best[p] = keep;
|
|
}
|
|
if(!improvedThisPass)
|
|
break; // coordinate descent has converged - further passes cannot move anything
|
|
}
|
|
//--- SELECTION GATE. bestScore is a MAXIMUM over every candidate scored, so it carries the same defect
|
|
//--- the barrier-geometry winner test and the lag profile were fixed for: the maximum of N draws from a
|
|
//--- null sits well above any single draw, and installing on "it beat the incumbent" alone crowns noise.
|
|
//--- The stakes here are higher than either of those, because this one ACTS - it replaces the user's
|
|
//--- deliberate indicator settings and forces BuildFreshTopology(), so the network then trains on
|
|
//--- whatever the noise picked.
|
|
//---
|
|
//--- Test: draw the winner's own permutation null once (the sample is extracted once and every draw
|
|
//--- reshuffles it - see ScoreMiSample), take the per-candidate p, then correct it for having CHOSEN
|
|
//--- this candidate out of N with Sidak: p_family = 1 - (1 - p)^N. Sidak rather than an explicit
|
|
//--- max-of-N resample because each candidate here has a DIFFERENT feature set, so their draws cannot
|
|
//--- be pooled the way the geometry scan's can; Sidak needs only the one null and is exact under
|
|
//--- independence, mildly anti-conservative under positive dependence - stated rather than hidden.
|
|
//---
|
|
//--- WHAT THIS DOES NOT ESTABLISH: that the winner beats the INCUMBENT by a significant margin. It
|
|
//--- bounds the "best of N noise draws" failure, which is the one that was actually live here. Requiring
|
|
//--- bestScore > startScore as well means a change needs both an improvement and a defensible signal.
|
|
bool install = (bestScore > startScore);
|
|
double pFamily = 1.0;
|
|
int distinct = (int)MathMax(evaluated + 1, 1); // candidates scored, plus the incumbent
|
|
if(install)
|
|
{
|
|
double wc[];
|
|
int wl[];
|
|
//--- same target as the sweep's scorer, or the gate would test the winner against a
|
|
//--- different question than the one it was selected on
|
|
int wn = BuildMiSample(wc, wl, 0, 0, MI_TUNE_TARGET);
|
|
if(wn >= MI_MIN_SAMPLES)
|
|
{
|
|
double obs = ScoreMiSample(wc, wl, wn, false);
|
|
int atLeast = 0, draws = 0;
|
|
for(int s = 0; s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
//--- A truncated null is not a smaller null, it is a WRONG one - fewer draws shifts p toward
|
|
//--- significance. So a stop here abandons the test entirely (draws stays 0, pFamily stays
|
|
//--- 1.0, install becomes false) rather than installing on a partial null.
|
|
if(ShutdownRequested())
|
|
{
|
|
draws = 0;
|
|
break;
|
|
}
|
|
double d = ScoreMiSample(wc, wl, wn, true);
|
|
if(d < 0.0)
|
|
continue;
|
|
if(d >= obs)
|
|
atLeast++;
|
|
draws++;
|
|
}
|
|
if(draws > 0)
|
|
{
|
|
double pSingle = (double)(1 + atLeast) / (draws + 1);
|
|
pFamily = 1.0 - MathPow(1.0 - pSingle, (double)distinct);
|
|
}
|
|
}
|
|
install = (pFamily <= MI_TUNE_ALPHA);
|
|
}
|
|
if(!install)
|
|
{
|
|
ArrayCopy(best, configured);
|
|
bestScore = startScore;
|
|
}
|
|
//--- install the winner and leave the indicators/feature cache consistent with it
|
|
m_indicatorTuner.Unflatten(best);
|
|
ReInitADIndicators(m_indicatorsPtr);
|
|
RefreshData();
|
|
//--- A gated INSTALL is chart-level news, not just this model's: persist the winning periods so
|
|
//--- the classic votes, the signal-DB key and every later tuner seed adopt them on the next
|
|
//--- attach (restart-grained - see Variables\TunedPeriods.mqh for why not mid-run).
|
|
if(install)
|
|
SaveTunedPeriods(m_indicatorTuner.maPeriod, m_indicatorTuner.maType, m_indicatorTuner.rsiPeriod,
|
|
m_indicatorTuner.macdFast, m_indicatorTuner.macdSlow, m_indicatorTuner.macdSignal,
|
|
m_indicatorTuner.ichiTenkan, m_indicatorTuner.ichiKijun, m_indicatorTuner.ichiSenkou);
|
|
double candSpread = (evaluated > 0 && candMax >= candMin) ? (candMax - candMin) : 0.0;
|
|
Print(ID + StringFormat(": auto-tune complete - %d candidate settings scored in %.1fs, "
|
|
"feature/label mutual information %.5f -> %.5f nats%s | candidate scores span "
|
|
"%.5f (%.5f..%.5f)%s",
|
|
evaluated, (GetTickCount() - t0) / 1000.0, startScore, bestScore,
|
|
(bestScore <= startScore ? " (no improvement - keeping the configured settings)" : ""),
|
|
candSpread, (evaluated > 0 ? candMin : 0.0), (evaluated > 0 ? candMax : 0.0),
|
|
(evaluated > 0 && candSpread <= 0.0
|
|
? StringFormat(" <-- ZERO SPREAD: every candidate scored identically, so the "
|
|
"parameter change is STILL not reaching the scored features even "
|
|
"with the post-re-init RefreshData(). Least-ready tunable handle "
|
|
"had %d bars calculated - if that is 0 or far below the study "
|
|
"window, the handles are simply not done calculating yet and the "
|
|
"tuner needs to yield between candidates rather than score them "
|
|
"back to back.", (readyMin == INT_MAX ? -1 : readyMin))
|
|
: StringFormat(" | winner %s (selection p=%.4f after correcting for %d "
|
|
"candidates, need <=%.2f)",
|
|
(install ? "INSTALLED" : "REJECTED - keeping the configured "
|
|
"settings, since the best of N noise draws beats its incumbent "
|
|
"almost every time"),
|
|
pFamily, distinct, MI_TUNE_ALPHA))));
|
|
//--- An EXACTLY zero score is not a weak feature set, it is a broken measurement. Mutual information
|
|
//--- estimated from finite samples is biased UPWARD - roughly (bins-1)(classes-1)/(2N) nats, ~0.0035
|
|
//--- here - so even columns of pure noise score above zero. Landing on 0.0000 means every column read
|
|
//--- back constant, which is what a feature-extraction fault looks like. Said out loud because the
|
|
//--- first version of this function did exactly that and reported it as "no improvement".
|
|
if(bestScore <= 0.0)
|
|
Print(ID + ": WARNING - every candidate scored 0.0000 nats. Finite-sample bias alone should put "
|
|
"noise above zero, so this indicates the feature values are not being read, not that the "
|
|
"features are uninformative. Indicator settings left at their configured values.");
|
|
ReportFeatureLabelInformation();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| "Do these features predict this label at all?" - answered without |
|
|
//| training, topology or convergence, so unlike every accuracy |
|
|
//| number in this codebase it cannot be confounded by an optimizer |
|
|
//| or an objective. |
|
|
//| |
|
|
//| DELIBERATELY SEPARATE FROM THE TUNER, and not gated on era 0 with |
|
|
//| it. The sweep must only run on a fresh model - re-tuning would |
|
|
//| change the input vector out from under weights already fitted to |
|
|
//| the old one - but this reads the same cached features and writes |
|
|
//| nothing, so tying it to that gate meant the only way to see the |
|
|
//| answer was to bin a model mid-run (45 trained eras, on 2026-08-01) |
|
|
//| purely to re-ask a read-only question. Runs once per attach. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportFeatureLabelInformation(void)
|
|
{
|
|
m_miReportDone = true;
|
|
//--- PERMUTATION TEST, done properly. Build the current settings' sample ONCE, take the observed
|
|
//--- statistics from it, then reuse that same sample for every null draw - extraction is the expensive
|
|
//--- part, so this makes a few hundred permutations cost about what five used to.
|
|
//---
|
|
//--- Five was not enough, and the 2026-08-01 log is the proof: all four charts scored the IDENTICAL
|
|
//--- 0.00401 nats on identical features and identical labels, yet reported z of +1.3, +2.0, +4.0 and
|
|
//--- +4.7 - two "at the noise floor", two "real". The whole swing came from estimating the null's spread
|
|
//--- from five draws, where the standard deviation of the standard-deviation estimate is ~35%. The
|
|
//--- denominator was noisier than the effect.
|
|
//---
|
|
//--- So: no z-score and no normality assumption. An EMPIRICAL p-value, counting how many null draws
|
|
//--- reached the observed value, with the +1/(B+1) correction (Phipson & Smyth 2010) that keeps p from
|
|
//--- ever being reported as exactly zero - the test can only ever bound p below by 1/(B+1).
|
|
double cols[];
|
|
int labels[];
|
|
int nSample = BuildMiSample(cols, labels);
|
|
double observed = (nSample >= MI_MIN_SAMPLES) ? ScoreMiSample(cols, labels, nSample, false) : -1.0;
|
|
double signalBestCol = m_miBestColumn;
|
|
double labelEntropy = m_miLabelEntropy;
|
|
double floorSum = 0.0, floorSumSq = 0.0, floorBestColSum = 0.0;
|
|
int draws = 0, atLeastMean = 0, atLeastBestCol = 0;
|
|
uint tPerm = GetTickCount();
|
|
for(int s = 0; observed >= 0.0 && s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
//--- A few hundred full MI scorings, and nothing below can start until they finish. Abandon the
|
|
//--- test outright rather than truncate it: fewer draws does not make a smaller null, it makes a
|
|
//--- WRONG one (p shifts toward significance), and this null is what licenses the direction target.
|
|
if(ShutdownRequested())
|
|
{
|
|
draws = 0;
|
|
break;
|
|
}
|
|
double sc = ScoreMiSample(cols, labels, nSample, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
floorSumSq += sc * sc;
|
|
floorBestColSum += m_miBestColumn;
|
|
if(sc >= observed)
|
|
atLeastMean++;
|
|
//--- The MAX over columns is compared against the null distribution OF THE MAX, which corrects for
|
|
//--- testing 26 features at once by construction - no Bonferroni needed, and far less conservative.
|
|
if(m_miBestColumn >= signalBestCol)
|
|
atLeastBestCol++;
|
|
draws++;
|
|
}
|
|
//--- Left the null early because the program is going away: everything from here on either prints a
|
|
//--- number derived from `draws` or starts another scan. Neither is worth a millisecond of the teardown
|
|
//--- budget, and m_dirEvidence staying false is the safe direction (a torn-down run deploys nothing).
|
|
if(ShutdownRequested())
|
|
return;
|
|
double floorMean = (draws > 0) ? floorSum / draws : -1.0;
|
|
double floorVar = (draws > 1) ? MathMax(0.0, floorSumSq / draws - floorMean * floorMean) : 0.0;
|
|
double floorSd = MathSqrt(floorVar * (draws > 1 ? (double)draws / (draws - 1) : 1.0));
|
|
double floorBestCol = (draws > 0) ? floorBestColSum / draws : -1.0;
|
|
double pMean = (draws > 0) ? (double)(1 + atLeastMean) / (draws + 1) : 1.0;
|
|
double pBestCol = (draws > 0) ? (double)(1 + atLeastBestCol) / (draws + 1) : 1.0;
|
|
//--- Two SEPARATE questions, because at these sample sizes a small p can accompany a worthless effect.
|
|
//--- (1) Is it real - the p-values. (2) Is it big enough to trade - the excess as a share of H(Y), i.e.
|
|
//--- of everything there is to know about the label. Both are printed; neither is collapsed into a verdict
|
|
//--- that hides the other.
|
|
double excessShare = (labelEntropy > 1e-9 && floorMean >= 0.0)
|
|
? 100.0 * (observed - floorMean) / labelEntropy : 0.0;
|
|
string verdict = (draws > 0 && pMean <= 0.05)
|
|
? "above the noise floor - a real association"
|
|
: "AT THE NOISE FLOOR - indistinguishable from shuffled labels";
|
|
//--- SCREEN, not commentary. This test asks directly whether the features carry information about the
|
|
//--- DIRECTIONAL barrier label, so clearing it licenses the direction target on its own - see
|
|
//--- m_dirEvidence. It is an OR with the excursion-asymmetry screen because the two look for the same
|
|
//--- thing by different routes and either finding is sufficient; requiring both would reject on the
|
|
//--- weaker of two independent measurements.
|
|
if(draws > 0 && pMean <= 0.05)
|
|
{
|
|
m_dirEvidence = true;
|
|
m_dirEvidenceWhy = "feature/label mutual information cleared its block-permuted null";
|
|
}
|
|
//--- Name the feature vector this was measured on. These numbers are only about the model if the two
|
|
//--- match, and on 2026-08-02 they did not: the report ran before the cross-asset panel existed and
|
|
//--- silently described a narrower vector than training used. Stating the width and the panel's
|
|
//--- presence makes that mismatch visible in the log instead of requiring a timestamp comparison.
|
|
string vecNote = StringFormat("%d features/bar, cross-asset %s", m_neuronsCount,
|
|
m_crossAsset.IsReady()
|
|
? "PRESENT"
|
|
: "ABSENT (reference symbols unsynchronised - these numbers describe "
|
|
"a NARROWER vector than training will use)");
|
|
Print(ID + StringFormat(": feature/label information - %.5f nats/feature vs a shuffled-label null of "
|
|
"%.5f +/- %.5f over %d permutations, p=%.4f; strongest single feature %.5f vs "
|
|
"%.5f (null max, p=%.4f); excess is %.2f%% of the label's %.3f nats of entropy "
|
|
"(%d samples %d bars apart = %d independent blocks over a %d-bar horizon, "
|
|
"%.1fs) [%s]. %s.",
|
|
observed, floorMean, floorSd, draws, pMean,
|
|
signalBestCol, floorBestCol, pBestCol, excessShare, labelEntropy,
|
|
nSample, m_miStrideBars, m_miNullBlocks, m_barrierHorizonBars,
|
|
(GetTickCount() - tPerm) / 1000.0, vecNote, verdict));
|
|
//--- POWER, stated up front. The block permutation above makes the p-value HONEST under overlapping
|
|
//--- labels, but it cannot manufacture information that overlap destroyed: when the sampling stride is
|
|
//--- far shorter than the horizon there are few genuinely independent blocks, and a handful of blocks
|
|
//--- cannot resolve an effect this small however many rows they contain. Saying so prevents the opposite
|
|
//--- error to the one this replaced - reading "not significant" as "no signal" when it means "not enough
|
|
//--- independent data to tell".
|
|
if(m_miNullBlocks > 0 && m_miNullBlocks < 30)
|
|
Print(ID + StringFormat(": NOTE - only %d independent label blocks in this sample (%d-bar horizon, "
|
|
"%d-bar sampling stride). The rows overlap heavily, so this test has little "
|
|
"power: treat a non-significant result here as 'not enough independent "
|
|
"history to answer', not as 'no signal'. More history, or a shorter horizon, "
|
|
"is what would settle it.", m_miNullBlocks, m_barrierHorizonBars,
|
|
m_miStrideBars));
|
|
//--- Stated every time, not only on a bad result: this measure is MARGINAL and PER-BAR, while the network
|
|
//--- reads m_historyBars bars at once. It can therefore only ever prove that signal EXISTS, never that it
|
|
//--- does not - an interaction across features or across time is invisible to it by construction. Said
|
|
//--- out loud so a floor-level reading is not over-read into "this instrument is unpredictable".
|
|
if(!(draws > 0 && pMean <= 0.05))
|
|
Print(ID + ": NOTE - that measure is marginal (one feature at a time) and per-bar, whereas the "
|
|
"network sees " + IntegerToString((int)m_historyBars) + " bars jointly. A floor-level reading "
|
|
"rules out a simple per-feature edge; it cannot rule out one that only exists in combination "
|
|
"or across time. It does mean no per-feature indicator retuning will help.");
|
|
if(observed < 0.0)
|
|
return;
|
|
//--- POSITIVE CONTROL. Three separate "measurements" in this codebase have turned out to be silent
|
|
//--- no-ops that produced plausible numbers (the MI scorer reading an array nobody filled; the
|
|
//--- eval-mode guard that switched off the imbalance correction; the alternation gate whose premise was
|
|
//--- never true). A floor reading is therefore worthless until the instrument is shown to respond to a
|
|
//--- signal that is KNOWN to be there. This one is free: the label of a NEIGHBOURING sample row. Rows are
|
|
//--- `stride` bars apart, far inside the barrier horizon, so their outcome windows overlap heavily and
|
|
//--- the two labels must be strongly associated. Fed through the identical binning and estimator as every
|
|
//--- other column. If THIS lands near the floor, the estimator is broken and no MI number above means
|
|
//--- anything; if it lands far above, a floor reading on the real features can be believed.
|
|
//--- The control pairs each row's label with the label of a bar a FIXED, KNOWN distance away, so the two
|
|
//--- outcome windows overlap heavily and must be strongly associated.
|
|
//--- THIS CONTROL HAS NOW CRIED WOLF TWICE, AND BOTH TIMES THE ESTIMATOR WAS INNOCENT.
|
|
//--- 2026-08-01 it paired with the NEXT SAMPLE ROW, whose distance is the sampling stride - and stride
|
|
//--- varies with how much history a symbol has, so the control's strength varied with the
|
|
//--- cell rather than with the estimator. All three M5 cells (stride 160-717 bars against a
|
|
//--- 128-bar horizon, i.e. windows that do not overlap AT ALL) voided their own results.
|
|
//--- 2026-08-02 the range was padded by |offset|, which moved the offset build's FIRST BAR as well as
|
|
//--- its label, so row k of one build sat `offset` bars from row k of the other and the
|
|
//--- label was shifted a further `offset`: the pair was 2x as far apart as reported. On
|
|
//--- SP500 H1 it printed 0.00307 nats for "24 bars apart" - which is the true value for 48
|
|
//--- bars - failed its 5x gate, and stamped "every mutual-information figure above is void"
|
|
//--- on measurements that were fine. Confirmed by computing the same quantity independently
|
|
//--- in research/test_mi_control.py: 0.01655 at 24 bars, 0.00298 at 48.
|
|
//--- The lesson both share: A CONTROL THAT DEPENDS ON THE THING IT CERTIFIES CANNOT CERTIFY IT. Pin the
|
|
//--- control's distance to something the data cannot move, and make it a distance where the association
|
|
//--- is overwhelming rather than marginal - hence the adjacent bar below.
|
|
//--- TWO distances, and the GATE is the adjacent bar. Its barrier window overlaps the reference one by
|
|
//--- (h-1)/h, so "these must be associated" is unarguable, and unlike a horizon-relative offset it does
|
|
//--- not vary with the horizon, the stride or the symbol. The quarter-horizon figure is kept as a
|
|
//--- DIAGNOSTIC because it says something the gate cannot: how fast a triple-barrier label decorrelates.
|
|
//--- Measured independently on SP500 H1 (research/test_mi_control.py, 74k bars): 0.542 nats at 1 bar,
|
|
//--- 0.017 at 24, 0.003 at 48, against a ~0.002 floor. Note what that means - a quarter-horizon control
|
|
//--- clears a 5x gate by under 2x even when everything is working, which is far too little headroom for
|
|
//--- the one measurement whose job is to certify all the others.
|
|
//--- The headline verdict is printed; everything from here down is corroboration (a positive control,
|
|
//--- then a nine-offset alignment scan) and each step is another full MI sample build. Diagnostics are
|
|
//--- worth their cost while the EA is running and worth nothing while it is being torn down.
|
|
if(ShutdownRequested())
|
|
return;
|
|
double controlMi = -1.0, decorrMi = -1.0;
|
|
int controlBars = 1;
|
|
int decorrBars = MathMax(1, MathMax(m_barrierHorizonBars, 1) / 4);
|
|
{
|
|
//--- Rebuilt rather than reused because the permutation loop above destroyed the honest label
|
|
//--- ordering, and controlling against a shuffled array would measure the floor twice.
|
|
double c0[], cK[];
|
|
int l0[], lK[];
|
|
int n0 = BuildMiSample(c0, l0);
|
|
if(n0 >= MI_MIN_SAMPLES)
|
|
{
|
|
int offs[2];
|
|
offs[0] = controlBars;
|
|
offs[1] = decorrBars;
|
|
for(int oi = 0; oi < 2; oi++)
|
|
{
|
|
int nK = BuildMiSample(cK, lK, offs[oi]);
|
|
//--- Both builds are padded by the SAME fixed amount, so they enumerate the same bars with
|
|
//--- the same stride and row k of one is row k of the other. Sized from what actually came
|
|
//--- back, never from the caller's count.
|
|
int nc = MathMin(n0, nK);
|
|
if(nc < MI_MIN_SAMPLES)
|
|
continue;
|
|
double neighbourLabel[];
|
|
int selfLabels[];
|
|
ArrayResize(neighbourLabel, nc);
|
|
ArrayResize(selfLabels, nc);
|
|
for(int k = 0; k < nc; k++)
|
|
{
|
|
selfLabels[k] = l0[k];
|
|
neighbourLabel[k] = (double)lK[k];
|
|
}
|
|
double v = FeatureColumnMI(neighbourLabel, selfLabels, nc);
|
|
if(oi == 0)
|
|
controlMi = v;
|
|
else
|
|
decorrMi = v;
|
|
}
|
|
}
|
|
}
|
|
Print(ID + StringFormat(": MI positive control - the ADJACENT bar's label (windows overlap %d of %d bars) "
|
|
"scores %.5f nats against the ~%.5f noise floor; by a quarter horizon (%d bars) "
|
|
"it is already down to %.5f, which is how fast this target decorrelates. %s",
|
|
MathMax(m_barrierHorizonBars, 1) - 1, MathMax(m_barrierHorizonBars, 1),
|
|
controlMi, floorMean, decorrBars, decorrMi,
|
|
(controlMi > floorMean * 5.0)
|
|
? "The estimator detects a known association on this exact data, so a "
|
|
"floor-level reading above is a real finding and not a broken measurement."
|
|
: "WARNING - the estimator FAILED to detect an association that must be there. "
|
|
"Every mutual-information figure above is void; fix this before drawing any "
|
|
"conclusion from them."));
|
|
//--- ALIGNMENT SCAN. A floor reading has two very different causes: the features genuinely do not predict
|
|
//--- this target, or they DO and something upstream has knocked the two out of step (an off-by-one in the
|
|
//--- label index, a horizon applied to the wrong bar, a feature window that lags what it claims). Both
|
|
//--- destroy the information before any topology sees it, and both look identical in every accuracy number
|
|
//--- this EA prints - which is exactly why four different architectures all landed on the same precision.
|
|
//--- Re-scoring against the label taken from bar i+k separates them: a peak at some k != 0 IS a
|
|
//--- misalignment (and names its size), a flat profile says the features simply do not carry this target.
|
|
//--- THE TWO DIRECTIONS ARE NOT SYMMETRIC, and the first version of this scan treated them as if they
|
|
//--- were - it read the k>0 rise as a misalignment and cried "fix this before concluding anything",
|
|
//--- which was a false alarm produced by the diagnostic's own design.
|
|
//---
|
|
//--- Bar indices here are MQL5 SERIES indices: HIGHER index = OLDER bar (TripleBarrierLabel walks its
|
|
//--- window with `for(t = idx-1; t >= idx-horizon; t--)`, i.e. decreasing index = forward in time).
|
|
//--- So:
|
|
//--- k < 0 the label belongs to a NEWER bar, whose barrier window opens AFTER the features exist.
|
|
//--- Nothing at bar i can legitimately know it. A peak here is real LOOKAHEAD and is a bug.
|
|
//--- k > 0 the label belongs to an OLDER bar, whose window is already k bars into its life by the
|
|
//--- time bar i happens - so the features at bar i legitimately contain the realised first k
|
|
//--- bars of that outcome. MI MUST rise with k. That is arithmetic, not a defect.
|
|
//--- Only the k<0 side can indict the pipeline. The k>0 side is a second positive control, and its
|
|
//--- GRADIENT is the useful number: it says how fast a barrier outcome becomes knowable once the window
|
|
//--- is running, against how little is knowable at entry (k=0).
|
|
int offsets[] = { -5, -3, -2, -1, 0, 1, 2, 3, 5 };
|
|
string profile = "";
|
|
double atZero = -1.0, worstFuture = -1.0, farPast = -1.0;
|
|
int worstFutureK = 0;
|
|
for(int oi = 0; oi < ArraySize(offsets); oi++)
|
|
{
|
|
//--- Nine sample builds. A partial profile cannot be read - the lookahead test compares the k<0
|
|
//--- side against k=0 and both must exist - so a stop abandons the scan rather than printing a row
|
|
//--- with holes in it that would look like a null result at the missing offsets.
|
|
if(ShutdownRequested())
|
|
return;
|
|
double oc[];
|
|
int ol[];
|
|
int on = BuildMiSample(oc, ol, offsets[oi]);
|
|
double os = (on >= MI_MIN_SAMPLES) ? ScoreMiSample(oc, ol, on, false) : -1.0;
|
|
profile += StringFormat("%s%+d:%.5f", (oi > 0 ? " " : ""), offsets[oi], os);
|
|
if(offsets[oi] == 0)
|
|
atZero = os;
|
|
else
|
|
if(offsets[oi] < 0 && os > worstFuture)
|
|
{
|
|
worstFuture = os;
|
|
worstFutureK = offsets[oi];
|
|
}
|
|
else
|
|
if(offsets[oi] > 0)
|
|
farPast = os; // offsets ascend, so this ends on the largest k
|
|
}
|
|
//--- A MARGIN, not a bare comparison. Every one of these offsets is an estimate with the same noise as
|
|
//--- the headline statistic, so "k=-3 came out above k=0" is meaningless when the gap is smaller than the
|
|
//--- null's own spread. Shipped without this, the 2026-08-01 sweep flagged LOOKAHEAD on 7 of 12 cells on
|
|
//--- gaps of 0.00008-0.00040 nats against a measured null sd of ~0.00030 - all noise, every one. Three
|
|
//--- SDs is the same discipline the deploy floor already applies to precision: an anomaly has to clear
|
|
//--- the measurement error before it gets a name. (Third time this session that comparing two point
|
|
//--- estimates without their spread produced a confident wrong answer - see MI_NOISE_PERMUTATIONS.)
|
|
double lookaheadMargin = 3.0 * floorSd;
|
|
string alignVerdict;
|
|
if(worstFuture > atZero + lookaheadMargin)
|
|
alignVerdict = StringFormat(" | LOOKAHEAD - k=%d (a label whose barrier window opens AFTER these "
|
|
"features exist) scores %.5f against %.5f at k=0, clearing the %.5f "
|
|
"margin (3 sd of the null). The features can only score there by "
|
|
"containing future information. Fix that before trusting any accuracy "
|
|
"number this EA prints.", worstFutureK, worstFuture, atZero, lookaheadMargin);
|
|
else
|
|
alignVerdict = StringFormat(" | clean: no future label (k<0) beats k=0, so there is no lookahead. "
|
|
"The rise on the k>0 side is expected - those windows are already open, "
|
|
"so the features hold part of the answer - and its size is the finding: "
|
|
"%.5f at k=+5 against %.5f at k=0, i.e. ~%.1fx more is knowable %d bars "
|
|
"into a %d-bar window than at the entry the model actually trades.",
|
|
farPast, atZero, (atZero > 1e-9 ? farPast / atZero : 0.0), 5,
|
|
m_barrierHorizonBars);
|
|
Print(ID + ": MI label-alignment scan (label from bar i+k; higher index = OLDER bar, so k<0 is the "
|
|
"future) - " + profile + alignVerdict);
|
|
ReportFeatureLagProfile();
|
|
//--- Runs after the lag profile and before the geometry scan on purpose: the geometry scan chooses
|
|
//--- among SL/TP pairings, and this asks whether predicting SL/TP is a well-posed problem at all.
|
|
//--- Reading them in that order stops a geometry winner from being interpreted as evidence that the
|
|
//--- exit is learnable.
|
|
ReportExcursionInformation();
|
|
ReportBarrierGeometryScan();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| WHICH BARRIER GEOMETRY IS ACTUALLY PREDICTABLE AT ENTRY. |
|
|
//| |
|
|
//| The alignment scan established the shape of the problem: 4.7x more |
|
|
//| is knowable 5 bars into a 128-bar window than at the entry the |
|
|
//| model trades on. A 6xATR target reached over 128 bars is decided |
|
|
//| overwhelmingly by what happens DURING the window, so whatever the |
|
|
//| entry state knows is buried under 128 bars of subsequent noise. |
|
|
//| That is a property of the TARGET, and no topology can undo it - |
|
|
//| which is why four different architectures all landed on precision |
|
|
//| exactly equal to the base rate. |
|
|
//| |
|
|
//| So measure the target instead of guessing at it. For each SL/TP |
|
|
//| pairing the user can actually select, relabel the same sampled |
|
|
//| bars and score how much the SAME features say about THAT outcome. |
|
|
//| Seconds, no training, no topology. |
|
|
//| |
|
|
//| RANKED ON EXCESS OVER ITS OWN NULL, IN NATS. The first version |
|
|
//| divided that by the geometry's own H(Y), reasoning that each label |
|
|
//| has a different amount of information available to find. That was |
|
|
//| backwards and it produced a wrong answer on the first run: it |
|
|
//| named 3:10, whose horizon is CLAMPED (it wants ~320 bars and gets |
|
|
//| BARRIER_HORIZON_MAX), so most trades never resolve, Neutral |
|
|
//| dominates, H(Y) collapses - and dividing by a collapsing |
|
|
//| denominator made the most degenerate label look like the most |
|
|
//| predictable one. Subtracting each geometry's own measured null |
|
|
//| already removes the class-balance bias, which is the only thing |
|
|
//| the normalisation was needed for. |
|
|
//| |
|
|
//| A clamped geometry is DISQUALIFIED outright, not merely ranked |
|
|
//| down. The deployed EA holds until SL or TP with no bar limit, so a |
|
|
//| truncated label trains the model on a question the strategy never |
|
|
//| asks. Directional share is printed for the same reason: a label |
|
|
//| nobody can trade is not a candidate however well it scores. |
|
|
//| |
|
|
//| What it cannot tell you: chance precision equals the break-even |
|
|
//| win rate at every geometry (both are m/(m+k) under a driftless |
|
|
//| walk), so a tighter target does NOT buy expectancy on its own. It |
|
|
//| buys PREDICTABILITY - a shorter window has less noise piled on top |
|
|
//| of what the entry state knows. The ranking finds where the signal |
|
|
//| is largest; it is still on the model to convert it. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| HOW FAR BACK THE FEATURES STILL SAY ANYTHING - see the declaration.|
|
|
//| |
|
|
//| Returns the deepest lag whose score clears the null, or 0 when |
|
|
//| none does. Read-only; the caller decides what to do with it. |
|
|
//| |
|
|
//| The null is redrawn PER LAG rather than measured once and reused. |
|
|
//| Finite-sample MI bias depends on the realised class counts and the |
|
|
//| bin occupancy, and both move with the lag because different rows |
|
|
//| survive the validity checks - so a single shared floor would be |
|
|
//| the right number for lag 0 and the wrong one everywhere else. |
|
|
//| Cost is the reason it is a REDUCED draw count: a full |
|
|
//| MI_NOISE_PERMUTATIONS sweep at every lag is 200 x historyBars |
|
|
//| scorings. The gate below is deliberately crude for the same |
|
|
//| reason - this profile decides a LOOKBACK, not a trade. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| IS "OPTIMAL SL/TP" LEARNABLE? Scores the same features against |
|
|
//| four excursion targets instead of the barrier class. |
|
|
//| |
|
|
//| The question this exists to settle: predicting an optimal stop and |
|
|
//| target decomposes into HOW FAR price travels and WHICH WAY it goes |
|
|
//| first, and those two behave nothing alike. Excursion SIZE is a |
|
|
//| volatility question, and volatility clustering is one of the most |
|
|
//| robust regularities in markets - RANGE is included precisely as a |
|
|
//| positive control that SHOULD clear, and a run where it does not is |
|
|
//| evidence the measurement is broken rather than that the market is |
|
|
//| unpredictable. ASYMMETRY is direction wearing different clothes, |
|
|
//| and it is the only one of the four that can produce expectancy. |
|
|
//| |
|
|
//| So the informative outcome is the CONTRAST, not any single number: |
|
|
//| RANGE/UP/DOWN clearing while ASYM sits at the floor says size is |
|
|
//| predictable and order is not - i.e. the payoff of a predicted |
|
|
//| SL/TP is position sizing and drawdown control, not edge. That is |
|
|
//| worth having under prop-firm limits, and it is not a signal. |
|
|
//| Exit management on RANDOM entries already moved the payoff ratio |
|
|
//| 0.92 -> 5.72 with expectancy FLAT, so this would agree with a test |
|
|
//| that has already been run a different way. |
|
|
//| |
|
|
//| Why this is not answered by the existing verdicts: every MI figure |
|
|
//| this project has produced scored the TRIPLE-BARRIER label, which |
|
|
//| is one specific question ("does the target come before the stop at |
|
|
//| this fixed geometry"). A noise-floor result there says nothing |
|
|
//| about whether excursion MAGNITUDE is learnable - different target, |
|
|
//| different answer, and worth measuring before rebuilding a head. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportExcursionInformation(void)
|
|
{
|
|
int targets[] = { MI_TARGET_EXC_RANGE, MI_TARGET_EXC_UP, MI_TARGET_EXC_DOWN, MI_TARGET_EXC_ASYM,
|
|
MI_TARGET_EXC_ASYM_NORM };
|
|
string names[] = { "RANGE up+dn (volatility control)", "UP (MFE)", "DOWN (MAE)",
|
|
"ASYMMETRY up-dn (RAW - confounded by volatility, read the NORM line instead)",
|
|
"ASYMMETRY NORMALISED (up-dn)/(up+dn) (THE ONE THAT MATTERS)" };
|
|
bool asymCleared = false, sizeCleared = false, rawAsymCleared = false;
|
|
for(int k = 0; k < ArraySize(targets); k++)
|
|
{
|
|
//--- Five targets, each with its own full permutation null. Leaving early costs only the targets
|
|
//--- not yet reached; the ones already printed stand, and m_dirEvidence can only have been set by
|
|
//--- a null that ran to completion (see the draws=0 abandon below).
|
|
if(ShutdownRequested())
|
|
return;
|
|
double cols[];
|
|
int labels[];
|
|
int n = BuildMiSample(cols, labels, 0, 0, targets[k]);
|
|
if(n < MI_MIN_SAMPLES)
|
|
{
|
|
Print(ID + ": MI excursion - " + names[k] + ": not enough usable bars to score");
|
|
continue;
|
|
}
|
|
double observed = ScoreMiSample(cols, labels, n, false);
|
|
if(observed < 0.0)
|
|
continue;
|
|
double floorSum = 0.0;
|
|
int draws = 0, atLeast = 0;
|
|
for(int s = 0; s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
//--- Abandon, never truncate: this null is what sets m_dirEvidence via the NORMALISED asymmetry
|
|
//--- line, and a short null biases p downward - i.e. toward licensing a direction target on a
|
|
//--- test that never finished. draws=0 makes the target below skip cleanly.
|
|
if(ShutdownRequested())
|
|
{
|
|
draws = 0;
|
|
break;
|
|
}
|
|
double sc = ScoreMiSample(cols, labels, n, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
if(sc >= observed)
|
|
atLeast++;
|
|
draws++;
|
|
}
|
|
if(draws <= 0)
|
|
continue;
|
|
double floorMean = floorSum / draws;
|
|
double p = (double)(1 + atLeast) / (draws + 1);
|
|
bool clears = (p <= MI_LAG_ALPHA);
|
|
//--- H(Y) is ln(3) by construction (equal-frequency bins), so excess-as-a-share-of-entropy is
|
|
//--- comparable across all four targets and against the barrier label's own figure.
|
|
Print(ID + StringFormat(": MI excursion - %s: %.5f nats/feature vs a block-permuted null of %.5f, "
|
|
"p=%.4f over %d draws%s | %.2f%% of the target's %.3f nats (%d samples)",
|
|
names[k], observed, floorMean, p, draws, (clears ? " <-- CLEARS" : ""),
|
|
100.0 * (observed - floorMean) / MathLog(3.0), MathLog(3.0), n));
|
|
if(targets[k] == MI_TARGET_EXC_ASYM_NORM)
|
|
asymCleared = clears; // the ONLY one a directional claim may rest on
|
|
else
|
|
if(targets[k] == MI_TARGET_EXC_ASYM)
|
|
rawAsymCleared = clears;
|
|
else
|
|
if(clears)
|
|
sizeCleared = true;
|
|
}
|
|
//--- The verdict is the CONTRAST. Spelled out rather than left to be read off five numbers, because
|
|
//--- the wrong reading of "UP clears" is "we can predict profitable trades", and that is precisely
|
|
//--- the inference this report exists to prevent.
|
|
//---
|
|
//--- ORDER MATTERS, and the first version had it wrong: the generic size-not-direction branch was
|
|
//--- tested first, and it is true whenever size clears - i.e. always - so the CONFOUND branch was
|
|
//--- unreachable. Measured 2026-08-07 across three symbols: raw asymmetry cleared on all three while
|
|
//--- normalised collapsed on all three, and the one message that explains why never printed.
|
|
//--- THE SCREEN, not just the commentary. Normalised asymmetry is the only measure in this suite that
|
|
//--- is directional AND free of the volatility confound, so it is the one that can license a direction
|
|
//--- target. Recorded rather than merely printed - see m_dirEvidence.
|
|
if(asymCleared)
|
|
{
|
|
m_dirEvidence = true;
|
|
m_dirEvidenceWhy = "normalised excursion asymmetry cleared its block-permuted null";
|
|
}
|
|
if(asymCleared)
|
|
Print(ID + ": MI excursion VERDICT - NORMALISED ASYMMETRY CLEARS. Scale-free directional "
|
|
"information survives dividing the volatility out, which no barrier-label test has ever "
|
|
"found and which the raw asymmetry could not have established on its own. Before acting: "
|
|
"replicate on instruments NOT used to find it, and check the effect is not concentrated in "
|
|
"one volatility regime. If it holds, this is the first real signal here.");
|
|
else
|
|
if(rawAsymCleared)
|
|
Print(ID + ": MI excursion VERDICT - raw asymmetry cleared but the NORMALISED one did not. That "
|
|
"is the signature of the VOLATILITY CONFOUND, not of direction: up-dn scales with sigma, "
|
|
"so a predictable sigma pushes the value into both outer bins and scores while carrying no "
|
|
"directional content at all - and it does so on every instrument, so replication does not "
|
|
"argue against it. Read the raw line as a restatement of RANGE. Excursion SIZE is "
|
|
"predictable and worth using for position sizing and drawdown control; DIRECTION is not, "
|
|
"so no SL/TP head can create expectancy. Agrees with the random-entry exit test (payoff "
|
|
"ratio 0.92->5.72, expectancy flat).");
|
|
else
|
|
if(sizeCleared)
|
|
Print(ID + ": MI excursion VERDICT - excursion SIZE is predictable, DIRECTION is not. A model "
|
|
"trained to output SL/TP will therefore learn volatility, which is real and useful for "
|
|
"position sizing and drawdown control, but it CANNOT create expectancy: knowing the "
|
|
"next leg spans 3 ATR is worth nothing without knowing which side it spans first. "
|
|
"Agrees with the random-entry exit test (payoff ratio 0.92->5.72, expectancy flat). "
|
|
"Build the head for risk control and stop looking for edge in the exit.");
|
|
else
|
|
Print(ID + ": MI excursion VERDICT - NOTHING clears, INCLUDING the range control. Volatility "
|
|
"clustering is about the most robust regularity in markets, so a range target at the "
|
|
"noise floor points at the measurement, not the market - check the excursion cache "
|
|
"filled and that the sample is not dominated by one volatility regime.");
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ReportFeatureLagProfile(void)
|
|
{
|
|
int maxLag = (int)MathMin(MathMax(m_historyBars, 0), MI_LAG_MAX_PROFILE - 1);
|
|
if(maxLag <= 0)
|
|
return 0;
|
|
//--- Per-lag draws retained for the SAME reason the geometry scan retains its own: this report reads a
|
|
//--- profile of ~20 lags, so "does lag k clear ITS OWN null" is the wrong question at every k. See the
|
|
//--- family-wise block below.
|
|
double lagDraws[MI_LAG_MAX_PROFILE][MI_LAG_PERMUTATIONS];
|
|
double lagExcess[MI_LAG_MAX_PROFILE];
|
|
int lagCount[MI_LAG_MAX_PROFILE];
|
|
bool lagValid[MI_LAG_MAX_PROFILE];
|
|
double atZero = 0.0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
//--- ~21 lags x MI_LAG_PERMUTATIONS full scorings, and the family-wise block below cannot run on a
|
|
//--- partial profile (its bar is the null of the MAXIMUM over lags - drop lags and the maximum is
|
|
//--- taken over a different family). So a stop abandons the whole report and returns "no depth",
|
|
//--- which is what the caller already does with an unmeasurable profile.
|
|
lagValid[k] = false;
|
|
lagExcess[k] = 0.0;
|
|
lagCount[k] = 0;
|
|
if(ShutdownRequested())
|
|
return 0;
|
|
double cols[];
|
|
int labels[];
|
|
int n = BuildMiSample(cols, labels, 0, k);
|
|
if(n < MI_MIN_SAMPLES)
|
|
continue;
|
|
double observed = ScoreMiSample(cols, labels, n, false);
|
|
if(observed < 0.0)
|
|
continue;
|
|
//--- ScoreMiSample shuffles IN PLACE, so the observed statistic must be taken first (above) and the
|
|
//--- draws then reuse the same extracted sample - which is what makes this affordable at all.
|
|
double floorSum = 0.0;
|
|
int draws = 0;
|
|
for(int s = 0; s < MI_LAG_PERMUTATIONS; s++)
|
|
{
|
|
if(ShutdownRequested())
|
|
return 0;
|
|
double sc = ScoreMiSample(cols, labels, n, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
lagDraws[k][draws] = sc;
|
|
draws++;
|
|
}
|
|
if(draws < 2)
|
|
continue;
|
|
lagExcess[k] = observed - (floorSum / draws);
|
|
lagCount[k] = draws;
|
|
lagValid[k] = true;
|
|
if(k == 0)
|
|
atZero = lagExcess[k];
|
|
}
|
|
//--- FAMILY-WISE CORRECTION ACROSS LAGS. The first version of this report tested each lag against its
|
|
//--- own null at alpha=0.05 across ~21 lags, which is one expected false positive per run before any
|
|
//--- signal exists - and correlated features make them arrive in CLUSTERS that read like a hump. It
|
|
//--- did exactly that on SP500 H1: 2026-08-06 13:55 starred nothing, 16:22 starred k6/k10/k12/k16 and
|
|
//--- concluded "information survives to lag 16" - same instrument, same 31 features, same 2009 samples,
|
|
//--- while the headline MI moved the other way (p 0.4478 -> 0.8756, observed BELOW its null mean).
|
|
//--- Non-replication on identical data is the signature of an uncorrected multiple comparison.
|
|
//---
|
|
//--- So the bar is the null OF THE MAXIMUM over lags, exactly as the barrier-geometry winner test does
|
|
//--- over candidates: one draw from every lag, keep the largest, repeat. A lag clears only by beating
|
|
//--- that. Draws are centred leave-one-out so each is centred by a mean excluding itself, matching how
|
|
//--- the observed excess is centred. Independence across lags overstates the spread of the maximum
|
|
//--- (neighbouring lags share nearly all their feature window), so this errs toward rejecting.
|
|
int fwDraws = MI_LAG_PERMUTATIONS;
|
|
int validLags = 0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
if(lagValid[k])
|
|
{
|
|
fwDraws = (int)MathMin(fwDraws, lagCount[k]);
|
|
validLags++;
|
|
}
|
|
double fwMax[MI_LAG_PERMUTATIONS];
|
|
if(validLags <= 0)
|
|
fwDraws = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
{
|
|
double worst = -DBL_MAX;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
if(!lagValid[k])
|
|
continue;
|
|
double sum = 0.0;
|
|
for(int q = 0; q < lagCount[k]; q++)
|
|
sum += lagDraws[k][q];
|
|
double loo = (sum - lagDraws[k][s]) / (lagCount[k] - 1);
|
|
double e = lagDraws[k][s] - loo;
|
|
if(e > worst)
|
|
worst = e;
|
|
}
|
|
fwMax[s] = worst;
|
|
}
|
|
string profile = "";
|
|
int deepest = 0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
if(!lagValid[k])
|
|
{
|
|
profile += StringFormat(" k%d=n/a", k);
|
|
continue;
|
|
}
|
|
int atLeast = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
if(fwMax[s] >= lagExcess[k])
|
|
atLeast++;
|
|
double pFw = (fwDraws > 0) ? (double)(1 + atLeast) / (fwDraws + 1) : 1.0;
|
|
bool clears = (fwDraws > 0 && pFw <= MI_LAG_ALPHA);
|
|
if(clears)
|
|
deepest = k;
|
|
profile += StringFormat(" k%d=%+.5f%s", k, lagExcess[k], (clears ? "*" : ""));
|
|
}
|
|
Print(ID + StringFormat(": MI feature-lag profile (features from bar i+k, LABEL PINNED to the entry "
|
|
"bar i, so every k is causal; value is excess over that lag's own "
|
|
"block-permutation null; '*' = p<=%.2f against the null of the MAXIMUM over "
|
|
"%d lags, not against the lag's own null - %d lags tested one at a time would "
|
|
"star one per run on noise alone) -%s",
|
|
MI_LAG_ALPHA, validLags, validLags, profile));
|
|
if(deepest <= 0)
|
|
Print(ID + StringFormat(": MI feature-lag profile - NOTHING clears the family-wise null at ANY lag "
|
|
"out to %d bars (entry bar itself %+.5f). The %d-bar lookback is not costing "
|
|
"us information; there is none to lose. This is the blind spot the earlier "
|
|
"reports had: they scored the entry bar alone, so they could not have "
|
|
"distinguished 'no signal anywhere' from 'signal only in the older bars'.",
|
|
maxLag, atZero, maxLag));
|
|
else
|
|
Print(ID + StringFormat(": MI feature-lag profile - information survives to lag %d of %d, clearing "
|
|
"the null of the maximum over %d lags. A lookback shorter than %d would "
|
|
"discard measurable information; a longer one adds input width for none. "
|
|
"BEFORE ACTING ON THIS: re-run it. An uncorrected version of this report "
|
|
"gave opposite answers on two runs over identical data, so one run is not "
|
|
"a result - the shape has to reappear, and ideally on a second instrument.",
|
|
deepest, maxLag, validLags, deepest + 1));
|
|
return deepest;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportBarrierGeometryScan(void)
|
|
{
|
|
//--- SL x1 is deliberately absent: MIN_SL_ATR_MULTIPLIER floors it anyway, and it was rejected on this
|
|
//--- instrument as too tight to survive normal noise. TP grid is exactly the TAKE_PROFIT_MODE enum.
|
|
double slGrid[] = { 2.0, 3.0 };
|
|
int tpGrid[] = { 2, 3, 4, 6, 8, 10 };
|
|
int savedHorizon = m_barrierHorizonBars;
|
|
int barsNow = m_labelCacheBars;
|
|
uint t0 = GetTickCount();
|
|
string rows = "";
|
|
double bestExcess = -1.0;
|
|
string bestName = "";
|
|
int bestSl = 0, bestTp = 0;
|
|
//--- The winner's own horizon, captured while it is in scope. Needed by the DETECTABILITY guard on the
|
|
//--- adoption below: a wider pairing takes longer to resolve, and how long it takes is what decides
|
|
//--- how many INDEPENDENT observations the OOS window can ever yield.
|
|
int bestHorizon = 0;
|
|
//--- Per-candidate null draws, retained so the winner can be tested against the null of the MAXIMUM
|
|
//--- rather than against its own. Only ELIGIBLE candidates are enrolled: the family the maximum was
|
|
//--- actually taken over is the family the gate must correct for, and a disqualified pairing can never
|
|
//--- be the winner however it scores.
|
|
double drawMat[MI_GEOMETRY_MAX_CANDIDATES][MI_GEOMETRY_PERMUTATIONS];
|
|
int drawCount[MI_GEOMETRY_MAX_CANDIDATES];
|
|
int candidates = 0;
|
|
double cfgSl = 0.0, cfgTp = 0.0;
|
|
BarrierMultiples(cfgSl, cfgTp);
|
|
double cfgExcess = -1.0;
|
|
//--- CANDIDATE LIST, built up front so the CONFIGURED pair is scored alongside the grid instead of
|
|
//--- being looked up in it. It was looked up in it until 2026-08-17, and the lookup could not
|
|
//--- succeed: DeriveBarrierGeometry produces CONTINUOUS multiples (3.07 / 1.70) that will never equal
|
|
//--- an integer grid point, so `cfgExcess` stayed at its -1.0 sentinel and the report printed
|
|
//--- "configured 3:2 scores -1.00000" - which reads as a catastrophic score and is actually "never
|
|
//--- evaluated". The one mechanism built to price the shipped geometry could not see it.
|
|
//--- Worse, the skip below excluded target<stop entirely on the grounds that it "inverts the trade's
|
|
//--- whole premise" - while the derivation was shipping exactly that. The grid keeps the skip (an
|
|
//--- inverted ENUM pairing is still not something to crown), but the incumbent is always scored.
|
|
double candSl[], candTp[];
|
|
bool candIsCfg[];
|
|
int candN = 0;
|
|
ArrayResize(candSl, ArraySize(slGrid) * ArraySize(tpGrid) + 1);
|
|
ArrayResize(candTp, ArraySize(slGrid) * ArraySize(tpGrid) + 1);
|
|
ArrayResize(candIsCfg, ArraySize(slGrid) * ArraySize(tpGrid) + 1);
|
|
for(int a = 0; a < ArraySize(slGrid); a++)
|
|
for(int b = 0; b < ArraySize(tpGrid); b++)
|
|
{
|
|
if((double)tpGrid[b] < slGrid[a])
|
|
continue;
|
|
candSl[candN] = slGrid[a];
|
|
candTp[candN] = (double)tpGrid[b];
|
|
candIsCfg[candN] = false;
|
|
candN++;
|
|
}
|
|
if(cfgSl > 0.0 && cfgTp > 0.0)
|
|
{
|
|
candSl[candN] = cfgSl;
|
|
candTp[candN] = cfgTp;
|
|
candIsCfg[candN] = true;
|
|
candN++;
|
|
}
|
|
m_barrierScanLiveLabels = true;
|
|
{
|
|
for(int c = 0; c < candN; c++)
|
|
{
|
|
//--- THE HEAVIEST SCAN IN THE EA: every pairing relabels the whole sampled history and then draws
|
|
//--- MI_GEOMETRY_PERMUTATIONS nulls off it. Break rather than return, so the four scan fields are
|
|
//--- always put back by the single restore below the loop - m_barrierScanLiveLabels left true
|
|
//--- makes ComputeLabelForBar read the LAST CANDIDATE's multiples as if they were the configured
|
|
//--- geometry, which would silently relabel on a pairing nothing chose.
|
|
if(ShutdownRequested())
|
|
break;
|
|
m_barrierScanSlMult = candSl[c];
|
|
m_barrierScanTpMult = candTp[c];
|
|
m_barrierHorizonBars = ComputeBarrierHorizonBars(barsNow);
|
|
bool clamped = m_barrierHorizonClamped;
|
|
m_barrierScanTimeouts = 0;
|
|
double gc[];
|
|
int gl[];
|
|
int gn = BuildMiSample(gc, gl);
|
|
if(gn < MI_MIN_SAMPLES)
|
|
continue;
|
|
double obs = ScoreMiSample(gc, gl, gn, false);
|
|
//--- Class shares of THIS geometry's label, so a geometry that scores well by having almost
|
|
//--- nothing left to predict is visible as such instead of winning quietly.
|
|
int cB = 0, cS = 0;
|
|
for(int q = 0; q < gn; q++)
|
|
{
|
|
if(gl[q] == 0)
|
|
cB++;
|
|
else
|
|
if(gl[q] == 1)
|
|
cS++;
|
|
}
|
|
double dirShare = 100.0 * (cB + cS) / gn;
|
|
double timeoutShare = 100.0 * m_barrierScanTimeouts / gn;
|
|
//--- The MIN REWARD:RISK test that used to gate enrolment here is GONE (2026-08-09) along with
|
|
//--- Min_Risk_Reward_Ratio itself. Its purpose was to stop the scan crowning a geometry the live
|
|
//--- rejection filter would then throw every setup away at - but with no rejection filter there
|
|
//--- is nothing to collide with, and excluding low-ratio pairings was excluding them on a rule
|
|
//--- rather than on the measurement. CLAMPING remains disqualifying, and for an unrelated reason
|
|
//--- that still holds: a clamped label describes a trade truncated by the horizon rather than
|
|
//--- resolved at SL/TP, so it is not the target the EA would actually hold to.
|
|
bool eligible = !clamped;
|
|
//--- These draws now serve two purposes. Per candidate they still centre the printed score. Across
|
|
//--- candidates they form the null of the maximum, which is the only thing that can say whether the
|
|
//--- WINNER is real - so they are retained rather than reduced to a mean and discarded.
|
|
double nullSum = 0.0;
|
|
int nd = 0;
|
|
for(int s = 0; s < MI_GEOMETRY_PERMUTATIONS; s++)
|
|
{
|
|
//--- Break only: the outer loop's check runs next and does the restore-and-return in one
|
|
//--- place. This candidate's partial draws are discarded with everything else, so a short
|
|
//--- null can never reach the family-wise gate below.
|
|
if(ShutdownRequested())
|
|
break;
|
|
double sc = ScoreMiSample(gc, gl, gn, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
nullSum += sc;
|
|
if(eligible && candidates < MI_GEOMETRY_MAX_CANDIDATES)
|
|
drawMat[candidates][nd] = sc;
|
|
nd++;
|
|
}
|
|
if(eligible && candidates < MI_GEOMETRY_MAX_CANDIDATES)
|
|
{
|
|
drawCount[candidates] = nd;
|
|
candidates++;
|
|
}
|
|
double nullMean = (nd > 0) ? nullSum / nd : -1.0;
|
|
double excess = (nullMean >= 0.0) ? (obs - nullMean) : 0.0;
|
|
//--- Base rate m/(m+k) IS the break-even win rate at this geometry - print it so the ranking is
|
|
//--- read next to the bar the model would have to clear, not in isolation.
|
|
double breakeven = 100.0 * candSl[c] / (candSl[c] + candTp[c]);
|
|
//--- Ranking is now purely the measurement: every unclamped pairing competes, whatever its
|
|
//--- reward:risk. Note the break-even printed alongside is what makes a low-ratio winner
|
|
//--- readable rather than alarming - 1:1 needs 50% precision where 1:3 needs 25%, and the
|
|
//--- scan's value column is already excess information over that geometry's OWN null, so the
|
|
//--- comparison across pairings is like-for-like.
|
|
string name = candIsCfg[c] ? StringFormat("CFG %.2f:%.2f", candSl[c], candTp[c])
|
|
: StringFormat("%.0f:%.0f", candSl[c], candTp[c]);
|
|
rows += StringFormat("%s%s(h%d%s,be%.0f%%,dir%.0f%%,to%.0f%%)=%+.5f", (rows == "" ? "" : " "),
|
|
name, m_barrierHorizonBars, (clamped ? "!" : ""), breakeven,
|
|
dirShare, timeoutShare, excess);
|
|
//--- Only unclamped, tradeable geometries are eligible to WIN - see the header. The rest are
|
|
//--- still printed, so a disqualification is visible rather than a silent omission.
|
|
//--- The CONFIGURED pair is scored for comparison but never crowned: it is already in force, and
|
|
//--- its continuous multiples are not an enum pairing this scan could adopt anyway.
|
|
if(eligible && !candIsCfg[c] && excess > bestExcess)
|
|
{
|
|
bestExcess = excess;
|
|
bestName = name;
|
|
//--- The grid values ARE the enum values (SL_ATR_x2 == 2, TP_ATR_x8 == 8), so the winning
|
|
//--- pairing can be adopted directly with no lookup table to drift out of step.
|
|
bestSl = (int)candSl[c];
|
|
bestTp = (int)candTp[c];
|
|
bestHorizon = m_barrierHorizonBars;
|
|
}
|
|
if(candIsCfg[c])
|
|
cfgExcess = excess;
|
|
}
|
|
}
|
|
m_barrierScanLiveLabels = false;
|
|
m_barrierScanSlMult = 0.0;
|
|
m_barrierScanTpMult = 0.0;
|
|
m_barrierHorizonBars = savedHorizon;
|
|
//--- THE ONE PLACE an abandoned scan leaves from, after the restore above and before anything reads what
|
|
//--- it collected. A stop can truncate the last candidate's null, and fwDraws below is the MINIMUM draw
|
|
//--- count across candidates - so one short candidate shrinks the null of the maximum for all of them and
|
|
//--- would let a pairing be ADOPTED on a null that never finished. Nothing here is worth that, and
|
|
//--- nothing here is worth the teardown budget either.
|
|
if(ShutdownRequested())
|
|
{
|
|
PrintFormat("%s: barrier-geometry scan ABANDONED - stop requested. Scan state restored; the geometry"
|
|
" in force is unchanged.", ID);
|
|
return;
|
|
}
|
|
Print(ID + StringFormat(": barrier-geometry scan (SL:TP; h=horizon, '!'=CLAMPED and disqualified - a clamped "
|
|
"label truncates a trade the EA would hold to SL/TP; be=break-even win rate, "
|
|
"dir=%%bars with a tradeable direction, to=%%timed out; value is entry-time "
|
|
"information in nats above that geometry's own null; 'CFG' is the geometry "
|
|
"actually in force, scored as a peer but never crowned) - %s | configured "
|
|
"%.2f:%.2f scores %+.5f, best eligible GRID pairing is %s at %+.5f (%.1fs)",
|
|
rows, cfgSl, cfgTp, cfgExcess, (bestName == "" ? "none" : bestName), bestExcess,
|
|
(GetTickCount() - t0) / 1000.0));
|
|
//--- FAMILY-WISE GATE. bestExcess is a MAXIMUM over the eligible candidates, and the maximum of several
|
|
//--- draws from a null sits well above any single draw from it - so testing the winner against its own
|
|
//--- null asks the wrong question and will crown a winner on pure noise almost every time. What follows
|
|
//--- rebuilds the null OF THE MAXIMUM: take one permutation draw from every candidate, keep the largest,
|
|
//--- repeat. bestExcess then has to beat that distribution, not a single-candidate one.
|
|
//---
|
|
//--- The draws are centred LEAVE-ONE-OUT so the comparison is like for like: the observed score is
|
|
//--- centred by draws that do not contain it, so each draw must be too. Centring a draw by a mean that
|
|
//--- includes it shrinks it toward zero, which would deflate the null and let the winner through.
|
|
//---
|
|
//--- Draws are independent across candidates here while the real ones are correlated (the candidates
|
|
//--- share features and heavily overlapping label windows). Independence makes the maximum MORE spread
|
|
//--- out than the truth, so the gate errs toward rejecting - the safe direction when passing costs a
|
|
//--- full relabel and retrain of every topology.
|
|
double pFamily = 1.0;
|
|
int fwDraws = 0;
|
|
double nullMaxSum = 0.0;
|
|
int nullMaxCount = 0;
|
|
if(candidates > 0)
|
|
{
|
|
fwDraws = MI_GEOMETRY_PERMUTATIONS;
|
|
for(int c = 0; c < candidates; c++)
|
|
fwDraws = (int)MathMin(fwDraws, drawCount[c]);
|
|
int atLeast = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
{
|
|
double worst = -DBL_MAX;
|
|
for(int c = 0; c < candidates; c++)
|
|
{
|
|
if(drawCount[c] < 2)
|
|
continue;
|
|
double sum = 0.0;
|
|
for(int q = 0; q < drawCount[c]; q++)
|
|
sum += drawMat[c][q];
|
|
double loo = (sum - drawMat[c][s]) / (drawCount[c] - 1);
|
|
double e = drawMat[c][s] - loo;
|
|
if(e > worst)
|
|
worst = e;
|
|
}
|
|
if(worst > -DBL_MAX && worst >= bestExcess)
|
|
atLeast++;
|
|
//--- WINNER'S-CURSE PENALTY, accumulated from the same draws the p-value uses. `worst` is the
|
|
//--- MAXIMUM excess over all candidates in a pure-noise draw, so its mean across draws is
|
|
//--- exactly what a best-of-K selection is expected to report when there is nothing there.
|
|
//--- Measured rather than assumed: this is the empirical form of the sqrt(2 ln K) x SE penalty
|
|
//--- the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it
|
|
//--- needs no normality assumption because the permutation draws ARE the null distribution.
|
|
if(worst > -DBL_MAX)
|
|
{
|
|
nullMaxSum += worst;
|
|
nullMaxCount++;
|
|
}
|
|
}
|
|
pFamily = (fwDraws > 0) ? (double)(1 + atLeast) / (fwDraws + 1) : 1.0;
|
|
}
|
|
//--- The winner's excess, SHRUNK toward zero by that penalty. A best-of-K maximum is biased upward by
|
|
//--- construction - it is the largest of K noisy draws - so the raw number overstates what adopting it
|
|
//--- will actually deliver, and it is the number the adoption message quotes and compares against the
|
|
//--- incumbent. Clearing the family-wise gate says the ranking is real; it does NOT say the effect is
|
|
//--- as large as the top row reads. James-Stein form: effect x max(0, 1 - penalty^2/effect^2), which
|
|
//--- leaves a large effect nearly untouched and collapses a marginal one to ~0.
|
|
double nullMaxMean = (nullMaxCount > 0) ? nullMaxSum / nullMaxCount : 0.0;
|
|
double shrunkExcess = bestExcess;
|
|
if(bestExcess > 0.0 && nullMaxMean > 0.0)
|
|
shrunkExcess = bestExcess * MathMax(0.0, 1.0 - (nullMaxMean * nullMaxMean) / (bestExcess * bestExcess));
|
|
bool winnerReal = (bestName != "" && bestExcess > 0.0 && fwDraws > 0 && pFamily <= MI_GEOMETRY_ALPHA);
|
|
Print(ID + StringFormat(": barrier-geometry winner test - %s at %+.5f is the best of %d ELIGIBLE "
|
|
"candidates, so it is tested against the null of the maximum over %d, not its "
|
|
"own: p=%.4f over %d draws (need <=%.2f). SHRUNK effect %+.5f (a best-of-K maximum is "
|
|
"biased upward by construction; the winner's-curse penalty here is the MEAN "
|
|
"noise-draw maximum %+.5f, measured on these same draws - plan on the shrunk "
|
|
"number, not the raw one). %s", (bestName == "" ? "none" : bestName),
|
|
bestExcess, candidates, candidates, pFamily, fwDraws, MI_GEOMETRY_ALPHA, shrunkExcess, nullMaxMean,
|
|
(winnerReal ? "CLEARS - the ranking is not selection noise."
|
|
: "DOES NOT CLEAR - a max this large happens routinely when every candidate is "
|
|
"pure noise, so the ranking carries no information and the top row is not a "
|
|
"finding. Change nothing.")));
|
|
//--- ADOPT, don't advise. SL_Mode/TP_Mode stopped being inputs on 2026-08-07, so this scan is now the
|
|
//--- thing that chooses the barrier - which is exactly why the family-wise gate above had to exist
|
|
//--- first. Three conditions, all necessary:
|
|
//--- winnerReal - it beat the null of the MAXIMUM, not merely the incumbent and not merely zero.
|
|
//--- m_eraCount==0 - relabelling a partly-trained net would move the target out from under weights
|
|
//--- already fitted to the old one. Same gate the indicator tuner uses.
|
|
//--- != current - nothing to do when the measurement agrees with the default.
|
|
//--- A model that already exists never reaches here with anything to change: its geometry is pinned in
|
|
//--- the .cfg and adopted at load, so the pairing a run trains on is the pairing it keeps.
|
|
//--- DETECTABILITY GUARD ON THE ADOPTION, and it exists because making this scan authoritative
|
|
//--- (2026-08-17) put two objectives in charge of one decision without reconciling them. The scan
|
|
//--- maximises entry-time INFORMATION in nats. The deploy gate needs enough INDEPENDENT observations
|
|
//--- to certify an edge. Those are not the same quantity and they pull opposite ways: a wider pairing
|
|
//--- carries more information per call AND takes longer to resolve, and overlapping labels are worth
|
|
//--- ~1/L each, so tripling the horizon divides the independent sample by ~3 and multiplies the SE the
|
|
//--- gate must beat by ~sqrt(3). USDJPY's winner today is 2:8 at h192 against an incumbent 1.61:3.21
|
|
//--- at h64 - exactly that trade, made silently.
|
|
//---
|
|
//--- Conservative by construction: the horizon is an UPPER bound on the mean label lifespan, so
|
|
//--- oosBars/horizon is a LOWER bound on the independent observations available. If even that bound
|
|
//--- cannot support a generous edge, no model trained on this pairing could ever be certified on this
|
|
//--- window - which makes adopting it a guaranteed dead end rather than a promising direction.
|
|
bool winnerDetectable = true;
|
|
double needIndep = 0.0, haveIndep = 0.0;
|
|
if(bestSl > 0 && bestTp > 0 && bestHorizon > 0)
|
|
{
|
|
double p = (double)bestSl / ((double)bestSl + (double)bestTp); // break-even at this pairing
|
|
double d = ADOPT_MIN_DETECTABLE_EDGE; // the edge we would deploy on
|
|
needIndep = (EDGE_MIN_SIGMAS * EDGE_MIN_SIGMAS) * p * (1.0 - p) / (d * d);
|
|
haveIndep = (double)barsNow * (MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0) / (double)bestHorizon;
|
|
winnerDetectable = (haveIndep >= needIndep);
|
|
}
|
|
if(winnerReal && !winnerDetectable && m_eraCount == 0)
|
|
Print(ID + StringFormat(": barrier-geometry winner %s NOT ADOPTED - it wins on information (%+.5f"
|
|
" nats) and loses on DETECTABILITY. At break-even %.0f%% it needs %.0f"
|
|
" independent calls to certify a %.0fpp edge, and its %d-bar horizon leaves"
|
|
" this OOS window at most %.0f - so no win rate a model could reach would"
|
|
" ever clear the deploy gate on this pairing. Keeping the incumbent. More"
|
|
" information per trade is worth nothing if it buys too few independent"
|
|
" trades to prove: WIDTH is not free, and this is the constraint that"
|
|
" decides, not the nats.",
|
|
bestName, bestExcess,
|
|
100.0 * (double)bestSl / ((double)bestSl + (double)bestTp),
|
|
needIndep, 100.0 * ADOPT_MIN_DETECTABLE_EDGE, bestHorizon, haveIndep));
|
|
if(winnerReal && winnerDetectable && m_eraCount == 0 && bestSl > 0 && bestTp > 0
|
|
&& (bestSl != m_sl_mode || bestTp != m_tp_mode))
|
|
{
|
|
Print(ID + StringFormat(": adopting barrier geometry %s - it carries %+.5f nats of entry-time "
|
|
"information against the configured %.0f:%.0f's %+.5f, and cleared the "
|
|
"family-wise gate. Relabelling and training on it. Chance precision equals "
|
|
"break-even at EVERY geometry, so this does not hand us expectancy; it puts "
|
|
"more of the answer inside the features' reach, which is the one thing no "
|
|
"change of topology can do.", bestName, bestExcess, cfgSl, cfgTp, cfgExcess));
|
|
//--- AND INTO THE PAIR THAT ACTUALLY GOVERNS, which the mode ints have not been since the derived
|
|
//--- geometry landed. BarrierMultiples() ranks DERIVED above the modes, so on any model carrying a
|
|
//--- derived pair - which is every model with a .cfg, including a fresh one whose weights are gone
|
|
//--- but whose sidecar survived - setting the modes alone changed NOTHING. The scan announced
|
|
//--- "adopting 2:8 ... relabelling and training on it" and the very next line labelled at
|
|
//--- 1.61:3.21 (USDJPY, 2026-08-17 14:24:12.844 vs .979). Two measurements choosing the same thing,
|
|
//--- one of them silently inert, and a log line that stated the opposite of what happened.
|
|
//--- ONE AUTHORITY: the derived pair. It is what the labels read, what the deploy gate certifies,
|
|
//--- what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The
|
|
//--- mode ints are legacy (SL_Mode/TP_Mode stopped being inputs on 2026-08-07) and are kept in step
|
|
//--- above only so the fallback path and the fingerprint stay coherent.
|
|
ApplyAdoptedGeometry((double)bestSl, (double)bestTp, bestSl, bestTp);
|
|
//--- PUBLISH IT TO THE CHART. The MI chain that ends in this scan runs once per chart, so the other
|
|
//--- three members never measure this and would otherwise keep labelling on their own derived pair
|
|
//--- while this one relabels - four members, two targets, one averaged vote. See the measurement in
|
|
//--- g_ensembleChartGeomAdopted's comment.
|
|
if(m_ensembleMember)
|
|
{
|
|
g_ensembleChartGeomAdopted = true;
|
|
g_ensembleChartGeomSl = m_derivedSlMult;
|
|
g_ensembleChartGeomTp = m_derivedTpMult;
|
|
g_ensembleChartGeomSlMode = m_sl_mode;
|
|
g_ensembleChartGeomTpMode = m_tp_mode;
|
|
}
|
|
}
|
|
else
|
|
if(winnerReal && m_eraCount > 0 && bestSl > 0 && (bestSl != m_sl_mode || bestTp != m_tp_mode))
|
|
Print(ID + ": barrier-geometry scan prefers " + bestName + ", but this model is already trained "
|
|
"(era " + IntegerToString(m_eraCount) + "). Its geometry is pinned to what it learned; "
|
|
"delete the weights if you want it re-measured.");
|
|
else
|
|
if(bestName == "")
|
|
Print(ID + ": barrier-geometry scan - every geometry with a long enough horizon was "
|
|
"disqualified or scored at zero. Nothing here to switch to; the limit is the feature "
|
|
"set, not the target.");
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Outer loop around Train(). Tuning is now a one-shot filter pass |
|
|
//| that runs BEFORE the first era and costs seconds, so this is a |
|
|
//| straight pass-through to Train() on every later call. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| See the declaration. Every side effect an adopted geometry must |
|
|
//| carry, in ONE place, because it has two callers: the member whose |
|
|
//| scan chose the pairing, and the siblings that learn it from |
|
|
//| g_ensembleChartGeomAdopted without ever running a scan. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ApplyAdoptedGeometry(double sl, double tp, int slMode, int tpMode)
|
|
{
|
|
//--- Legacy mode ints first. SL_Mode/TP_Mode stopped being inputs on 2026-08-07 and no longer decide
|
|
//--- anything, but the fallback path in BarrierMultiples and the config fingerprint still read them, so
|
|
//--- leaving them behind the derived pair is how the two disagree.
|
|
m_sl_mode = slMode;
|
|
m_tp_mode = tpMode;
|
|
//--- Same floor DeriveBarrierGeometry applies, for the same reason: OpenParams widens any stop tighter
|
|
//--- than this, and a live stop wider than the labelled one grades the model on a bet it is not placing.
|
|
if(sl < MIN_SL_ATR_MULTIPLIER)
|
|
sl = MIN_SL_ATR_MULTIPLIER;
|
|
m_derivedSlMult = sl;
|
|
m_derivedTpMult = tp;
|
|
m_geometryDerived = true;
|
|
//--- LATCHED, or the very next label prebuild undoes this. DeriveBarrierGeometry re-measures the scale
|
|
//--- from the excursion quantiles on every era-0 pass and overwrites the pair - and this sets
|
|
//--- m_labelCachePrebuilt = false below, which is exactly what triggers that pass. Without the latch the
|
|
//--- sequence is: adopt 2:8, invalidate the cache, re-derive 1.61:3.21, train on 1.61:3.21, and print
|
|
//--- that it adopted 2:8.
|
|
m_geometryAdopted = true;
|
|
//--- Republished immediately, not at the next era end: between here and there the label cache is rebuilt
|
|
//--- under the adopted pair, and a live order placed in that window would otherwise carry the superseded
|
|
//--- geometry.
|
|
g_DerivedSlAtrMult = m_derivedSlMult;
|
|
g_DerivedTpAtrMult = m_derivedTpMult;
|
|
//--- The sidecar holds the OLD pair; force it to be rewritten so a restart resumes on the adopted one
|
|
//--- rather than silently reverting to what this just replaced.
|
|
m_geometryCfgSaved = false;
|
|
Print(ID + StringFormat(": geometry authority - stop %.2f*ATR / target %.2f*ATR is now the pair the"
|
|
" labels, the deploy gate and the live order ALL read. The SL_Mode/TP_Mode"
|
|
" ints are kept in step but no longer decide anything.",
|
|
m_derivedSlMult, m_derivedTpMult));
|
|
//--- The cache holds labels computed under the OLD barriers, so it has to be discarded rather than
|
|
//--- appended to - Train()'s !m_labelCachePrebuilt gate then rebuilds it under the adopted pair before
|
|
//--- era 0 starts.
|
|
m_labelCachePrebuilt = false;
|
|
ArrayInitialize(m_labelCacheHasValue, false);
|
|
//--- AND UNLATCH THE HORIZON, which is otherwise resolved once per process and held. Adopting a wider
|
|
//--- target without this labels the new geometry against the OLD ceiling - 2:8 wants ~192 bars and would
|
|
//--- silently get 2:6's 128 - which is precisely the truncation that made every model learn "target
|
|
//--- within 128 bars" while the EA holds to SL/TP (fixed 2026-08-01 in 168422f). The truncation lands in
|
|
//--- Neutral, not in the timeout counter that watches for it, so it does not announce itself.
|
|
//--- EnsureBarrierHorizon() re-derives and re-logs on the next call.
|
|
m_barrierHorizonResolved = false;
|
|
}
|
|
void CExpertSignalAIBase::TuneIndicatorsAndTrain(datetime StartTrainBar = 0)
|
|
{
|
|
//--- FIRST STATEMENT IN THE WHOLE TRAINING ENTRY POINT, ahead of every latch below it (m_tuneFilterDone,
|
|
//--- g_ensembleChartTuneDone) so a stop cannot mark a sweep as "already run" without running it. The
|
|
//--- individual scans yield on ShutdownRequested() as well; this simply refuses to start the chain.
|
|
if(ShutdownRequested())
|
|
return;
|
|
//--- Publish the caller's window anchor so StartLabelCachePrebuild() sizes its window with the SAME
|
|
//--- expression Train() uses. This assignment existed before the God-class split and was dropped by
|
|
//--- it, leaving m_tuneStartTrainBar permanently 0 - harmless while nothing read it, and a real defect
|
|
//--- the moment 0c85c54 made the prebuild reset dtStudied from it: Train() then computed
|
|
//--- max(StartTrainBar, windowFloor) while the prebuild computed max(0, windowFloor), so the two
|
|
//--- disagreed about `bars`, EnsureBarCachesCapacity() saw a changed size every era start, and each
|
|
//--- era immediately re-armed a full 38k-bar prebuild instead of training.
|
|
m_tuneStartTrainBar = StartTrainBar;
|
|
bool anyTunable = (m_useADCumulativeDelta || m_useADShorteningOfThrust || m_useADWyckoffEventStream ||
|
|
m_useADWyckoffFailedStructure || m_useADWyckoffSignificantBarInversion ||
|
|
m_useMA || m_useRSI || m_useMACD || m_useIchimoku);
|
|
//--- Tune once per fresh model, before any weight has been trained. Gated on m_labelCachePrebuilt
|
|
//--- because the score needs labels, and on era 0 because re-tuning a partly-trained network would
|
|
//--- change its inputs out from under weights already fitted to the old ones.
|
|
if(m_autoTuneIndicators && anyTunable && !m_tuneFilterDone && m_labelCachePrebuilt && m_eraCount == 0)
|
|
{
|
|
m_tuneFilterDone = true;
|
|
if(m_ensembleMember && g_ensembleChartTuneDone)
|
|
{
|
|
//--- Another member on this chart already ran the identical sweep - apply its outcome instead
|
|
//--- of recomputing it (see g_ensembleChartTuneDone at the top of this file).
|
|
//--- SAY IT ON THE PANEL, not only in the journal. Until now this branch printed and published
|
|
//--- nothing, so an ensemble warm-up showed ONE row reading "scoring indicator settings..." for
|
|
//--- a minute while the other three sat on a stale line - which reads as three members doing
|
|
//--- nothing, rather than three members correctly declining to redo a chart-wide measurement.
|
|
PublishStatus(ID + " : adopting the chart's tuned indicators...");
|
|
//--- THE PARAMETERS are adopted only when a winner was installed...
|
|
if(g_ensembleChartTuneInstalled)
|
|
m_indicatorTuner.Unflatten(g_ensembleChartTuneSettings);
|
|
//--- ...but the HANDLES must be rebuilt EITHER WAY, and that is not a tidiness point - it is the
|
|
//--- cause of the "silent block failure" that cost six sessions. MEASURED 2026-08-17, twice:
|
|
//--- USDJPY 18:10:03 PAI sweeps -> 18:10:29.86/.91/.95 CONV/LSTM/HYB take this branch
|
|
//--- (winner REJECTED, so the old code re-inited NOTHING)
|
|
//--- -> 18:10:30.06/.07/.07 all three: MA=-1(h13), every sweep bar rejected
|
|
//--- XAUUSD 18:10:11 -> 18:10:42.19/.23/.27 -> 18:10:42.33 - identical, same 100 ms gap
|
|
//--- ALL TWELVE dead handles across both charts and all four members reported the SAME NUMBER,
|
|
//--- h13. That is the whole finding: it was never four handles, it was ONE. MT5 refcounts
|
|
//--- indicator requests, so four members asking for the same iMA on the same symbol/period get
|
|
//--- one shared handle - and TuneIndicatorsByFilter creates and drops ~35 of them scoring
|
|
//--- candidates. When it restores the configured settings the runner ends up holding a live
|
|
//--- handle while its siblings still hold the number the terminal has already freed.
|
|
//--- The old guard made re-initialising conditional on the winner being INSTALLED, which is
|
|
//--- exactly backwards: a REJECTED sweep churns just as many handles, and every one of the
|
|
//--- twelve recoveries above followed a rejection. Four handle creations per chart is nothing.
|
|
ReInitADIndicators(m_indicatorsPtr);
|
|
RefreshData();
|
|
Print(ID + ": indicator auto-tune already ran on this chart - same indicators, same features, "
|
|
"same labels, same answer. " +
|
|
(g_ensembleChartTuneInstalled
|
|
? "Adopting the installed winner so every member trains on the same feature vector."
|
|
: "Keeping the configured settings (the sweep's winner was rejected by the selection gate).") +
|
|
" The first member's auto-tune report above is this model's too.");
|
|
}
|
|
else
|
|
{
|
|
//--- Names the SCOPE, because the scope is what the other rows' silence means. This sweep is one
|
|
//--- measurement of one chart's indicators against one chart's labels, so exactly one member runs
|
|
//--- it and the rest adopt the answer - saying "for the whole chart" here is what tells the
|
|
//--- operator that three idle rows are the design and not a stall.
|
|
PublishStatus(ID + (m_ensembleMember
|
|
? " : scoring indicator settings for the whole chart..."
|
|
: " : scoring indicator settings..."));
|
|
//--- Snapshot the configured settings first: "did the sweep install?" is answered by comparing
|
|
//--- against the final settings, since a rejected winner is restored to exactly these values.
|
|
double tuneCfgBefore[];
|
|
m_indicatorTuner.Flatten(tuneCfgBefore);
|
|
TuneIndicatorsByFilter();
|
|
if(m_ensembleMember)
|
|
{
|
|
m_indicatorTuner.Flatten(g_ensembleChartTuneSettings);
|
|
g_ensembleChartTuneInstalled = false;
|
|
for(int tp = 0; tp < ArraySize(tuneCfgBefore); tp++)
|
|
if(g_ensembleChartTuneSettings[tp] != tuneCfgBefore[tp])
|
|
{
|
|
g_ensembleChartTuneInstalled = true;
|
|
break;
|
|
}
|
|
g_ensembleChartTuneDone = true;
|
|
//--- The sweep ends in ReportFeatureLabelInformation(), so the chart-level MI report is
|
|
//--- done too - mark it, or every other member would rerun the ~200-draw nulls the MI
|
|
//--- gate below exists to save.
|
|
if(m_miReportDone)
|
|
g_ensembleChartMiReportDone = true;
|
|
}
|
|
}
|
|
//--- the winning parameters change the input vector, so the network must start from scratch on it
|
|
BuildFreshTopology();
|
|
}
|
|
//--- The DIAGNOSTIC half runs even when the sweep does not: on a resumed model, on one whose tuner is
|
|
//--- switched off, and on one with nothing tunable. It reads the cached features and writes nothing,
|
|
//--- so none of the reasons the sweep is gated apply to it - and tying it to that gate meant the only
|
|
//--- way to see the answer on a running model was to delete the model.
|
|
//---
|
|
//--- THAT INTENT WAS NOT ACHIEVED UNTIL 2026-08-07. Moving the diagnostic out of the tuner's gate
|
|
//--- left it behind m_labelCachePrebuilt, which has exactly the same effect: the eager label pre-scan
|
|
//--- runs only on a FRESH start, because a resumed net labels lazily per bar (see the "skipped
|
|
//--- entirely when a trained net was loaded from disk" note in the prebuild). So on a resumed model
|
|
//--- the flag is false forever and the entire MI block - headline, positive control, alignment scan,
|
|
//--- lag profile, geometry scan, winner test - silently never ran. Measured on SP500 H1 2026-08-07:
|
|
//--- attached at era 271, still nothing by era 314, and every diagnostic captured on 08-05/06 came
|
|
//--- immediately after a weights reset. The only way to see the answer was still to delete the model.
|
|
//---
|
|
//--- So drive the prebuild ourselves when it is the only thing missing. It is safe on a trained net:
|
|
//--- its one fresh-net side effect, pushing the output-layer bias toward the dominant class, is
|
|
//--- already gated on m_eraCount == 0, and the scan itself only fills label caches. Train()'s own
|
|
//--- m_labelPrebuildActive gate advances it to completion, so this costs one short deferral (~1s at
|
|
//--- 38k bars) on the first attach and nothing afterwards.
|
|
//---
|
|
//--- NOT sampled from the lazily-filled cache instead: BuildMiSample skips bars that carry no cached
|
|
//--- label, so on a resumed model it would quietly score whichever subset training happened to have
|
|
//--- visited. That is a biased subsample presented as a measurement - the failure mode this whole
|
|
//--- diagnostic exists to catch.
|
|
else if(!m_miReportDone && !m_labelCachePrebuilt && !m_labelPrebuildActive)
|
|
{
|
|
//--- Announce only on a start that actually took. StartLabelCachePrebuild() returns without arming
|
|
//--- if the buffers/history are not ready yet and is simply retried on the next call, so printing
|
|
//--- unconditionally would repeat the line once per bar event until it succeeds.
|
|
StartLabelCachePrebuild();
|
|
//--- Says WHICH case this is rather than asserting the resumed one. The first version claimed
|
|
//--- "resumed from disk" unconditionally, and then printed it above a "seeding era 0" line on a
|
|
//--- brand-new model - the branch fires whenever the cache is not built, which is equally true
|
|
//--- before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse
|
|
//--- than one that says nothing, because it gets quoted back as evidence.
|
|
if(m_labelPrebuildActive)
|
|
Print(ID + (m_modelLoadedFromDisk
|
|
? ": MI diagnostics need a complete label cache and this model resumed from disk "
|
|
"(labels are filled lazily, so the cache covers only the bars training has "
|
|
"visited) - running the one-time pre-scan now, then the report. Training resumes "
|
|
"where it left off."
|
|
: ": MI diagnostics need a complete label cache and this model has not built one yet "
|
|
"- running the pre-scan now, then the report."));
|
|
}
|
|
else if(!m_miReportDone && m_labelCachePrebuilt)
|
|
{
|
|
//--- WAIT FOR THE CROSS-ASSET PANEL. It is part of the feature vector but it is built inside
|
|
//--- Train(), so on a fresh run this diagnostic would otherwise describe a NARROWER vector than
|
|
//--- the one training goes on to use. Observed 2026-08-02 on SP500 H1: the MI report, the
|
|
//--- alignment scan and the barrier-geometry scan all ran at 00:41:25, while the panel first
|
|
//--- built successfully at 01:12:47 - so every number they printed, including the geometry scan
|
|
//--- that is supposed to CHOOSE the training target, was measured on a feature set training
|
|
//--- never saw. Train() rebuilds the panel each era, so simply deferring lands the report on an
|
|
//--- era where the vector is complete.
|
|
//--- Never wait forever: a terminal that cannot sync the reference symbols (the tester loads
|
|
//--- auxiliary symbols from the terminal, not the server) must still get its diagnostics, with
|
|
//--- the gap stated rather than hidden.
|
|
if(m_ensembleMember && g_ensembleChartMiReportDone)
|
|
{
|
|
//--- see g_ensembleChartMiReportDone at the top of this file
|
|
m_miReportDone = true;
|
|
//--- Same reasoning as the tuner's adopt branch above: published, not just printed, so the row
|
|
//--- says why it is not repeating the measurement.
|
|
PublishStatus(ID + " : reusing the chart's information report...");
|
|
Print(ID + ": MI diagnostics already measured by another ensemble member on this chart - "
|
|
"same features, same labels, same answer. Skipped (saves the slowest part of the "
|
|
"ensemble's warm-up; the first member's report above is this model's too).");
|
|
//--- ...BUT THE GEOMETRY IS NOT A REPORT, IT IS A DECISION, and skipping the chain that makes it
|
|
//--- is not the same as declining it. "Same features, same labels, same answer" was true of the
|
|
//--- MI numbers and FALSE of the barrier the donor adopted: it relabelled itself and left the
|
|
//--- other three on their own derived pair, so the ensemble trained on two different targets and
|
|
//--- the joint gate certified the average as if it were one. Take the donor's pairing here, on
|
|
//--- the same reasoning the indicator settings are taken above - members training on divergent
|
|
//--- targets are not an ensemble.
|
|
//--- Guarded on era 0 for the same reason the donor's own adoption is: relabelling a partly
|
|
//--- trained net moves the target out from under weights already fitted to the old one.
|
|
if(g_ensembleChartGeomAdopted && m_eraCount == 0 && g_ensembleChartGeomSl > 0.0
|
|
&& g_ensembleChartGeomTp > 0.0
|
|
&& (m_derivedSlMult != g_ensembleChartGeomSl || m_derivedTpMult != g_ensembleChartGeomTp))
|
|
{
|
|
PrintFormat("%s: adopting the barrier geometry the chart's scan chose - %.2f*ATR / %.2f*ATR."
|
|
" This member never ran the scan (the MI chain runs once per chart), and keeping"
|
|
" its own derived pair would put this ensemble's members on DIFFERENT targets"
|
|
" while the orchestrator averages their votes as one.",
|
|
ID, g_ensembleChartGeomSl, g_ensembleChartGeomTp);
|
|
ApplyAdoptedGeometry(g_ensembleChartGeomSl, g_ensembleChartGeomTp,
|
|
g_ensembleChartGeomSlMode, g_ensembleChartGeomTpMode);
|
|
}
|
|
}
|
|
else
|
|
if(m_crossAsset.IsReady() || m_miReportDeferrals >= MI_REPORT_MAX_DEFERRALS)
|
|
{
|
|
//--- THE LONGEST SINGLE STRETCH OF THE WARM-UP - the MI suite, the lag profile, the excursion
|
|
//--- targets and the geometry scan, each with its own few-hundred-draw permutation null - and
|
|
//--- until now it published NOTHING. The tuner announced itself and then handed over to a
|
|
//--- chain that runs many times longer in silence, so the panel's last word during most of
|
|
//--- the warm-up described a step that had already finished.
|
|
PublishStatus(ID + (m_ensembleMember
|
|
? " : measuring feature/label information for the whole chart..."
|
|
: " : measuring feature/label information..."));
|
|
ReportFeatureLabelInformation();
|
|
if(m_ensembleMember && m_miReportDone)
|
|
g_ensembleChartMiReportDone = true;
|
|
}
|
|
else
|
|
m_miReportDeferrals++;
|
|
}
|
|
Train(StartTrainBar);
|
|
}
|
|
#endif // WARRIOR_AIBASE_AUTOTUNE_MQH
|