forked from animatedread/Warrior_EA
1bf3eba68a
The user should not need a tester corpus run per symbol. Every pattern
condition in Signals\Signal{MA,RSI,MACD,Ichimoku}.mqh anchors its reads on
`int idx = StartIndex()` with zero hardcoded indices (verified), so a
name-hiding StartIndex override + EvalShift(i) on CExpertSignalCustom makes
the EXACT live ladder code answer "what would you have fired at bar i" -
the silent-divergence trap that justified the DB corpus does not exist on
this path, and neither do the GMT-offset ambiguity, the DB row caps, or
the wipe procedure.
- CExpertSignalCustom: m_evalShift + StartIndex()/EvalShift() +
SweepPrepare(bars) (deep-resizes the shared price series); the four
classic signal classes override SweepPrepare to deep-resize their own
indicator buffers.
- CSignalMETA::BuildCorpusBySweep: per bar x per source filter, run
Direction() shifted, harvest the per-side pattern slots + netVote into
the same corpus arrays the DB loader fills; entry=bar open so
MetaPrepareEra's resolution matches at offset +0 with zero price error.
DB corpus remains the fallback when classic filters are disabled.
- Warrior_EA.mq5: META gets the enabled classic filters as candidate
sources (family ids match the descriptor one-hot).
- UseDatabaseRanking default false -> true (user request): a META chart
journals + ranks out of the box.
Workflow per symbol is now: attach ONE chart with AIType=META (optionally
Meta_ExportDataset=true for the offline pool) - candidates, labels,
training and export all happen in place, ~10 seconds of sweep instead of a
tester run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
598 lines
29 KiB
MQL5
598 lines
29 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Warrior_EA |
|
|
//| AnimateDread |
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
#include "..\Expert\ExpertSignalAIBase.mqh"
|
|
#include "..\Expert\AIBase\MetaCorpus.mqh"
|
|
// wizard description start
|
|
//+------------------------------------------------------------------+
|
|
//| Description of the class |
|
|
//| Title=Signals of indicator 'Meta AI' |
|
|
//| Type=SignalAdvanced |
|
|
//| Name=Meta AI |
|
|
//| ShortName=META |
|
|
//| Class=CSignalMETA |
|
|
//| Page=signal_meta |
|
|
//+------------------------------------------------------------------+
|
|
// wizard description end
|
|
//+------------------------------------------------------------------+
|
|
//| Class CSignalMETA - stage S2 of Meta_Labeling_Design.md. |
|
|
//| |
|
|
//| The meta-labeling head: instead of asking "which way will the |
|
|
//| next bar go" (measured dead - direction-closed verdict), it asks |
|
|
//| "given that a SPECIFIC classic-pattern candidate just fired, |
|
|
//| will THAT trade reach its target before its stop, at the EA's |
|
|
//| own geometry, net of cost". One net for all 52 pattern-sides; |
|
|
//| pattern identity rides in as input features. |
|
|
//| |
|
|
//| Sample: the signal-DB corpus (built by an 18-year backtest |
|
|
//| with UseDatabaseRanking on - per-side journaling means |
|
|
//| the DB IS the candidate stream, uncensored). |
|
|
//| Label: triple-barrier win/loss of the candidate's own side |
|
|
//| from its fire bar - the side-conditional win caches |
|
|
//| the label prebuild already computes; the DB's stop- |
|
|
//| and-reverse outcome is NEVER reused as a label. |
|
|
//| Features: the shared BuildFeatureWindow() output plus a 31-wide |
|
|
//| setup descriptor appended at the input (26-slot |
|
|
//| pattern one-hot, side, tanh-squashed netVote, SL/TP |
|
|
//| in ATR, spread/ATR at fire time). Appended at the |
|
|
//| input rather than "at the head" because CNet has no |
|
|
//| concat layer; on the MLP front end the two are |
|
|
//| equivalent up to one linear layer. |
|
|
//| Head: 2 outputs, softmax+CE (== logistic/BCE); see the |
|
|
//| total==2 branches in AI\Impl\NetForward.mqh. |
|
|
//| Front end: MLP only in S2 (AddCustomLayers no-op inherited). |
|
|
//| Conv/LSTM meta variants would need the descriptor |
|
|
//| padded to whole pseudo-bars to keep their bar-major |
|
|
//| window/step geometry - deliberately out of S2 scope. |
|
|
//| |
|
|
//| S2 trains and reports (coverage x (win rate - break-even) vs the |
|
|
//| base-rate null, in Training.mqh's era-end META line). It casts |
|
|
//| NO votes: dPrevSignal never leaves its sentinel, so the base |
|
|
//| LongCondition/ShortCondition return 0. Live gating of fired |
|
|
//| candidates via the per-side hooks is S3. |
|
|
//+------------------------------------------------------------------+
|
|
|
|
//--- Setup descriptor layout (AppendCandidateFeatures): 26 one-hot + side + netVote + SL + TP + spread/ATR.
|
|
//--- MetaDescWidth() returns this and the input layer is sized with it - the three MUST stay in step.
|
|
#define META_ONE_HOT_SLOTS 26
|
|
#define META_DESC_FEATURES (META_ONE_HOT_SLOTS + 5)
|
|
|
|
class CSignalMETA : public CExpertSignalAIBase
|
|
{
|
|
protected:
|
|
//--- The corpus: every journaled pattern instance, loaded ONCE per attach from the largest signal
|
|
//--- DB on disk (see LoadMetaCorpus for why largest-by-rows rather than the chart's own config
|
|
//--- fingerprint). GMT times are fixed; bar INDICES are re-resolved every era (they shift).
|
|
datetime m_corpusGmt[];
|
|
char m_corpusSide[];
|
|
double m_corpusNetVote[];
|
|
short m_corpusFamily[];
|
|
short m_corpusPattern[];
|
|
double m_corpusEntry[]; // touchable price at fire time - the offset oracle
|
|
int m_corpusCount;
|
|
bool m_corpusLoaded;
|
|
bool m_prepareReported; // first-era diagnostics print loudly, later eras verbose
|
|
bool m_datasetExported; // one export per attach (Meta_ExportDataset input)
|
|
//--- ON-CHART CANDIDATE SOURCES (the classic filters attached to this same chart). When present,
|
|
//--- the corpus is generated by SWEEPING these real ladders over the chart's own history via the
|
|
//--- StartIndex/EvalShift mechanism - no tester corpus run, no DB dependency, no GMT-offset
|
|
//--- resolution ambiguity (the sweep IS on this chart's bars). The DB loader stays as fallback.
|
|
CExpertSignalCustom *m_srcFilter[4];
|
|
int m_srcFamily[4];
|
|
int m_srcCount;
|
|
|
|
public:
|
|
void AddCandidateSource(CExpertSignalCustom *filter, const int family)
|
|
{
|
|
if(m_srcCount < 4 && CheckPointer(filter) != POINTER_INVALID)
|
|
{
|
|
m_srcFilter[m_srcCount] = filter;
|
|
m_srcFamily[m_srcCount] = family;
|
|
m_srcCount++;
|
|
}
|
|
}
|
|
CSignalMETA(void);
|
|
virtual bool InitIndicators(CIndicators *indicators) override;
|
|
virtual int MetaDescWidth(void) const override { return META_DESC_FEATURES; }
|
|
virtual bool MetaPrepareEra(const int bars) override;
|
|
virtual void AppendCandidateFeatures(const int candId) override;
|
|
|
|
protected:
|
|
bool LoadMetaCorpus(void);
|
|
bool BuildCorpusBySweep(void);
|
|
long CountDbPatternRows(const int db);
|
|
void ExportMetaDataset(void);
|
|
int OneHotSlot(const int family, const int pattern) const
|
|
{
|
|
//--- MA 0-3, RSI 4-7, MACD 8-13, Ichimoku 14-25 - matches MetaFamilyPatterns' 4/4/6/12
|
|
int base = -1;
|
|
switch(family)
|
|
{
|
|
case 0: base = 0; break;
|
|
case 1: base = 4; break;
|
|
case 2: base = 8; break;
|
|
case 3: base = 14; break;
|
|
}
|
|
if(base < 0)
|
|
return -1;
|
|
int slot = base + pattern;
|
|
return (slot >= 0 && slot < META_ONE_HOT_SLOTS) ? slot : -1;
|
|
}
|
|
};
|
|
//+------------------------------------------------------------------+
|
|
//| Constructor |
|
|
//+------------------------------------------------------------------+
|
|
CSignalMETA::CSignalMETA(void) : m_corpusCount(0), m_corpusLoaded(false), m_prepareReported(false),
|
|
m_datasetExported(false), m_srcCount(0)
|
|
{
|
|
SetIdentity("Meta", "META");
|
|
//--- identity-defining, set once (feeds the |TGT:META fingerprint token and every IsMetaTarget seam)
|
|
m_trainTarget = 1;
|
|
for(int k = 0; k < 4; k++)
|
|
{
|
|
m_srcFilter[k] = NULL;
|
|
m_srcFamily[k] = -1;
|
|
}
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Create indicators and bootstrap/load the network. |
|
|
//+------------------------------------------------------------------+
|
|
bool CSignalMETA::InitIndicators(CIndicators *indicators)
|
|
{
|
|
return InitNeuralNetwork(indicators);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Total rows across the 52 pattern tables of an open DB handle. |
|
|
//+------------------------------------------------------------------+
|
|
long CSignalMETA::CountDbPatternRows(const int db)
|
|
{
|
|
long rows = 0;
|
|
for(int f = 0; f < 4; f++)
|
|
for(int p = 0; p < MetaFamilyPatterns(f); p++)
|
|
for(int s = 0; s < 2; s++)
|
|
{
|
|
string table = MetaFamilyName(f) + "_Pattern_" + IntegerToString(p) +
|
|
(s == 0 ? "_Buy" : "_Sell");
|
|
int stmt = DatabasePrepare(db, "SELECT COUNT(*) FROM " + table);
|
|
if(stmt == INVALID_HANDLE)
|
|
continue; // table absent (family disabled in the populating run)
|
|
long c = 0;
|
|
if(DatabaseRead(stmt))
|
|
DatabaseColumnLong(stmt, 0, c);
|
|
DatabaseFinalize(stmt);
|
|
rows += c;
|
|
}
|
|
return rows;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Load the candidate corpus from the LARGEST signal DB on disk. |
|
|
//| |
|
|
//| Deliberately NOT through dbm/the chart's own config fingerprint: |
|
|
//| the DB filename hashes the journaling inputs, so a training |
|
|
//| chart whose inputs differ by one journal setting from the |
|
|
//| corpus-building tester run would open a different (empty) file |
|
|
//| and silently train on nothing - the exact procedural trap that |
|
|
//| burned four corpus-build attempts. The corpus is DATA, not |
|
|
//| config identity; the biggest coherent set of candidates on disk |
|
|
//| is the right training set, and the pick is logged so the run is |
|
|
//| auditable. All files in the folder share one semantics era - |
|
|
//| the DatabaseVersion wipe guarantees it. |
|
|
//| |
|
|
//| Read-only open: this must never write, prune, or lock the DB |
|
|
//| the journaling side owns. |
|
|
//+------------------------------------------------------------------+
|
|
bool CSignalMETA::LoadMetaCorpus(void)
|
|
{
|
|
const string folder = "Warrior_EA\\Databases\\Signals\\";
|
|
//--- Only THIS symbol's + THIS timeframe's corpora are candidates. The DB filename is
|
|
//--- <symbol>_<period>_<fingerprint>.db, and "largest on disk" without this filter would happily
|
|
//--- hand an H4 chart the (bigger) H1 corpus - whose rows then resolve onto the wrong bars - or an
|
|
//--- SP500 chart another symbol's DB entirely. Candidates journaled on a different grid are not
|
|
//--- this chart's candidates.
|
|
const string mustPrefix = _Symbol + "_" + IntegerToString((int)_Period) + "_";
|
|
string bestFile = "";
|
|
long bestRows = 0;
|
|
string fname;
|
|
long find = FileFindFirst(folder + "*.db", fname, FILE_COMMON);
|
|
if(find != INVALID_HANDLE)
|
|
{
|
|
do
|
|
{
|
|
if(StringFind(fname, mustPrefix) != 0)
|
|
continue;
|
|
int db = DatabaseOpen(folder + fname, DATABASE_OPEN_READONLY | DATABASE_OPEN_COMMON);
|
|
if(db == INVALID_HANDLE)
|
|
continue;
|
|
long rows = CountDbPatternRows(db);
|
|
DatabaseClose(db);
|
|
if(rows > bestRows)
|
|
{
|
|
bestRows = rows;
|
|
bestFile = fname;
|
|
}
|
|
}
|
|
while(FileFindNext(find, fname));
|
|
FileFindClose(find);
|
|
}
|
|
if(bestFile == "" || bestRows <= 0)
|
|
{
|
|
Print(ID + ": META CORPUS UNAVAILABLE - no signal DB matching " + mustPrefix + "*.db with pattern"
|
|
" rows found under Common\\Files\\" + folder + ". Build one for THIS symbol+timeframe first:"
|
|
" wipe the Signals folder, then run a long backtest on this chart's symbol AND timeframe"
|
|
" with UseDatabaseRanking=true and DB_MaxRowsPerTable raised (see Meta_Labeling_Design.md S1).");
|
|
return false;
|
|
}
|
|
int db = DatabaseOpen(folder + bestFile, DATABASE_OPEN_READONLY | DATABASE_OPEN_COMMON);
|
|
if(db == INVALID_HANDLE)
|
|
{
|
|
Print(ID + ": failed to reopen corpus DB " + bestFile + " (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
return false;
|
|
}
|
|
ArrayResize(m_corpusGmt, (int)bestRows);
|
|
ArrayResize(m_corpusSide, (int)bestRows);
|
|
ArrayResize(m_corpusNetVote, (int)bestRows);
|
|
ArrayResize(m_corpusFamily, (int)bestRows);
|
|
ArrayResize(m_corpusPattern, (int)bestRows);
|
|
ArrayResize(m_corpusEntry, (int)bestRows);
|
|
m_corpusCount = 0;
|
|
for(int f = 0; f < 4; f++)
|
|
for(int p = 0; p < MetaFamilyPatterns(f); p++)
|
|
for(int s = 0; s < 2; s++)
|
|
{
|
|
string table = MetaFamilyName(f) + "_Pattern_" + IntegerToString(p) +
|
|
(s == 0 ? "_Buy" : "_Sell");
|
|
int stmt = DatabasePrepare(db, "SELECT year, month, day, hour, minutes, netVote, entryPrice"
|
|
" FROM " + table);
|
|
if(stmt == INVALID_HANDLE)
|
|
continue;
|
|
while(DatabaseRead(stmt) && m_corpusCount < (int)bestRows)
|
|
{
|
|
long y = 0, mo = 0, d = 0, h = 0, mi = 0;
|
|
double nv = 0.0, ep = 0.0;
|
|
DatabaseColumnLong(stmt, 0, y);
|
|
DatabaseColumnLong(stmt, 1, mo);
|
|
DatabaseColumnLong(stmt, 2, d);
|
|
DatabaseColumnLong(stmt, 3, h);
|
|
DatabaseColumnLong(stmt, 4, mi);
|
|
DatabaseColumnDouble(stmt, 5, nv);
|
|
DatabaseColumnDouble(stmt, 6, ep);
|
|
MqlDateTime t;
|
|
t.year = (int)y;
|
|
t.mon = (int)mo;
|
|
t.day = (int)d;
|
|
t.hour = (int)h;
|
|
t.min = (int)mi;
|
|
t.sec = 0;
|
|
m_corpusGmt[m_corpusCount] = StructToTime(t);
|
|
m_corpusSide[m_corpusCount] = (char)(s == 0 ? 1 : -1);
|
|
m_corpusNetVote[m_corpusCount] = nv;
|
|
m_corpusFamily[m_corpusCount] = (short)f;
|
|
m_corpusPattern[m_corpusCount] = (short)p;
|
|
m_corpusEntry[m_corpusCount] = ep;
|
|
m_corpusCount++;
|
|
}
|
|
DatabaseFinalize(stmt);
|
|
}
|
|
DatabaseClose(db);
|
|
m_corpusLoaded = (m_corpusCount > 0);
|
|
Print(ID + StringFormat(": meta corpus loaded from %s - %d candidates across the pattern tables"
|
|
" (largest of the DBs found; corpus rows are GMT-stamped, resolution to"
|
|
" server bars happens per era).", bestFile, m_corpusCount));
|
|
return m_corpusLoaded;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Resolve the corpus onto this era's bar grid. |
|
|
//| |
|
|
//| DB rows are GMT; bar history is server time (EET-ish, DST moves |
|
|
//| it). The offset is MEASURED PER ROW, not assumed: rows are |
|
|
//| journaled at the bar's opening tick with the touchable price, so |
|
|
//| the RIGHT offset's bar open matches entryPrice to within the |
|
|
//| spread while a wrong offset lands a full hourly move away. Each |
|
|
//| row takes the offset (0..+4h) minimizing |open - entry| over |
|
|
//| offsets that land on an exact bar, then must pass a tolerance - |
|
|
//| decisive per row, and immune to DST regime changes across an |
|
|
//| 18-year corpus (the histogram printed below shows the winter/ |
|
|
//| summer split directly). |
|
|
//| |
|
|
//| The window-span filter kills the pre-2017 daily-backfill regime |
|
|
//| (measured 2026-08-13: hour-0-only rows, one per day): a |
|
|
//| candidate whose m_historyBars-deep window spans more than 4x its |
|
|
//| nominal duration is sitting on bars that are not really H1, and |
|
|
//| its geometry/labels would be silently wrong. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| Generate the candidate corpus by sweeping the REAL classic |
|
|
//| ladders over this chart's own history. |
|
|
//| |
|
|
//| Every pattern condition anchors on StartIndex() (verified across |
|
|
//| all four signal files), so EvalShift(i) makes the exact live |
|
|
//| code answer "what would you have fired at bar i" - the silent- |
|
|
//| divergence trap that justified the DB corpus does not exist on |
|
|
//| this path, and neither do the tester run, the GMT-offset |
|
|
//| ambiguity, or the DB row caps. Times/prices stored are this |
|
|
//| chart's own bar opens, so MetaPrepareEra's resolution matches at |
|
|
//| offset +0 with zero price error by construction. |
|
|
//| |
|
|
//| One-time cost at first era: bars x sources Direction() calls - |
|
|
//| a few seconds. Runs on the chart thread like the label prebuild. |
|
|
//+------------------------------------------------------------------+
|
|
bool CSignalMETA::BuildCorpusBySweep(void)
|
|
{
|
|
if(m_srcCount <= 0)
|
|
return false;
|
|
ENUM_TIMEFRAMES per = (ENUM_TIMEFRAMES)m_period;
|
|
int bars = Bars(_Symbol, per);
|
|
if(bars <= 300)
|
|
return false;
|
|
for(int s = 0; s < m_srcCount; s++)
|
|
if(!m_srcFilter[s].SweepPrepare(bars))
|
|
{
|
|
Print(ID + ": candidate sweep - filter " + m_srcFilter[s].GetFilterID() +
|
|
" could not prepare deep buffers; falling back to a DB corpus.");
|
|
return false;
|
|
}
|
|
//--- skip the indicator warm-up tail at the oldest edge of history (reads there are EMPTY/garbage
|
|
//--- and would fabricate patterns); 150 bars comfortably covers every classic period in use.
|
|
int deepest = bars - 150;
|
|
int cap = bars * 2;
|
|
ArrayResize(m_corpusGmt, cap);
|
|
ArrayResize(m_corpusSide, cap);
|
|
ArrayResize(m_corpusNetVote, cap);
|
|
ArrayResize(m_corpusFamily, cap);
|
|
ArrayResize(m_corpusPattern, cap);
|
|
ArrayResize(m_corpusEntry, cap);
|
|
m_corpusCount = 0;
|
|
Print(ID + StringFormat(": sweeping %d classic ladder(s) over %d bars for candidates - the chart"
|
|
" is busy for a few seconds...", m_srcCount, deepest));
|
|
uint t0 = GetTickCount();
|
|
for(int i = deepest; i >= 2 && m_corpusCount + 2 <= cap; i--)
|
|
{
|
|
datetime bt = iTime(_Symbol, per, i);
|
|
double bo = iOpen(_Symbol, per, i);
|
|
if(bt <= 0 || bo <= 0.0)
|
|
continue;
|
|
for(int s = 0; s < m_srcCount; s++)
|
|
{
|
|
CExpertSignalCustom *f = m_srcFilter[s];
|
|
f.EvalShift(i);
|
|
f.Direction();
|
|
f.EvalShift(0);
|
|
string pl = f.GetActivePatternLong();
|
|
string ps = f.GetActivePatternShort();
|
|
double nv = f.LastNetVote();
|
|
//--- "Pattern_N" -> N; same per-side, one-candidate-per-bar semantics as live journaling
|
|
if(pl != "NULL")
|
|
{
|
|
m_corpusGmt[m_corpusCount] = bt;
|
|
m_corpusEntry[m_corpusCount] = bo;
|
|
m_corpusSide[m_corpusCount] = 1;
|
|
m_corpusFamily[m_corpusCount] = (short)m_srcFamily[s];
|
|
m_corpusPattern[m_corpusCount] = (short)StringToInteger(StringSubstr(pl, 8));
|
|
m_corpusNetVote[m_corpusCount] = nv;
|
|
m_corpusCount++;
|
|
}
|
|
if(ps != "NULL")
|
|
{
|
|
m_corpusGmt[m_corpusCount] = bt;
|
|
m_corpusEntry[m_corpusCount] = bo;
|
|
m_corpusSide[m_corpusCount] = -1;
|
|
m_corpusFamily[m_corpusCount] = (short)m_srcFamily[s];
|
|
m_corpusPattern[m_corpusCount] = (short)StringToInteger(StringSubstr(ps, 8));
|
|
m_corpusNetVote[m_corpusCount] = nv;
|
|
m_corpusCount++;
|
|
}
|
|
}
|
|
}
|
|
Print(ID + StringFormat(": candidate sweep done - %d candidates from %d bars in %.1fs (no tester"
|
|
" corpus run needed; DB corpus not used).", m_corpusCount, deepest,
|
|
(GetTickCount() - t0) / 1000.0));
|
|
m_corpusLoaded = (m_corpusCount > 0);
|
|
return m_corpusLoaded;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CSignalMETA::MetaPrepareEra(const int bars)
|
|
{
|
|
//--- corpus source order: the on-chart ladder sweep (self-contained, preferred), then a
|
|
//--- tester-built DB corpus as fallback for charts whose classic filters are disabled.
|
|
if(!m_corpusLoaded && !BuildCorpusBySweep() && !LoadMetaCorpus())
|
|
return false;
|
|
ENUM_TIMEFRAMES per = (ENUM_TIMEFRAMES)m_period;
|
|
ArrayResize(m_metaCandHead, bars);
|
|
ArrayInitialize(m_metaCandHead, -1);
|
|
ArrayResize(m_metaCandBar, m_corpusCount);
|
|
ArrayResize(m_metaCandSide, m_corpusCount);
|
|
ArrayResize(m_metaCandNetVote, m_corpusCount);
|
|
ArrayResize(m_metaCandFamily, m_corpusCount);
|
|
ArrayResize(m_metaCandPattern, m_corpusCount);
|
|
ArrayResize(m_metaCandNext, m_corpusCount);
|
|
m_metaCandCount = 0;
|
|
int offCount[5] = {0, 0, 0, 0, 0};
|
|
int dropNoBar = 0, dropPrice = 0, dropRegime = 0, dropRange = 0;
|
|
long spanCap = (long)PeriodSeconds(per) * (long)MathMax(m_historyBars, 1) * 4;
|
|
for(int r = 0; r < m_corpusCount; r++)
|
|
{
|
|
int bestSh = -1, bestOff = -1;
|
|
double bestDiff = DBL_MAX;
|
|
for(int off = 0; off <= 4; off++)
|
|
{
|
|
int sh = iBarShift(_Symbol, per, m_corpusGmt[r] + off * 3600, true);
|
|
if(sh < 0)
|
|
continue;
|
|
double diff = MathAbs(iOpen(_Symbol, per, sh) - m_corpusEntry[r]);
|
|
if(diff < bestDiff)
|
|
{
|
|
bestDiff = diff;
|
|
bestSh = sh;
|
|
bestOff = off;
|
|
}
|
|
}
|
|
if(bestSh < 0)
|
|
{
|
|
dropNoBar++;
|
|
continue;
|
|
}
|
|
//--- price tolerance: right-offset |diff| <= the spread; wrong-offset ~ an hourly move. 15% of
|
|
//--- the bar's ATR separates the two with a wide margin either way; the fallback (5 basis
|
|
//--- points) covers bars where the ATR indicator has no value that deep in history.
|
|
double atr = m_ATR.Main(bestSh);
|
|
double tol = (MathIsValidNumber(atr) && atr > 0.0) ? 0.15 * atr : 0.0005 * m_corpusEntry[r];
|
|
if(bestDiff > tol)
|
|
{
|
|
dropPrice++;
|
|
continue;
|
|
}
|
|
if(bestSh >= bars)
|
|
{
|
|
dropRange++;
|
|
continue;
|
|
}
|
|
datetime tSh = iTime(_Symbol, per, bestSh);
|
|
datetime tDeep = iTime(_Symbol, per, bestSh + (int)MathMax(m_historyBars, 1));
|
|
if(tSh <= 0 || tDeep <= 0 || (long)(tSh - tDeep) > spanCap)
|
|
{
|
|
dropRegime++;
|
|
continue;
|
|
}
|
|
int id = m_metaCandCount;
|
|
m_metaCandBar[id] = bestSh;
|
|
m_metaCandSide[id] = m_corpusSide[r];
|
|
m_metaCandNetVote[id] = m_corpusNetVote[r];
|
|
m_metaCandFamily[id] = m_corpusFamily[r];
|
|
m_metaCandPattern[id] = m_corpusPattern[r];
|
|
m_metaCandNext[id] = m_metaCandHead[bestSh];
|
|
m_metaCandHead[bestSh] = id;
|
|
m_metaCandCount++;
|
|
if(bestOff >= 0 && bestOff <= 4)
|
|
offCount[bestOff]++;
|
|
}
|
|
string line = StringFormat("%s: era candidate resolution - %d of %d corpus rows usable | GMT->server"
|
|
" offset histogram +0h:%d +1h:%d +2h:%d +3h:%d +4h:%d | dropped: %d no"
|
|
" exact bar, %d price mismatch (wrong offset/bad data), %d non-H1 regime"
|
|
" (pre-intraday backfill), %d beyond era grid",
|
|
ID, m_metaCandCount, m_corpusCount, offCount[0], offCount[1], offCount[2],
|
|
offCount[3], offCount[4], dropNoBar, dropPrice, dropRegime, dropRange);
|
|
if(!m_prepareReported)
|
|
{
|
|
m_prepareReported = true;
|
|
Print(line);
|
|
}
|
|
else
|
|
PrintVerbose(line);
|
|
//--- one-shot dataset export for offline pooled training - runs here because this is the first
|
|
//--- moment candidates AND labels both exist on the current bar grid (the label prebuild completed
|
|
//--- before the era start that called us).
|
|
if(Meta_ExportDataset && !m_datasetExported && m_metaCandCount > 0)
|
|
ExportMetaDataset();
|
|
return m_metaCandCount > 0;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Dump the full training set for offline (cross-sectional pooled) |
|
|
//| work: one float32 row per resolved, LABELED candidate - |
|
|
//| [barTime int64][family int32][pattern int32][side int32] |
|
|
//| [won int32][NetInputWidth() floats: window + descriptor]. |
|
|
//| The sidecar .meta.csv carries the layout + the geometry/BE the |
|
|
//| labels were computed at, so the offline side never guesses. |
|
|
//| This is EXACTLY what pass 2 trains on - same window builder, |
|
|
//| same descriptor, same caches - so an offline model on this file |
|
|
//| and the EA's own training see byte-equivalent examples. |
|
|
//+------------------------------------------------------------------+
|
|
void CSignalMETA::ExportMetaDataset(void)
|
|
{
|
|
m_datasetExported = true;
|
|
const string base = "Warrior_EA\\MetaExport\\" + _Symbol + "_" + IntegerToString((int)m_period);
|
|
int fh = FileOpen(base + ".f32", FILE_BIN | FILE_WRITE | FILE_COMMON);
|
|
if(fh == INVALID_HANDLE)
|
|
{
|
|
Print(ID + ": dataset export FAILED - cannot open Common\\Files\\" + base + ".f32 (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
return;
|
|
}
|
|
Print(ID + StringFormat(": exporting %d candidates to Common\\Files\\%s.f32 - one pass-1-sized"
|
|
" sweep, the chart is busy for it...", m_metaCandCount, base));
|
|
int width = NetInputWidth();
|
|
int rows = 0, skipLabel = 0, skipWindow = 0;
|
|
for(int cd = 0; cd < m_metaCandCount; cd++)
|
|
{
|
|
int idx = m_metaCandBar[cd];
|
|
if(idx < 0 || idx >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[idx])
|
|
{
|
|
skipLabel++;
|
|
continue;
|
|
}
|
|
if(!BuildFeatureWindow(idx))
|
|
{
|
|
skipWindow++;
|
|
continue;
|
|
}
|
|
AppendCandidateFeatures(cd);
|
|
if(TempData.Total() != width)
|
|
{
|
|
skipWindow++;
|
|
continue;
|
|
}
|
|
FileWriteLong(fh, (long)iTime(_Symbol, (ENUM_TIMEFRAMES)m_period, idx));
|
|
FileWriteInteger(fh, (int)m_metaCandFamily[cd]);
|
|
FileWriteInteger(fh, (int)m_metaCandPattern[cd]);
|
|
FileWriteInteger(fh, (int)m_metaCandSide[cd]);
|
|
FileWriteInteger(fh, MetaCandidateWon(cd, idx) ? 1 : 0);
|
|
for(int k = 0; k < width; k++)
|
|
FileWriteFloat(fh, (float)TempData.At(k));
|
|
rows++;
|
|
}
|
|
FileClose(fh);
|
|
double slMult = 0.0, tpMult = 0.0;
|
|
BarrierMultiples(slMult, tpMult);
|
|
double bePct = (slMult + tpMult > 0.0) ? 100.0 * slMult / (slMult + tpMult) : 50.0;
|
|
int mh = FileOpen(base + ".meta.csv", FILE_CSV | FILE_WRITE | FILE_COMMON, ',');
|
|
if(mh != INVALID_HANDLE)
|
|
{
|
|
FileWrite(mh, "symbol", "period", "rows", "width", "historyBars", "featuresPerBar", "descWidth",
|
|
"slMult", "tpMult", "breakEvenPct", "horizonBars", "spreadPoints");
|
|
FileWrite(mh, _Symbol, IntegerToString((int)m_period), IntegerToString(rows),
|
|
IntegerToString(width), IntegerToString((int)m_historyBars),
|
|
IntegerToString(m_neuronsCount), IntegerToString(MetaDescWidth()),
|
|
DoubleToString(slMult, 4), DoubleToString(tpMult, 4), DoubleToString(bePct, 2),
|
|
IntegerToString(m_barrierHorizonBars), IntegerToString((int)m_symbol.Spread()));
|
|
FileClose(mh);
|
|
}
|
|
Print(ID + StringFormat(": dataset exported - %d rows (%d skipped: %d unlabeled near the era edge,"
|
|
" %d unusable windows) x %d floats | geometry %.2f/%.2f BE %.1f%% |"
|
|
" %s.f32 + .meta.csv", rows, skipLabel + skipWindow, skipLabel, skipWindow,
|
|
width, slMult, tpMult, bePct, base));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| The setup descriptor - MUST append exactly META_DESC_FEATURES |
|
|
//| values (the input layer is sized for them; a short append fails |
|
|
//| the NetInputWidth() guard and the sample is skipped, a long one |
|
|
//| would corrupt the forward pass). |
|
|
//+------------------------------------------------------------------+
|
|
void CSignalMETA::AppendCandidateFeatures(const int candId)
|
|
{
|
|
if(candId < 0 || candId >= m_metaCandCount)
|
|
return;
|
|
int idx = m_metaCandBar[candId];
|
|
int slot = OneHotSlot(m_metaCandFamily[candId], m_metaCandPattern[candId]);
|
|
for(int k = 0; k < META_ONE_HOT_SLOTS; k++)
|
|
TempData.Add(k == slot ? 1.0 : 0.0);
|
|
TempData.Add((double)m_metaCandSide[candId]);
|
|
//--- netVote is in raw pattern-weight units (+-100ish); tanh(nv/20) keeps resolution where the
|
|
//--- votes actually live while bounding the tails. MQL5 has no MathTanh - via exp.
|
|
double e2 = MathExp(2.0 * (m_metaCandNetVote[candId] / 20.0));
|
|
TempData.Add((e2 - 1.0) / (e2 + 1.0));
|
|
//--- geometry in ATR units - constant across candidates TODAY (one pinned barrier pair), but the
|
|
//--- design's excursion-head integration makes it per-candidate later, so it rides as a feature.
|
|
double slMult = 0.0, tpMult = 0.0;
|
|
BarrierMultiples(slMult, tpMult);
|
|
TempData.Add(slMult);
|
|
TempData.Add(tpMult);
|
|
//--- spread/ATR at the fire bar (EnsureSpreadSeries copies unconditionally for the meta target)
|
|
double sprAtr = 0.0;
|
|
double atr = m_ATR.Main(idx);
|
|
if(idx >= 0 && idx < m_spreadSeriesBars && MathIsValidNumber(atr) && atr > 0.0)
|
|
sprAtr = (double)m_spreadSeries[idx] * m_symbol.Point() / atr;
|
|
TempData.Add(sprAtr);
|
|
}
|
|
//+------------------------------------------------------------------+
|