chiki2bum2/SniperGold_ML
Watch
1
0
Fork
You've already forked SniperGold_ML
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
SniperGold_ML/docs/P2_1_HTF_FORENSIC.md

6.8 KiB
Raw Permalink Blame History

P2.1 — HTF BIAS FORENSIC TRACE (ROOT CAUSE)

Status: ROOT CAUSE FOUND + REPRODUCED (14849/14850 rows)
Date: 2026-08-21
Classification: B = EA bug (runtime indexing deviates from v4.4 and from the comment's intent)

1. Problem

f0 (D1 bias)  mismatch py_full vs EA = 0.3609
f1 (H4 bias)  mismatch               = 0.4516
f2 (H1 bias)  mismatch               = 0.5130
f18_conf      mismatch               = 0.5189   (derived: follows f0-f2)

The Python replica tf_bias_asof (200 most-recent bars ending at the last closed bar) does not reproduce the EA BTTFBias output.

2. Method

  1. Trace the call chain ComputeAll -> g_bTF1/2/3 = BTTFBias(sD1/sH4/sH1).
  2. Read the Engine 1 cache (AF_Engine1_MTFData.mqh): closed-bar lock, 250-bar capacity per HTF slot, series index bars[0]=newest.
  3. Test window hypotheses against the parity mode-2 CSV (14850 rows): PY / CLOSED / ANCHOR / ANCHOR250 — all FAILED (mismatch 0.36-0.79).
  4. Added debug mode 3 to the EA (full HTF cache dump at 5 picked timestamps + compact per-bar count/need/bias rows). Ran tester 2026.01.01-07.21.
  5. Minimal EA isolate: VERBATIM copy of BTTFBias + pivot/break trace on the exact window from the tester cache.
  6. Quantitative verification: EA_hat = tf_bias_asof(E_ea - 50) vs CSV.

3. Evidence

3.1 EA cache == training npz data (IDENTICAL INPUT STATE)

Mode 3 FULLDUMP (5 timestamps) compared bar-by-bar with Files\AlgoForge\Data\XAUUSD_{D1,H4,H1}.npz:

D1/H4/H1 @ 2025.12.31 20:00, 2026.01.05 12:00, 2026.03.18 01:00,
          2026.06.16 13:30, 2026.07.20 15:45
bars=250 matched=250 diff=0 missing=0
max|d| open=0.00000 high=0.00000 low=0.00000 close=0.00000

Conclusion: not an input-state mismatch (P1 suspected the tester D1 cache differed from the training feed — NOT proven; caches are identical).

3.2 The tester provides ~1 year of pre-test history

Tester log (mode 3):

XAUUSD,H1: history cache ... contains 5907 bars from 2025.01.02 01:00 to 2025.12.31 20:00
XAUUSD,H4: history begins from 2025.01.02 00:00
XAUUSD,Daily: history begins from 2025.01.02 00:00

The full HTF cache (250 bars) is available FROM THE FIRST ROW -> need = min(200, count) = 200 always, count = 250 always (the tester pre-loads 1 year before FromDate).

3.3 EA BTTFBias window = 200 OLDEST bars of the 250-bar cache

EA (AlgoForge_Backtest_Baseline.mq5):

string BTTFBias(int slot)
{
   int need=MathMin(200,e1.Count(slot));      // need=200, count=250
   ...
   for(int i=0;i<need;i++)
   {
      AFBar b;
      if(!e1.GetBar(slot,e1.Count(slot)-1-i,b)) return "Neutral";
      h[i]=b.high; l[i]=b.low; cl[i]=b.close;
   }

GetBar(idxFromRight): 0 = newest closed bar. GetBar(count-1-i) with i=0..199 fetches bars[249..50] = the 200 OLDEST bars of the 250-bar cache. The newest bars 1..50 (bars[0..49]) NEVER enter the window.

Isolate evidence (AlgoForge_BTTF_Isolate): at 2026.03.18 01:00, the EA window = 2025.03.28 .. 2026.01.06 (not ending 03-17), result Bullish.

3.4 Legacy v4.4 reference = 200 NEWEST bars

SniperGold_SMC_ProPlus_v4_4.mq5 TFBias():

int need=200;
CopyHigh(_Symbol,tf,0,need,h);   // 200 NEWEST bars (index 0 = forming)
ArraySetAsSeries(h,false);       // index 0 = oldest
int s=3, m=need-1;               // m excludes the still-forming HTF bar

v4.4 = 200 newest bars (the forming bar is inert at index 199) = tf_bias_asof(forming). The EA deviates from v4.4 -> EA adaptation bug, not legacy semantics.

3.5 Quantitative reproduction

EA_hat = tf_bias_asof(E_ea - 50) where E_ea = last closed HTF bar at tc = t + 900 (M15 bar close time), lag = count - need = 250 - 200 = 50:

D1: mismatch =     0 / 14850   rate = 0.0000
H4: mismatch =     0 / 14850   rate = 0.0000
H1: mismatch =     1 / 14850   rate = 0.0001

3.6 The single H1 exception — cache staleness during a trading gap

Row 2026-02-16 21:30: EA H1 = -1, hat = +1. Cause: H1 data gap 21:00 -> 01:00 (daily break). The 20:00 H1 bar closes at 21:00, but e1.Refresh() only rebuilds when Bars(symbol,tf) changes; the next bar only appears at 01:00 -> the H1 cache stays ending at 19:00 during the gap. Effect: the window endpoint shifts by one extra bar. Only 1/14850 rows flip (bias robustness).

4. Root Cause (summary)

EA BTTFBias uses GetBar(count-1-i) -> window = the 200 OLDEST bars of the 250-bar
cache = HTF bias lagged by 50 bars (D1: 50 days, H4: ~8.3 days, H1: ~2.1 days).

Not an input-state mismatch (cache == npz).
Not legacy semantics (v4.4 uses the 200 newest bars).
Deterministic & reproducible (14849/14850).

Per protocol classification: B = EA bug. The secondary effect (gap) is an Engine 1 design characteristic (Refresh-on-Bars-change), documented as an edge case.

5. Source-of-Truth Decision

SOURCE_OF_TRUTH = CORRECTED SEMANTICS (not EA-as-deployed, not old py-full)

Definition:
  f_htf(t) = BTTFBias over the 200 newest CLOSED HTF bars ending at E_ea(t)
  E_ea(t)  = last closed HTF bar at tc = t + 900
           = searchsorted(ht, tc - period, 'right') - 1
  = tf_bias_asof(E_ea)

Runtime (EA)  : fix BTTFBias -> window = the 200 NEWEST bars of the cache
                (GetBar(slot, need-1-i) or equivalent), closed-bar lock kept.
Training (py) : as-of M15 bar close mapping (E_ea), not as-of open (containing-1).

Note on the 1-bar difference vs v4.4 (window ending at the inert forming bar):
  documented as a legacy nuance; the corrected contract uses the last closed
  bar (E_ea) — non-look-ahead, deterministic, one definition for both sides.

RETRAINING IS PROHIBITED before full parity (P2.5) is complete.

6. Evidence Artifacts

ml/parity/htf_forensic.py            tests 4 window hypotheses vs CSV (PY/CLOSED/ANCHOR)
ml/parity/compare_htf_dump.py        EA FULLDUMP vs npz (identical)
ml/parity/verify_bttf_dump.py        bttf_window on the exact EA input
ml/parity/verify_bttf_rootcause.py   EA_hat = tf_bias_asof(E_ea-50) -> 0/0/1 mismatches
ml/parity/probe_h4_window.py         H4 window search (all -1 from npz)
ml/parity/check_h1_gap.py            H1 21:00-01:00 gap (cache staleness)
Experts/AlgoForge_BTTF_Isolate.mq5   verbatim BTTFBias isolate + trace
Experts/AlgoForge_Backtest_Baseline.mq5  (debug mode 3 added, additive)
Profiles/Tester/AlgoForge_HTFDebug...ini / AlgoForge_BTTF_Isolate...ini
Tester/Agent-127.0.0.1-3000/MQL5/Files/AlgoForge_htf_debug_XAUUSD_M15.csv  (dump)

7. P2.1 Status

Root cause         : FOUND (B = EA bug, 50-bar lag)
Source of truth    : DECIDED (corrected semantics, tf_bias_asof(E_ea))
Runtime semantics  : 200 newest bars ending at E_ea (BTTFBias fix)
Training semantics : as-of close tc, tf_bias_asof(E_ea)
Parity (current)   : EA-as-deployed = tf_bias_asof(E_ea-50) -> 0/0/1 mismatches
                     (runtime + training fix implemented in P2.5)