SniperGold_ML/docs/P2_1_HTF_FORENSIC.md

# P2.1 — HTF BIAS FORENSIC TRACE (ROOT CAUSE)

Status: **ROOT CAUSE FOUND + REPRODUCED (14849/14850 rows)**  
Date: 2026-08-21  
Classification: **B = EA bug** (runtime indexing deviates from v4.4 and from the comment's intent)

---

## 1. Problem

```text
f0 (D1 bias)  mismatch py_full vs EA = 0.3609
f1 (H4 bias)  mismatch               = 0.4516
f2 (H1 bias)  mismatch               = 0.5130
f18_conf      mismatch               = 0.5189   (derived: follows f0-f2)
```

The Python replica `tf_bias_asof` (200 most-recent bars ending at the last closed bar)
does not reproduce the EA `BTTFBias` output.

---

## 2. Method

1. Trace the call chain `ComputeAll -> g_bTF1/2/3 = BTTFBias(sD1/sH4/sH1)`.
2. Read the Engine 1 cache (`AF_Engine1_MTFData.mqh`): closed-bar lock, 250-bar
   capacity per HTF slot, series index `bars[0]=newest`.
3. Test window hypotheses against the parity mode-2 CSV (14850 rows): PY / CLOSED /
   ANCHOR / ANCHOR250 — all FAILED (mismatch 0.36-0.79).
4. Added debug mode 3 to the EA (full HTF cache dump at 5 picked timestamps +
   compact per-bar count/need/bias rows). Ran tester 2026.01.01-07.21.
5. Minimal EA isolate: VERBATIM copy of BTTFBias + pivot/break trace on the exact
   window from the tester cache.
6. Quantitative verification: `EA_hat = tf_bias_asof(E_ea - 50)` vs CSV.

---

## 3. Evidence

### 3.1 EA cache == training npz data (IDENTICAL INPUT STATE)

Mode 3 FULLDUMP (5 timestamps) compared bar-by-bar with
`Files\AlgoForge\Data\XAUUSD_{D1,H4,H1}.npz`:

```text
D1/H4/H1 @ 2025.12.31 20:00, 2026.01.05 12:00, 2026.03.18 01:00,
          2026.06.16 13:30, 2026.07.20 15:45
bars=250 matched=250 diff=0 missing=0
max|d| open=0.00000 high=0.00000 low=0.00000 close=0.00000
```

Conclusion: **not an input-state mismatch** (P1 suspected the tester D1 cache
differed from the training feed — NOT proven; caches are identical).

### 3.2 The tester provides ~1 year of pre-test history

Tester log (mode 3):

```text
XAUUSD,H1: history cache ... contains 5907 bars from 2025.01.02 01:00 to 2025.12.31 20:00
XAUUSD,H4: history begins from 2025.01.02 00:00
XAUUSD,Daily: history begins from 2025.01.02 00:00
```

The full HTF cache (250 bars) is available FROM THE FIRST ROW -> `need = min(200, count) = 200`
always, `count = 250` always (the tester pre-loads 1 year before FromDate).

### 3.3 EA BTTFBias window = 200 OLDEST bars of the 250-bar cache

EA (`AlgoForge_Backtest_Baseline.mq5`):

```mql5
string BTTFBias(int slot)
{
   int need=MathMin(200,e1.Count(slot));      // need=200, count=250
   ...
   for(int i=0;i<need;i++)
   {
      AFBar b;
      if(!e1.GetBar(slot,e1.Count(slot)-1-i,b)) return "Neutral";
      h[i]=b.high; l[i]=b.low; cl[i]=b.close;
   }
```

`GetBar(idxFromRight)`: 0 = newest closed bar. `GetBar(count-1-i)` with
i=0..199 fetches `bars[249..50]` = the **200 OLDEST bars** of the 250-bar cache.
The newest bars 1..50 (bars[0..49]) NEVER enter the window.

Isolate evidence (`AlgoForge_BTTF_Isolate`): at 2026.03.18 01:00, the EA window =
`2025.03.28 .. 2026.01.06` (not ending 03-17), result `Bullish`.

### 3.4 Legacy v4.4 reference = 200 NEWEST bars

`SniperGold_SMC_ProPlus_v4_4.mq5` TFBias():

```mql5
int need=200;
CopyHigh(_Symbol,tf,0,need,h);   // 200 NEWEST bars (index 0 = forming)
ArraySetAsSeries(h,false);       // index 0 = oldest
int s=3, m=need-1;               // m excludes the still-forming HTF bar
```

v4.4 = 200 newest bars (the forming bar is inert at index 199) = `tf_bias_asof(forming)`.
The EA deviates from v4.4 -> **EA adaptation bug**, not legacy semantics.

### 3.5 Quantitative reproduction

`EA_hat = tf_bias_asof(E_ea - 50)` where `E_ea` = last closed HTF bar at
`tc = t + 900` (M15 bar close time), lag = `count - need = 250 - 200 = 50`:

```text
D1: mismatch =     0 / 14850   rate = 0.0000
H4: mismatch =     0 / 14850   rate = 0.0000
H1: mismatch =     1 / 14850   rate = 0.0001
```

### 3.6 The single H1 exception — cache staleness during a trading gap

Row `2026-02-16 21:30`: EA H1 = -1, hat = +1. Cause: H1 data gap
`21:00 -> 01:00` (daily break). The 20:00 H1 bar closes at 21:00, but
`e1.Refresh()` only rebuilds when `Bars(symbol,tf)` changes; the next bar only
appears at 01:00 -> the H1 cache stays ending at 19:00 during the gap. Effect:
the window endpoint shifts by one extra bar. Only 1/14850 rows flip (bias robustness).

---

## 4. Root Cause (summary)

```text
EA BTTFBias uses GetBar(count-1-i) -> window = the 200 OLDEST bars of the 250-bar
cache = HTF bias lagged by 50 bars (D1: 50 days, H4: ~8.3 days, H1: ~2.1 days).

Not an input-state mismatch (cache == npz).
Not legacy semantics (v4.4 uses the 200 newest bars).
Deterministic & reproducible (14849/14850).
```

Per protocol classification: **B = EA bug**. The secondary effect (gap) is an
Engine 1 design characteristic (Refresh-on-Bars-change), documented as an edge case.

---

## 5. Source-of-Truth Decision

```text
SOURCE_OF_TRUTH = CORRECTED SEMANTICS (not EA-as-deployed, not old py-full)

Definition:
  f_htf(t) = BTTFBias over the 200 newest CLOSED HTF bars ending at E_ea(t)
  E_ea(t)  = last closed HTF bar at tc = t + 900
           = searchsorted(ht, tc - period, 'right') - 1
  = tf_bias_asof(E_ea)

Runtime (EA)  : fix BTTFBias -> window = the 200 NEWEST bars of the cache
                (GetBar(slot, need-1-i) or equivalent), closed-bar lock kept.
Training (py) : as-of M15 bar close mapping (E_ea), not as-of open (containing-1).

Note on the 1-bar difference vs v4.4 (window ending at the inert forming bar):
  documented as a legacy nuance; the corrected contract uses the last closed
  bar (E_ea) — non-look-ahead, deterministic, one definition for both sides.

RETRAINING IS PROHIBITED before full parity (P2.5) is complete.
```

---

## 6. Evidence Artifacts

```text
ml/parity/htf_forensic.py            tests 4 window hypotheses vs CSV (PY/CLOSED/ANCHOR)
ml/parity/compare_htf_dump.py        EA FULLDUMP vs npz (identical)
ml/parity/verify_bttf_dump.py        bttf_window on the exact EA input
ml/parity/verify_bttf_rootcause.py   EA_hat = tf_bias_asof(E_ea-50) -> 0/0/1 mismatches
ml/parity/probe_h4_window.py         H4 window search (all -1 from npz)
ml/parity/check_h1_gap.py            H1 21:00-01:00 gap (cache staleness)
Experts/AlgoForge_BTTF_Isolate.mq5   verbatim BTTFBias isolate + trace
Experts/AlgoForge_Backtest_Baseline.mq5  (debug mode 3 added, additive)
Profiles/Tester/AlgoForge_HTFDebug...ini / AlgoForge_BTTF_Isolate...ini
Tester/Agent-127.0.0.1-3000/MQL5/Files/AlgoForge_htf_debug_XAUUSD_M15.csv  (dump)
```

---

## 7. P2.1 Status

```text
Root cause         : FOUND (B = EA bug, 50-bar lag)
Source of truth    : DECIDED (corrected semantics, tf_bias_asof(E_ea))
Runtime semantics  : 200 newest bars ending at E_ea (BTTFBias fix)
Training semantics : as-of close tc, tf_bias_asof(E_ea)
Parity (current)   : EA-as-deployed = tf_bias_asof(E_ea-50) -> 0/0/1 mismatches
                     (runtime + training fix implemented in P2.5)
```