193 lines
6.8 KiB
Markdown
193 lines
6.8 KiB
Markdown
# P2.1 — HTF BIAS FORENSIC TRACE (ROOT CAUSE)
|
|
|
|
Status: **ROOT CAUSE FOUND + REPRODUCED (14849/14850 rows)**
|
|
Date: 2026-08-21
|
|
Classification: **B = EA bug** (runtime indexing deviates from v4.4 and from the comment's intent)
|
|
|
|
---
|
|
|
|
## 1. Problem
|
|
|
|
```text
|
|
f0 (D1 bias) mismatch py_full vs EA = 0.3609
|
|
f1 (H4 bias) mismatch = 0.4516
|
|
f2 (H1 bias) mismatch = 0.5130
|
|
f18_conf mismatch = 0.5189 (derived: follows f0-f2)
|
|
```
|
|
|
|
The Python replica `tf_bias_asof` (200 most-recent bars ending at the last closed bar)
|
|
does not reproduce the EA `BTTFBias` output.
|
|
|
|
---
|
|
|
|
## 2. Method
|
|
|
|
1. Trace the call chain `ComputeAll -> g_bTF1/2/3 = BTTFBias(sD1/sH4/sH1)`.
|
|
2. Read the Engine 1 cache (`AF_Engine1_MTFData.mqh`): closed-bar lock, 250-bar
|
|
capacity per HTF slot, series index `bars[0]=newest`.
|
|
3. Test window hypotheses against the parity mode-2 CSV (14850 rows): PY / CLOSED /
|
|
ANCHOR / ANCHOR250 — all FAILED (mismatch 0.36-0.79).
|
|
4. Added debug mode 3 to the EA (full HTF cache dump at 5 picked timestamps +
|
|
compact per-bar count/need/bias rows). Ran tester 2026.01.01-07.21.
|
|
5. Minimal EA isolate: VERBATIM copy of BTTFBias + pivot/break trace on the exact
|
|
window from the tester cache.
|
|
6. Quantitative verification: `EA_hat = tf_bias_asof(E_ea - 50)` vs CSV.
|
|
|
|
---
|
|
|
|
## 3. Evidence
|
|
|
|
### 3.1 EA cache == training npz data (IDENTICAL INPUT STATE)
|
|
|
|
Mode 3 FULLDUMP (5 timestamps) compared bar-by-bar with
|
|
`Files\AlgoForge\Data\XAUUSD_{D1,H4,H1}.npz`:
|
|
|
|
```text
|
|
D1/H4/H1 @ 2025.12.31 20:00, 2026.01.05 12:00, 2026.03.18 01:00,
|
|
2026.06.16 13:30, 2026.07.20 15:45
|
|
bars=250 matched=250 diff=0 missing=0
|
|
max|d| open=0.00000 high=0.00000 low=0.00000 close=0.00000
|
|
```
|
|
|
|
Conclusion: **not an input-state mismatch** (P1 suspected the tester D1 cache
|
|
differed from the training feed — NOT proven; caches are identical).
|
|
|
|
### 3.2 The tester provides ~1 year of pre-test history
|
|
|
|
Tester log (mode 3):
|
|
|
|
```text
|
|
XAUUSD,H1: history cache ... contains 5907 bars from 2025.01.02 01:00 to 2025.12.31 20:00
|
|
XAUUSD,H4: history begins from 2025.01.02 00:00
|
|
XAUUSD,Daily: history begins from 2025.01.02 00:00
|
|
```
|
|
|
|
The full HTF cache (250 bars) is available FROM THE FIRST ROW -> `need = min(200, count) = 200`
|
|
always, `count = 250` always (the tester pre-loads 1 year before FromDate).
|
|
|
|
### 3.3 EA BTTFBias window = 200 OLDEST bars of the 250-bar cache
|
|
|
|
EA (`AlgoForge_Backtest_Baseline.mq5`):
|
|
|
|
```mql5
|
|
string BTTFBias(int slot)
|
|
{
|
|
int need=MathMin(200,e1.Count(slot)); // need=200, count=250
|
|
...
|
|
for(int i=0;i<need;i++)
|
|
{
|
|
AFBar b;
|
|
if(!e1.GetBar(slot,e1.Count(slot)-1-i,b)) return "Neutral";
|
|
h[i]=b.high; l[i]=b.low; cl[i]=b.close;
|
|
}
|
|
```
|
|
|
|
`GetBar(idxFromRight)`: 0 = newest closed bar. `GetBar(count-1-i)` with
|
|
i=0..199 fetches `bars[249..50]` = the **200 OLDEST bars** of the 250-bar cache.
|
|
The newest bars 1..50 (bars[0..49]) NEVER enter the window.
|
|
|
|
Isolate evidence (`AlgoForge_BTTF_Isolate`): at 2026.03.18 01:00, the EA window =
|
|
`2025.03.28 .. 2026.01.06` (not ending 03-17), result `Bullish`.
|
|
|
|
### 3.4 Legacy v4.4 reference = 200 NEWEST bars
|
|
|
|
`SniperGold_SMC_ProPlus_v4_4.mq5` TFBias():
|
|
|
|
```mql5
|
|
int need=200;
|
|
CopyHigh(_Symbol,tf,0,need,h); // 200 NEWEST bars (index 0 = forming)
|
|
ArraySetAsSeries(h,false); // index 0 = oldest
|
|
int s=3, m=need-1; // m excludes the still-forming HTF bar
|
|
```
|
|
|
|
v4.4 = 200 newest bars (the forming bar is inert at index 199) = `tf_bias_asof(forming)`.
|
|
The EA deviates from v4.4 -> **EA adaptation bug**, not legacy semantics.
|
|
|
|
### 3.5 Quantitative reproduction
|
|
|
|
`EA_hat = tf_bias_asof(E_ea - 50)` where `E_ea` = last closed HTF bar at
|
|
`tc = t + 900` (M15 bar close time), lag = `count - need = 250 - 200 = 50`:
|
|
|
|
```text
|
|
D1: mismatch = 0 / 14850 rate = 0.0000
|
|
H4: mismatch = 0 / 14850 rate = 0.0000
|
|
H1: mismatch = 1 / 14850 rate = 0.0001
|
|
```
|
|
|
|
### 3.6 The single H1 exception — cache staleness during a trading gap
|
|
|
|
Row `2026-02-16 21:30`: EA H1 = -1, hat = +1. Cause: H1 data gap
|
|
`21:00 -> 01:00` (daily break). The 20:00 H1 bar closes at 21:00, but
|
|
`e1.Refresh()` only rebuilds when `Bars(symbol,tf)` changes; the next bar only
|
|
appears at 01:00 -> the H1 cache stays ending at 19:00 during the gap. Effect:
|
|
the window endpoint shifts by one extra bar. Only 1/14850 rows flip (bias robustness).
|
|
|
|
---
|
|
|
|
## 4. Root Cause (summary)
|
|
|
|
```text
|
|
EA BTTFBias uses GetBar(count-1-i) -> window = the 200 OLDEST bars of the 250-bar
|
|
cache = HTF bias lagged by 50 bars (D1: 50 days, H4: ~8.3 days, H1: ~2.1 days).
|
|
|
|
Not an input-state mismatch (cache == npz).
|
|
Not legacy semantics (v4.4 uses the 200 newest bars).
|
|
Deterministic & reproducible (14849/14850).
|
|
```
|
|
|
|
Per protocol classification: **B = EA bug**. The secondary effect (gap) is an
|
|
Engine 1 design characteristic (Refresh-on-Bars-change), documented as an edge case.
|
|
|
|
---
|
|
|
|
## 5. Source-of-Truth Decision
|
|
|
|
```text
|
|
SOURCE_OF_TRUTH = CORRECTED SEMANTICS (not EA-as-deployed, not old py-full)
|
|
|
|
Definition:
|
|
f_htf(t) = BTTFBias over the 200 newest CLOSED HTF bars ending at E_ea(t)
|
|
E_ea(t) = last closed HTF bar at tc = t + 900
|
|
= searchsorted(ht, tc - period, 'right') - 1
|
|
= tf_bias_asof(E_ea)
|
|
|
|
Runtime (EA) : fix BTTFBias -> window = the 200 NEWEST bars of the cache
|
|
(GetBar(slot, need-1-i) or equivalent), closed-bar lock kept.
|
|
Training (py) : as-of M15 bar close mapping (E_ea), not as-of open (containing-1).
|
|
|
|
Note on the 1-bar difference vs v4.4 (window ending at the inert forming bar):
|
|
documented as a legacy nuance; the corrected contract uses the last closed
|
|
bar (E_ea) — non-look-ahead, deterministic, one definition for both sides.
|
|
|
|
RETRAINING IS PROHIBITED before full parity (P2.5) is complete.
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Evidence Artifacts
|
|
|
|
```text
|
|
ml/parity/htf_forensic.py tests 4 window hypotheses vs CSV (PY/CLOSED/ANCHOR)
|
|
ml/parity/compare_htf_dump.py EA FULLDUMP vs npz (identical)
|
|
ml/parity/verify_bttf_dump.py bttf_window on the exact EA input
|
|
ml/parity/verify_bttf_rootcause.py EA_hat = tf_bias_asof(E_ea-50) -> 0/0/1 mismatches
|
|
ml/parity/probe_h4_window.py H4 window search (all -1 from npz)
|
|
ml/parity/check_h1_gap.py H1 21:00-01:00 gap (cache staleness)
|
|
Experts/AlgoForge_BTTF_Isolate.mq5 verbatim BTTFBias isolate + trace
|
|
Experts/AlgoForge_Backtest_Baseline.mq5 (debug mode 3 added, additive)
|
|
Profiles/Tester/AlgoForge_HTFDebug...ini / AlgoForge_BTTF_Isolate...ini
|
|
Tester/Agent-127.0.0.1-3000/MQL5/Files/AlgoForge_htf_debug_XAUUSD_M15.csv (dump)
|
|
```
|
|
|
|
---
|
|
|
|
## 7. P2.1 Status
|
|
|
|
```text
|
|
Root cause : FOUND (B = EA bug, 50-bar lag)
|
|
Source of truth : DECIDED (corrected semantics, tf_bias_asof(E_ea))
|
|
Runtime semantics : 200 newest bars ending at E_ea (BTTFBias fix)
|
|
Training semantics : as-of close tc, tf_bias_asof(E_ea)
|
|
Parity (current) : EA-as-deployed = tf_bias_asof(E_ea-50) -> 0/0/1 mismatches
|
|
(runtime + training fix implemented in P2.5)
|
|
```
|