SniperGold_ML/docs/TAHAP4_PLAN.md

# Algo Forge — Phase 4: Hybrid ML Plan (2-layer LSTM + Regime)

> Status: **SUB-SESSIONS 1–3 COMPLETE — GATE No-Go (2-layer LSTM LOSES)** — 2026-08-21.
> This document is the result of the Phase-4 audit + execution plan. Binding for
> subsequent sub-sessions. Test protocol: `FASE3_BATASAN_UJI.md` (G1–G7).
> **FINAL PHASE-4 DECISION (per protocol G7, one No-Go = done, no negotiation): the
> hybrid 2-layer LSTM + regime ML model DOES NOT pass the deploy gate →
> the baseline `SniperGold_ML.mqh` (MLP 19→12→2, 0.627/0.621) REMAINS the deployed model.
> The hybrid .mqh export is CANCELLED (no model worthy of export).**

---

## 1. Executive Summary

- **±20 years** of data from the HFM feed (symbol XAUUSD, 2-digit, Gold Spot):
  - **D1: 4.842 bars, 2007-11-29 → 2026-08-21 (18.73 years)** — fully dense ✅
  - **M15: 199.386 total bars, BUT dense only since 2018** (2007–2016 ≈ 1 bar/day)
  - Dense M15: **2018-01-01 → 2026-08-21 ≈ 8.6 years ≈ 186.500 bars** (≈3x the 60k baseline) ✅
- **Honest decision**: the "20-year M15" target is NOT available on the HFM feed. M15
  uses the dense 2018+ window; D1 uses the full 18.7 years as HTF context.
  Dukascopy (different feed) is only an experimental fallback — not required.
- **Deploy gate**: still G1–G7 vs the baseline **long 0.6270 / short 0.6207**
  (SniperGold_ML.mqh, MLP 19→12→2, freeze). Comparison MUST be like-for-like:
  the new model is tested on the **same baseline cache** (XAUUSDc 60k, 75/25 split)
  before claiming a win. AUC on new data is reported separately.

## 2. Old Python Pipeline Audit (Shared Projects\SniperGold_ML\)

| File | Content | Phase-4 Relevance |
|---|---|---|
| `train_model.py` (28KB) | 19 SMC features + 24×0.75ATR label + pure-numpy MLP + walk-forward + .mqh export | **Reused** (features/label/harness) |
| `train_lstm.py` (19KB) | 1-layer numpy LSTM (W=32, H=32) + BPTT + bootstrap CI | **Developed** → 2 layers |
| `train_regime.py` (11.5KB) | GARCH-t MLE + 2-state HMM + RV24 (causal) | **Reused** (regime features) |
| `train_regime_gate.py` | Regime as gate/context (Phase 2b, lost) | Reference — do not repeat the gate pattern |
| `train_ensemble.py` | 5-seed MLP ensemble | Multi-seed pattern reused |
| `retrain_deploy.py` | Deploy+rollback+audit log | Export/audit pattern reused |
| `download_data.py` | Old 60k-bar download (XAUUSDc) | Replaced by `download_bars.py` |
| `features_XAUUSDc.npz` | 60k-bar cache, 19 features, Phase-3 freeze | **Read-only** (for the like-for-like gate) |

### 2.1 Baseline features (19) — runtime–training consistency
`htf1_bias, htf2_bias, htf3_bias, swing_trend, internal_trend, chart_bias,
eq_pos_norm, sweep_dir, choch_dir, choch_confirms, eqh_swept, eql_swept,
delta_sign, delta_mag, dist_high_atr, dist_low_atr, mom20_atr, range_atr, confluence`
→ computed from M15 + H1/H4/D1 (HTF bias), ATR-normalized, train-only standardization.

### 2.2 Label & split (baseline)
- Label: `forward 24 bars × 0.75 ATR` (long=+1, short=−1, else 0/dropped).
- Split: walk-forward 75/25 (split_pos = 75% of labeled bars); validation = last 15% of train.
- Anti-lookahead: features at bar t use only data ≤ t; standardization only from train.

### 2.3 AlgoForge Engine 2 (source of NEW features, runtime–training consistent)
- Agents N/C/E/P produce structure/bias/confirmation (stateless, closed-bar lock).
- Candidate new features for Phase 4 (backlog, tested later — NOT in the data sub-session):
  `agent N/C/E/P scores & agreement`, `OB/FVG freshness`, `sweep distance`,
  `premium/discount magnitude`, `displacement strength`. Used only if they raise
  validation AUC (not by assumption).

## 3. Data — Results & Decisions (SUB-SESSION 1 COMPLETE)

### 3.1 Downloads (complete)
- `ml\download_bars.py` → `Files\AlgoForge\Data\XAUUSD_{M15,H1,H4,D1}.{csv,npz}`
- Continuity verification: **0 duplicates, 0 anomalies (>4 days), longest gap 96 hours** (long weekend).

### 3.2 Density (analysis `ml\probe_density.py` / `probe_byyear.py`)
| Year | M15 bars/year | D1 bars/year | Note |
|---|---|---|---|
| 2007–2016 | ~260 | ~260 | M15 ≈ 1 bar/day (midnight UTC) → **unusable for an M15 model** |
| 2017 | 1.490 | 257 | transition |
| 2018–2026 | ~23.500/yr | ~258/yr | **dense M15** (~65 bars/day, hours 2–18 UTC) |

### 3.3 Data decisions (BINDING)
1. **M15 model window**: `2018-01-01` → now (~8.6 years, ~186.500 bars).
   This is the honest replacement for "20 years": the broker feed does not provide denser old M15.
2. **Full 18.7-year D1**: used for HTF bias features (context) + a separate D1-model
   experiment (backlog, optional).
3. **Label**: `24 bars × 0.75 ATR` kept (baseline-consistent) — note: with the new
   M15 density of ~65 bars/day, 24 bars ≈ 6 session hours (not 6 full calendar hours).
   The effect is recorded and verified when features are built (sub-session 2).
4. **Baseline cache `features_XAUUSDc.npz` NOT touched** (Phase-3 deploy freeze;
   used read-only for the like-for-like gate).
5. Symbol: **XAUUSD** (XAUUSDc no longer exists in the terminal). ATR-normalized
   features → price scale (4532 vs 45320) does not change feature interpretation.

## 4. Target Architecture (Phase 4)

```
OHLCV XAUUSD (dense M15 2018+ | D1 18.7y)
   → 19 SMC features (train_model.py, runtime-consistent)
   → causal regime features (GARCH-t sigma/ATR, HMM state prob, RV24/ATR)   [train_regime.py]
   → [optional] Engine 2 features (N/C/E/P) — only if they win in validation
   → 2-layer LSTM (pure numpy, W=32, H1=H2=32, linear→sigmoid 2-head)
   → P(long), P(short) → Platt calibration (deploy) 
   → export AlgoForge_Hybrid.mqh + self-test parity
```

- **Anti-lookahead**: regime fitted only on train; causal filters with fixed
  parameters; features at bar t only ≤ t; train-only standardization; closed-bar lock.
- **2 heads**: long & short (each BCE), same as the 2-head baseline.

## 5. Validation Protocol (G1–G7, BINDING)

- Seeds: **[42, 7, 123, 2024, 999]** (5 seeds; all reported, no cherry-picking).
- Paired bootstrap CI: **2000×** (ΔAUC vs baseline on the same test).
- Walk-forward: **75/25 primary** + **70/30 & 80/20** (G7) — all on new data.
- Like-for-like gate: the best model (chosen from validation) is tested **once** on
  the XAUUSDc baseline test (identical split) → compare with 0.6270/0.6207.
- The test set is touched **ONCE** at the end; tuning decisions ONLY from validation.

| Gate | Condition |
|---|---|
| G1 | unit tests B1–B5 (2-layer LSTM & regime: numerical vs reference) |
| G2 | baseline reproduced (MLP 0.627/0.621) in the same run |
| G3 | dAUC ≥ +0.005 vs baseline (like-for-like) |
| G4 | 95% bootstrap CI does NOT contain 0 |
| G5 | gap AUC_val − AUC_test < 0.05 |
| G6 | inter-seed std < 0.01 & ≥4/5 seeds > baseline |
| G7 | gain holds at 70/30 & 80/20 splits |

**ONE No-Go = done (no negotiation).** Deploy freeze: `SniperGold_ML.mqh`
untouched; Phase-4 deploy output = a NEW file (`AlgoForge_Hybrid.mqh` + integration).

## 6. Sub-Session Plan (split; each sub-session = checkpoint + handoff)

| Sub-session | Content | Output | Status |
|---|---|---|---|
| **1** | Survey + audit + data download + continuity verification + plan | `TAHAP4_PLAN.md`, `Files\AlgoForge\Data\*.npz` (19y D1, 8.6y M15) | ✅ COMPLETE |
| **2 (THIS)** | `build_features.py`: 19 SMC features + causal regime (GARCH-t + RV24 + regime score; HMM/MSM collapsed → replaced by proxy) + label + new npz cache + unit tests B1–B5 | `features_XAUUSD.npz` + B1–B5 PASS | ✅ COMPLETE |
| 3 | `train_hybrid.py`: 2-layer LSTM + 2-head + unit tests B1–B5 + gate G1–G7 like-for-like | audit log + Go/No-Go decision | ✅ COMPLETE — **No-Go** |
| 4 | Export best model → `AlgoForge_Hybrid.mqh` + parity | — | ❌ CANCELLED (no model passed the gate) |
| 5 | Integration + backtest + documentation + Forge push | release (baseline stays) | ⏳ (Phase 5 is for the baseline, not hybrid) |

### Sub-session 3 results (gate — FINAL decision)
- `ml\train_hybrid.py`: **2-layer** LSTM (W=32, H1=H2=32) + **2-target** long/short
  head, pure numpy; unit tests **B1–B5 PASS** (2-layer forward == 2x 1-layer,
  BPTT gradcheck rel=2.7e-5, determinism, anti-lookahead, overfit AUC=1.0).
- **Like-for-like gate on the XAUUSDc cache (identical 75/25 baseline split), 5 seeds
  [42,7,123,2024,999], 2000× bootstrap CI** (log `ml\TAHAP4_HYBRID.log`):

| | Hybrid 2-layer (ens) | Baseline target | dAUC | G3 | G4 | G6 |
|---|---|---|---|---|---|---|
| LONG | 0.6074 (mean 0.5898±0.0066) | 0.6270 | −0.0196 | ❌ | ❌ (negative CI) | 0/5 seeds ❌ |
| SHORT | 0.6073 (mean 0.5900±0.0056) | 0.6207 | −0.0134 | ❌ | ❌ (negative CI) | 0/5 seeds ❌ |

- Additional comparison: MLP reproduction on the same split = LONG 0.6795 / SHORT 0.6560
  (consistent with Phase 3); hybrid−MLP dAUC = **−0.0723 / −0.0487**, 95% CI ALL
  NEGATIVE → hybrid is significantly WORSE.
- **CONCLUSION: 2-layer LSTM + regime does NOT beat the baseline; GATE No-Go →
  protocol complete, no negotiation.** Consistent with old evidence (Phase 3: LSTM lost
  on 60k bars). Full walk-forward on new data NOT run (the gate was already decisive;
  §7 rule to stop when >2x estimate / No-Go).

### Sub-session 2 results (detail)
- `ml\build_features.py` (reuses `train_model.py`/`train_regime.py`; runtime–training consistent features):
  - 19 SMC features identical to baseline + `24×0.75ATR` label, dense 2018+ window.
  - Causal regime: **t-GARCH(1,1)** fitted with classic bounds + 0.5% winsor → interior
    (alpha=0.0615, beta=0.9230, persistence=0.9844, nu=7.12); causal sigma (unconditional
    init); z standardization train-only; **RV24 from raw returns**.
  - **2-state Gaussian HMM (MSM) COLLAPSES on this feed** (P(high)=1 on z, |z|,
    log(RV24), RV24 — evidence `diag_hmm.py`; consistent with the Phase-2 lesson). Replaced
    by a **causal logistic regime score** (RV24 vs train median/MAD): mean 0.585,
    std 0.229, test mean 0.701 (2024+ is indeed volatile). HONEST DECISION recorded;
    the final effect is still determined by the AUC gate (sub-session 3).
- `Files\AlgoForge\Data\features_XAUUSD.npz`: F(197032,19), F2(197032,6), label,
  ATR, close, time, split_bar=148470, split_pos=115631, meta JSON. 0 NaN/Inf,
  0 constant columns; bull 80.939 / bear 73.236 (ratio 1.105).
- **24-bar forward = median 6.00 session hours** (p10=6, p90=7; >8 hours 7%) —
  the label ≈ 6 session hours, per note §3.3.2.
- Unit tests **B1–B5 PASS** (numerical ATR, streaming feature anti-lookahead,
  forward label, causal GARCH sigma, causal HMM filter + valid probs).

## 7. Risks & Honest Notes

1. **M15 ≠ 20 years** (HFM feed) — decision: 8.6-year dense window + 18.7-year D1.
   This is HONESTLY recorded; do not claim "20-year M15".
2. **LSTM/regime lost before on 60k bars** — not assumed to win; the like-for-like
   AUC gate decides. Regime is used as a feature/context, not a gate.
3. **XAUUSD vs XAUUSDc** — different feeds; the like-for-like gate uses the same
   XAUUSDc cache for a fair comparison; the deploy model is trained on XAUUSD.
4. **Label density**: 24 new M15 bars ≈ 6 session hours (median 6.00 hours; verified
   in sub-session 2). If the impact is large, a wall-clock label option is recorded as an alternative.
5. **Compute estimate**: 186k bars × W=32 × H=32 2-layer LSTM × 5 seeds × 3 splits
   — pure numpy can take 1–4 hours/seed; limit epoch_max=30, patience=10 (protocol).
   If >2x the estimate → stop & report (Phase-3 rule §7).
6. **2-state Gaussian HMM/MSM collapses on this feed** (evidence `diag_hmm.py`): regime
   features use t-GARCH (valid) + a causal logistic RV24 regime score (proxy, not MSM).
   This is an honest deviation from "MSM" in the specification — recorded, not hidden.
   The gate still decides whether regime helps direction.
7. **GARCH MLE is fragile**: classic bounds + 0.5% winsor are required for an interior
   fit (alpha 0.06/beta 0.92/nu 7); without winsor the MLE falls to an ARCH-heavy
   boundary solution (evidence `diag_garch*.py`). These parameters are EMBEDDED in the cache meta.