SniperGold_ML/docs/PROGRESS.md

Algo Forge — PROGRESS (session log)

Session 2026-08-21 — Phases 0 & 1 complete

Phase 0 — Project Structure + Attribution ✅

• Folders created:
  • MQL5\Include\AlgoForge\ (AF_Attribution.mqh, AF_Defines.mqh, AF_Engine1_MTFData.mqh)
  • MQL5\Indicators\AlgoForge\ (empty — Engine-3 reserve)
  • MQL5\Experts\AlgoForge_Engine1_UnitTest.mq5 (unit test; must be at the root MQL5\Experts because the tester tool only recognizes EAs whose .mq5 source is at the root)
  • Shared Projects\AlgoForge\ (README.md, DESIGN.md, PROGRESS.md)
• Attribution header (MANDATORY): source https://www.mql5.com/en/code/75466, Waseem Shahrukh, baseline v4.4 AUC 0.627/0.621 — in AF_Attribution.mqh.

Phase 1 — Engine 1 (MTF Data Collector + Cache) ✅

• AF_Engine1_MTFData.mqh: class AFEngine1MTF + struct AFBar.
• Features: index cache Bars() → CopyRates once per new TF bar (anti-freeze); closed-bar lock (IsBarClosed); chart-TF independent (PERIOD_CURRENT rejected); retry throttle when history is not ready; reentrancy guard; HistoryCalls statistics.
• API: Register / Refresh / IsReady / Count / GetBar / GetBarByTime / FindBarIndex / Open/High/Low/Close/Time/TickVolume / ATR / HistoryCalls / RefreshCount / LastBars.

Compilation

• AlgoForge_Engine1_UnitTest.mq5 → 0 errors, 0 warnings (build 6093, AVX512 target).

Unit Test (Strategy Tester, XAUUSD)

Run	Range	Model	Result
Full	2026.08.01–08.19	every tick	PASS=115070 FAIL=0 (5.39M ticks, 4.16 s)
Verification	2026.08.01–08.03	every tick	PASS=8870 FAIL=0 (clean log)

• Anti-freeze proven: historyCalls = 1 per new bar per TF (M1=17928, M5=3589, M15=1197, H1≈300 in the full run).
• Checks: T1 ready, T2 valid OHLC, T3 strict time order, T4 closed-bar lock, T5 anti-freeze, T6 count<=Bars, T7 chart-TF independence (static: 0 uses of _Period; PERIOD_CURRENT only in the Register rejection).

Constraints & Solutions (recorded)

1. MQL5: array-element references not allowed → use direct index access.
2. #property version "0.1.0" triggers a Market warning → use X.YY format ("1.00").
3. The MCP tester only accepts EAs whose .mq5 is at the root MQL5\Experts (subfolders and .ex5-only copies rejected) → unit-test sources placed at the root.
4. The "1-min OHLC" tester model rejects sub-chart TF requests → use the "every tick" model (XAUUSD has tick data; M1/M5/M15/H1 available).
5. "Array out of range" bug: bars[] has maxBars capacity but CopyRates fetches maxBars+1 → n capped with MathMin(got-skip, maxBars).
6. Per-M1-bar stat log too large → stats printed only when InpVerbose=true; per-slot final summary in OnTester.

Session 2026-08-21 (continued) — PUBLICATION ✅

• New repository: https://forge.mql5.io/chiki2bum2/SniperGold_ML (public, main branch).
• Contents: README (mandatory attribution: 75466/Waseem Shahrukh), LICENSE, .gitignore, MQL5\Include\AlgoForge*.mqh (3), MQL5\Experts\AlgoForge_Engine1_UnitTest.mq5, docs\DESIGN.md + PROGRESS.md — commit f4dc7cd, 9 files.
• Process: Forgejo API (create repo) + git push (user access token).
• Publication source package: Shared Projects\AlgoForge\publish\AlgoForge\ (local git repo).

Session 2026-08-21 (this session) — Phase 2 COMPLETE ✅

Phase 2 — Engine 2 (4 Independent Signal Agents N/C/E/P + Fuzzy + Aggregator)

• New files:
  • MQL5\Include\AlgoForge\AF_Engine2_Agents.mqh — struct AFSignalOut, fuzzy helpers (AF_MF_Tri/Trap, AFFuzzyEval), structure analysis (fractal pivots, trend+clarity, CHoCH, BOS, sweep, displacement, OB, FVG, PA patterns) and 4 agents: AFAgentNarrative (N), AFAgentContext (C), AFAgentEntry (E), AFAgentPriceAction (P) — all stateless, read Engine 1 only through one slot.
  • MQL5\Include\AlgoForge\AF_Engine2_Aggregator.mqh — AFAggOut, AFAggregator (2-pass dynamic weights + agreement boost), facade AFEngine2Signals.
  • MQL5\Experts\AlgoForge_Engine2_UnitTest.mq5 — unit tests T1–T6.
• AF_Defines.mqh extended: Engine-2 constants (AF_E2_, AF_AGG_).
• Per-agent timeframe input: InpHtfS1..S4 = N, C, E, P.
• Data source: Engine 1 only (AFEngine1MTF) — no direct History calls from Engine 2; closed-bar lock guaranteed by Engine 1.

Compilation

• AlgoForge_Engine2_UnitTest.mq5 → 0 errors, 0 warnings (build 6093, AVX512).
• AlgoForge_Engine1_UnitTest.mq5 (regression, AF_Defines changed) → 0/0.

Unit Test (Strategy Tester, XAUUSD, every tick)

Run	Range	Result
Full (initial code)	2026.08.01–08.20	PASS=21534 FAIL=0
Verbose	2026.08.01–08.11	PASS=9942 FAIL=0 (T1–T6 shown explicitly)
Full (final code)	2026.08.01–08.20	PASS=21534 FAIL=0

• T1: all 4 agent outputs valid (buy/sell/bias/conf/dir consistent).
• T2: independence — changing agent X's input (tamper slot, different TF) does NOT change other agents' outputs; determinism — same input → identical output.
• T3: closed-bar lock on all slots. T4: non-repaint — identical output within the same bar.
• T5: aggregator valid (dir/bias/score/confidence/entry-SL-TP).
• T6: synthetic aggregator, 5 cases (neutral→WAIT, buy→BUY, sell→SELL, N-only→BUY, balanced N vs C→WAIT).
• Engine-1 anti-freeze maintained: historyCalls M5=3589, M15=1197, H1=300 (same as Phase 1).
• Real signal examples (verbose): Narrative "uptrend -premium zone", Context "-premium zone", Entry "+in zone (OB/FVG)", PriceAction "+2-bar momentum (trending)", aggregate BUY/WAIT/SELL with ATR-based entry/SL/TP.

Constraints & Solutions (this session)

7. replace_text_in_file multi-line edits fail (CRLF) → single-line edits (2 statements on 1 line when necessary).
8. The independence test needs a tamper slot ≥ 80 bars → choose dense TFs (M6/M12/M20/M30).
9. g_indepDone was originally set true even when T2 was skipped → T2 now returns bool and is only marked done when it actually runs (auto-retry).

Next checklist

• Phase 3: Engine 3 — display like the original 75466 code (reads Engine 1–2 output).
• Phase 4: ML LSTM + Regime (MSM/GARCH) — COMPLETE with GATE No-Go.
• Phase 5: final Algo Forge verification & publication (75466 attribution).

Phase 2 Publication (complete)

• Commit cfcdf23 "Algo Forge Phase 2: Engine 2 (4 independent signal agents N/C/E/P + dynamic-weight fuzzy + separate aggregator)" — already in the local publish repo.

Session 2026-08-21 (this session) — Phase 3 COMPLETE ✅

Phase 3 — Engine 3 (Display: pure render, reads Engine 1 & 2 output only)

• New files:
  • MQL5\Include\AlgoForge\AF_Engine2_Display.mqh — display-computation layer in Engine 2: struct AFDisplayData, AFDispZone, AFDispLine, AFDispPivot + AF_BuildDisplayData() (structure, swing points, OB, FVG, premium/discount, MTF PDH/PDL/PWH/PWL levels, agent bias) — uses the same analysis helpers as the agents.
  • MQL5\Include\AlgoForge\AF_Engine3_Render.mqh — pure renderer: AFRenderCfg, AFR_DefaultCfg, AFR_DrawAll(), AFR_Clear(); lines/text/boxes/panel; dashboard mimicking the original 75466 layout (bias, structure, liquidity, context, key levels, N/C/E/P alignment, trade setup, legend). NO analysis computation.
  • MQL5\Indicators\AlgoForge\AF_Engine3_Display.mq5 — consumer indicator (0 buffers, object-only): draws only when a new closed bar appears on the display TF (g_lastDispBar).
  • MQL5\Experts\AlgoForge_Engine3_UnitTest.mq5 — unit tests T1–T5 (prefix AFTEST3).
• AF_Defines.mqh extended: Engine-3 constants (AF_E3_*: window, zone counts, min OB/FVG sizes, PD bands, D1/W1 capacity, object prefix AF3_).
• Principle: Engine 3 READS ONLY Engine 1 & 2 output; anti-freeze stays in Engine 1.

Compilation

• AlgoForge_Engine3_UnitTest.mq5 → 0 errors, 0 warnings (build 6093, AVX512).
• AF_Engine3_Display.mq5 (indicator) → 0 errors, 0 warnings.
• Regression: AlgoForge_Engine1_UnitTest.mq5 & AlgoForge_Engine2_UnitTest.mq5 (AF_Defines.mqh changed) → 0/0.

Unit Test (Strategy Tester, XAUUSD M15, every tick)

Run	Range	Result
Full	2026.08.01–08.20	PASS=21528 FAIL=0 (finished 10:14:42; 506 AFTEST3 log lines)

• T1: display data valid (structure/eqPos/range/PDH-PDL-PWH-PWL must be >0 when the D1/W1 slots are ready).
• T2: non-repaint — identical display data within the same bar.
• T3: closed-bar lock on all slots (N/C/E/P/DISP).
• T4: render creates 214–232 objects per bar (prefix AF3_) + the P_BIAS dashboard panel exists.
• T5: render deterministic — re-rendering the same bar → same object count.
• Real examples: structure Range/Bullish/Bearish; eqPos Premium/Equilibrium/Discount; HH/HL/LH/LL varying (21–22 / 17–18 / 20–21 / 17–18); piv=32, lines=43–45, ob=5, fvg=6.

Constraints & Solutions (this session)

10. ObjectCreate in the Strategy Tester: on this build, chart objects are created even with Visual=0 (object probe succeeds → T4/T5 run, 232 objects); when unsupported, the probe marks an automatic SKIP (not a failure) — see DESIGN.md §17.

Phase 3 Publication (complete)

• Commit 92f1fbc "Algo Forge Phase 3: Engine 3 (pure display render - structure/zones/ signals/dashboard, reads Engine 1-2 output only) + AF_Engine3_Display indicator + unit test (PASS=21528 FAIL=0)" — 9 files, +1681/−79 — pushed to https://forge.mql5.io/chiki2bum2/SniperGold_ML (main).

Phase 4 — Hybrid ML (SUB-SESSION 1: DATA + SURVEY ✅, 2026-08-21)

• Handoff + initial prompt: Shared Projects\SniperGold_ML\REFACTOR_ALGOFORGE.md §14.
• Baseline: MQL5\Include\SniperGold_ML.mqh (MLP 19→12→2, AUC 0.627/0.621) — deploy freeze, do not change during testing.
• Binding protocol: Shared Projects\SniperGold_ML\FASE3_BATASAN_UJI.md (G1–G7).
• Estimate: ≥150–250k tokens, multi-session; split into sub-sessions (data → regime → LSTM → validation → export → integration), each ending with a checkpoint + handoff.

Sub-session 1 (COMPLETE) — Survey + Audit + ~19 years of data

Old pipeline audit (Shared Projects\SniperGold_ML): train_model.py (19 SMC features, 24×0.75ATR label, MLP, walk-forward, .mqh export), train_lstm.py (1-layer numpy LSTM, W=32 H=32, bootstrap CI), train_regime.py (GARCH-t + 2-state HMM + causal RV24), train_regime_gate.py (regime as gate — proven to lose), train_ensemble.py (5-seed), retrain_deploy.py (deploy+rollback+audit). Details: docs\TAHAP4_PLAN.md §2.

Data (downloaded + continuity verified):

• ml\download_bars.py → Files\AlgoForge\Data\XAUUSD_{M15,H1,H4,D1}.{csv,npz}.
• Verification: 0 duplicates, 0 anomalies (>4 days), longest gap 96 hours (long weekend).

TF	Bars	Range	Decision
D1	4.842	2007-11-29 → 2026-08-21 (18.73 y)	✅ fully dense — HTF context
M15	199.386 total	2007–2016 ≈ 1 bar/day	⚠️ NOT usable for an M15 model
M15	~186.500	2018-01-01 → now (8.6 y)	✅ DENSE window (≈3x the 60k baseline)

Binding decisions (honest):

1. The "20-year M15" target is NOT available on the HFM feed → M15 uses the dense 2018+ window (8.6 years); D1 18.7 years for HTF bias. Dukascopy is only an experiment (different feed).
2. Label stays 24 bars × 0.75 ATR; note: the new density ~65 bars/day → 24 bars ≈ 6 session hours; verified when building features (sub-session 2).
3. Baseline cache features_XAUUSDc.npz read-only (like-for-like gate vs 0.627/0.621).
4. Symbol: XAUUSD (2-digit) — XAUUSDc no longer exists in the terminal.
5. Architecture: 19 SMC features + causal regime (GARCH/HMM/RV) → 2-layer LSTM → 2-head P(long)/P(short); Engine-2 features (N/C/E/P) as backlog candidates, only if they win validation. Gates G1–G7 binding; test touched once.

New files: ml\download_bars.py, ml\probe_density.py, ml\probe_byyear.py, docs\TAHAP4_PLAN.md.

Sub-session 2 (COMPLETE) — Features + Regime + Label

ml\build_features.py (reuses train_model.py/train_regime.py — runtime–training consistency):

• 19 SMC features identical to baseline + 24×0.75ATR label; dense 2018+ window (2017 warmup for structure/ATR/HTF lookbacks).
• Causal regime (strict anti-lookahead):
  • t-GARCH(1,1) interior with classic bounds + 0.5% return winsor: alpha=0.0615 beta=0.9230 persist=0.9844 nu=7.12 (without winsor the MLE falls to an ARCH-heavy boundary solution; evidence diag_garch.py/diag_garch2.py).
  • causal sigma (unconditional init — improvement over train_regime.py which used np.var(r) over the whole array = mild lookahead at bar 0).
  • z standardization train-only; RV24 from RAW returns.
  • 2-state Gaussian HMM (MSM) COLLAPSES on this feed (P(high)=1 on z, |z|, log(RV24), RV24 — evidence diag_hmm.py; consistent with the Phase-2 lesson). Replaced by a causal logistic regime score (RV24 vs train median/MAD): mean 0.585, std 0.229, test mean 0.701 (2024+ is indeed volatile). The honest deviation is recorded in TAHAP4_PLAN §7.6; the final effect is decided by the AUC gate.
• Cache Files\AlgoForge\Data\features_XAUUSD.npz: F(197032,19), F2(197032,6), label, ATR, close, time, split_bar=148470, split_pos=115631, meta JSON. 0 NaN/Inf; 0 constant columns.
• Label: 154.175 samples (bull 80.939 / bear 73.236, ratio 1.105); train bull 0.516, test bull 0.551. 24-bar forward = median 6.00 session hours (p10=6, p90=7).
• Unit tests B1–B5 PASS (numerical ATR, streaming-feature anti-lookahead prefix==full, forward label, causal GARCH sigma, causal HMM filter + valid probs).
• Verification: ml\verify_cache.py (shape/NaN/variation/label density) OK.
• Baseline features_XAUUSDc.npz & SniperGold_ML.mqh NOT touched.

Next (sub-session 3): train_hybrid.py — 2-layer LSTM (W=32, H=32) + 2-target long/short head; walk-forward 75/25 (+70/30, 80/20); 5 seeds [42,7,123,2024,999]; 2000× bootstrap CI; gates G1–G7 vs baseline 0.627/0.621 (like-for-like on the XAUUSDc cache) — test touched once.

Sub-session 3 (COMPLETE) — 2-layer LSTM + Gate → No-Go

ml\train_hybrid.py (pure numpy):

• 2-layer LSTM (W=32, H1=H2=32) + 2-target long/short head (BCE mean B×2); 2-layer BPTT (L2 → dX → L1). Unit tests B1–B5 PASS: forward == 2×1-layer (err 0), gradcheck rel=2.7e-5, determinism, anti-lookahead, overfit AUC=1.0.
• Calibration: 6.6 s/epoch (5.1k train) → gate estimate ~53 min.

LIKE-FOR-LIKE GATE (XAUUSDc 60k cache, 75/25 split, 5 seeds, 2000× bootstrap) (log: ml\TAHAP4_HYBRID.log):

• Hybrid ens: LONG=0.6074 (mean 0.5898±0.0066) · SHORT=0.6073 (mean 0.5900±0.0056)
• vs target 0.6270/0.6207 → dAUC −0.0196 / −0.0134 → G3 FAILED
• vs MLP reproduction (0.6795/0.6560) → dAUC −0.0723 / −0.0487, 95% CI all NEGATIVE → G4 FAILED (significantly worse); G6 0/5 seeds win.
• DECISION: GATE No-Go → protocol complete (one No-Go = done, no negotiation). Consistent with Phase 3 (LSTM lost on 60k bars). Full walk-forward on new data NOT run (gate decisive; stop rule).
• Hybrid .mqh export CANCELLED (no model passed the gate). The baseline SniperGold_ML.mqh (MLP 0.627/0.621) REMAINS the deployed model (freeze kept).

Phase-4 status: data ✅ → regime/features/label ✅ → LSTM ✅ → validation/gate ✅ (No-Go) → export ❌ (cancelled) → integration — baseline stays. Proceed to Phase 5 (baseline verification & publication / Algo Forge release) — see REFACTOR §20.

Phase 3 Publication (complete, reference)

• Commit 92f1fbc "Algo Forge Phase 3: Engine 3 (pure display render - structure/zones/ signals/dashboard, reads Engine 1-2 output only) + AF_Engine3_Display indicator + unit test (PASS=21528 FAIL=0)" — 9 files, +1681/−79 — pushed to https://forge.mql5.io/chiki2bum2/SniperGold_ML (main).

Phase 4 Sub-session 1 Publication (complete)

• Commit 33fcb37 "Algo Forge Phase 4 sub-session 1: survey + old ML pipeline audit + XAUUSD 19y data ..." — 5 files, +471 — pushed (main).

Phase 5 — BASELINE / ALGO FORGE VERIFICATION & PUBLICATION (COMPLETE, 2026-08-21)

1. 0/0 compile of the whole package + unit-test regression

• Compile 0 errors / 0 warnings (build 6093, AVX512): AlgoForge_Backtest_Baseline.mq5, AlgoForge_Engine1/2/3_UnitTest.mq5, AF_Engine3_Display.mq5.
• Strategy Tester unit-test regression (XAUUSD M15, every tick, 2026.08.01–08.20):

Engine	PASS	FAIL	Status
Engine 1	115070	0	identical to the Phase-1 baseline
Engine 2	21534	0	identical to the Phase-2 baseline
Engine 3	21528	0	identical to the Phase-3 baseline

2. Net-of-spread baseline backtest (internal features + CSV, NOT iCustom)

• New EA MQL5\Experts\AlgoForge_Backtest_Baseline.mq5:
  • 19 SMC features computed INTERNALLY (identical to v4.4 ComputeMLFeatures + dependencies), data source ONLY Engine 1 (AFEngine1MTF; closed-bar lock, anti-freeze).
  • Calls the frozen model SniperGold_ML.mqh (MLP 19→12→2, v20260821_2head).
  • Mode 0 = per-bar M15 prob CSV (parity/AUC via ml\backtest_eval.py); Mode 1 = net-of-spread trading in the tester (market OrderSend, actual spread).

Mode 0 (CSV/AUC) — XAUUSD M15, 2026.01.01–08.20, every tick, 14.850 bars:

Metric	Value	Baseline freeze
AUC LONG	0.5305	0.6270
AUC SHORT	0.5487	0.6207
Precision LONG @0.60	0.5352 (n=4454)	—
Precision SHORT @0.60	0.5292 (n=7010)	—

• Poor calibration: prob long 0.65+ → actual frequency only 0.545.
• HONEST CONCLUSION: on the XAUUSD feed (not the XAUUSDc training feed), the baseline model probabilities are NOT calibrated and the AUC is far lower. Evidence that the 0.627/0.621 metrics apply only to the training feed/split → do not claim an edge on another feed without a like-for-like gate.

Mode 1 (net-of-spread trade) — XAUUSD M15, 2026.05.01–08.20, every tick, lot 0.01, deposit 10k, thresh L/S=0.60, SL=1.0ATR, TP=1.5ATR, maxHold=24:

Stat	Value
Trades	308
Net profit	+1125.32
Max drawdown	1188.18
Profit factor	1.32

• Honest note: some signals failed to execute (requote ret=10018, no retry); the positive result is NOT significant (runtime AUC 0.53/0.55; small sample; model not calibrated on this feed) → reported as-is, not an edge claim.

3. Documentation & publication

• README/DESIGN/PROGRESS updated; AlgoForge_Backtest_Baseline.mq5 + tester config + ml\backtest_eval.py added to the repo; mandatory 75466/Waseem Shahrukh attribution kept in all files.
• Pushed to https://forge.mql5.io/chiki2bum2/SniperGold_ML (main) — see REFACTOR_ALGOFORGE.md §21.

Agent base-TF change — HARDCODED H4/M30/M15/M3 (COMPLETE, 2026-08-21)

• Decision: manual inputs InpHtfS1..S4 removed; agent TFs hardcoded via new macros in AF_Defines.mqh: AF_E2_TF_S1=H4 (Narrative), AF_E2_TF_S2=M30 (Context), AF_E2_TF_S3=M15 (Entry), AF_E2_TF_S4=M3 (PriceAction). Top-down analysis basis: H4→M30→M15→M3.
• Wording correction: S4 was originally mistyped M5 → corrected to M3 (user request).
• Files changed: AF_Defines.mqh, AF_Engine3_Display.mq5 (indicator), AlgoForge_Engine2_UnitTest.mq5, AlgoForge_Engine3_UnitTest.mq5 (independence-test tamper slots: H1/M20/M12/M6 — different from the base), Engine-2/3 tester configs.
• Compile 0/0 (indicator + 2 unit tests + Engine-1 regression).
• Re-test (XAUUSD M15, every tick, 2026.08.01–08.20, new base):
  • Engine 2: PASS=35886 FAIL=0 (final: N=H4 hist=79 · C=M30 hist=599 · E=M15 hist=1197 · P=M3 hist=5981).
  • Engine 3: PASS=35880 FAIL=0 (DISP=M15, D1/W1 OK).
  • (PASS rose vs 21534/21528 because M3 adds more new bars per range; FAIL stays 0, no SKIP.)
• Anti-freeze maintained: historyCalls = 1 per new bar per TF.