# Algo Forge — PROGRESS (session log) ## Session 2026-08-21 — Phases 0 & 1 complete ### Phase 0 — Project Structure + Attribution ✅ - Folders created: - `MQL5\Include\AlgoForge\` (AF_Attribution.mqh, AF_Defines.mqh, AF_Engine1_MTFData.mqh) - `MQL5\Indicators\AlgoForge\` (empty — Engine-3 reserve) - `MQL5\Experts\AlgoForge_Engine1_UnitTest.mq5` (unit test; must be at the root `MQL5\Experts` because the tester tool only recognizes EAs whose `.mq5` source is at the root) - `Shared Projects\AlgoForge\` (README.md, DESIGN.md, PROGRESS.md) - Attribution header (MANDATORY): source `https://www.mql5.com/en/code/75466`, Waseem Shahrukh, baseline v4.4 AUC 0.627/0.621 — in `AF_Attribution.mqh`. ### Phase 1 — Engine 1 (MTF Data Collector + Cache) ✅ - `AF_Engine1_MTFData.mqh`: class `AFEngine1MTF` + struct `AFBar`. - Features: index cache `Bars()` → `CopyRates` once per new TF bar (anti-freeze); closed-bar lock (`IsBarClosed`); chart-TF independent (`PERIOD_CURRENT` rejected); retry throttle when history is not ready; reentrancy guard; HistoryCalls statistics. - API: `Register / Refresh / IsReady / Count / GetBar / GetBarByTime / FindBarIndex / Open/High/Low/Close/Time/TickVolume / ATR / HistoryCalls / RefreshCount / LastBars`. ### Compilation - `AlgoForge_Engine1_UnitTest.mq5` → **0 errors, 0 warnings** (build 6093, AVX512 target). ### Unit Test (Strategy Tester, XAUUSD) | Run | Range | Model | Result | |---|---|---|---| | Full | 2026.08.01–08.19 | every tick | **PASS=115070 FAIL=0** (5.39M ticks, 4.16 s) | | Verification | 2026.08.01–08.03 | every tick | **PASS=8870 FAIL=0** (clean log) | - Anti-freeze proven: `historyCalls` = 1 per new bar per TF (M1=17928, M5=3589, M15=1197, H1≈300 in the full run). - Checks: T1 ready, T2 valid OHLC, T3 strict time order, T4 closed-bar lock, T5 anti-freeze, T6 count<=Bars, T7 chart-TF independence (static: 0 uses of `_Period`; `PERIOD_CURRENT` only in the Register rejection). ### Constraints & Solutions (recorded) 1. **MQL5: array-element references not allowed** → use direct index access. 2. **`#property version "0.1.0"` triggers a Market warning** → use `X.YY` format ("1.00"). 3. **The MCP tester only accepts EAs whose `.mq5` is at the root `MQL5\Experts`** (subfolders and `.ex5`-only copies rejected) → unit-test sources placed at the root. 4. **The "1-min OHLC" tester model rejects sub-chart TF requests** → use the "every tick" model (XAUUSD has tick data; M1/M5/M15/H1 available). 5. **"Array out of range" bug**: `bars[]` has `maxBars` capacity but `CopyRates` fetches `maxBars+1` → `n` capped with `MathMin(got-skip, maxBars)`. 6. **Per-M1-bar stat log too large** → stats printed only when `InpVerbose=true`; per-slot final summary in `OnTester`. ## Session 2026-08-21 (continued) — PUBLICATION ✅ - New repository: **https://forge.mql5.io/chiki2bum2/SniperGold_ML** (public, main branch). - Contents: README (mandatory attribution: 75466/Waseem Shahrukh), LICENSE, .gitignore, MQL5\Include\AlgoForge\*.mqh (3), MQL5\Experts\AlgoForge_Engine1_UnitTest.mq5, docs\DESIGN.md + PROGRESS.md — commit `f4dc7cd`, 9 files. - Process: Forgejo API (create repo) + git push (user access token). - Publication source package: `Shared Projects\AlgoForge\publish\AlgoForge\` (local git repo). ## Session 2026-08-21 (this session) — Phase 2 COMPLETE ✅ ### Phase 2 — Engine 2 (4 Independent Signal Agents N/C/E/P + Fuzzy + Aggregator) - New files: - `MQL5\Include\AlgoForge\AF_Engine2_Agents.mqh` — struct `AFSignalOut`, fuzzy helpers (`AF_MF_Tri/Trap`, `AFFuzzyEval`), structure analysis (fractal pivots, trend+clarity, CHoCH, BOS, sweep, displacement, OB, FVG, PA patterns) and 4 agents: `AFAgentNarrative` (N), `AFAgentContext` (C), `AFAgentEntry` (E), `AFAgentPriceAction` (P) — all **stateless, read Engine 1 only through one slot**. - `MQL5\Include\AlgoForge\AF_Engine2_Aggregator.mqh` — `AFAggOut`, `AFAggregator` (2-pass dynamic weights + agreement boost), facade `AFEngine2Signals`. - `MQL5\Experts\AlgoForge_Engine2_UnitTest.mq5` — unit tests T1–T6. - `AF_Defines.mqh` extended: Engine-2 constants (AF_E2_*, AF_AGG_*). - Per-agent timeframe input: `InpHtfS1..S4 = N, C, E, P`. - Data source: **Engine 1 only** (`AFEngine1MTF`) — no direct History calls from Engine 2; closed-bar lock guaranteed by Engine 1. ### Compilation - `AlgoForge_Engine2_UnitTest.mq5` → **0 errors, 0 warnings** (build 6093, AVX512). - `AlgoForge_Engine1_UnitTest.mq5` (regression, AF_Defines changed) → **0/0**. ### Unit Test (Strategy Tester, XAUUSD, every tick) | Run | Range | Result | |---|---|---| | Full (initial code) | 2026.08.01–08.20 | **PASS=21534 FAIL=0** | | Verbose | 2026.08.01–08.11 | **PASS=9942 FAIL=0** (T1–T6 shown explicitly) | | Full (final code) | 2026.08.01–08.20 | **PASS=21534 FAIL=0** | - **T1**: all 4 agent outputs valid (buy/sell/bias/conf/dir consistent). - **T2**: independence — changing agent X's input (tamper slot, different TF) does NOT change other agents' outputs; determinism — same input → identical output. - **T3**: closed-bar lock on all slots. **T4**: non-repaint — identical output within the same bar. - **T5**: aggregator valid (dir/bias/score/confidence/entry-SL-TP). - **T6**: synthetic aggregator, 5 cases (neutral→WAIT, buy→BUY, sell→SELL, N-only→BUY, balanced N vs C→WAIT). - Engine-1 anti-freeze maintained: historyCalls M5=3589, M15=1197, H1=300 (same as Phase 1). - Real signal examples (verbose): Narrative "uptrend -premium zone", Context "-premium zone", Entry "+in zone (OB/FVG)", PriceAction "+2-bar momentum (trending)", aggregate BUY/WAIT/SELL with ATR-based entry/SL/TP. ### Constraints & Solutions (this session) 7. `replace_text_in_file` multi-line edits fail (CRLF) → single-line edits (2 statements on 1 line when necessary). 8. The independence test needs a tamper slot ≥ 80 bars → choose dense TFs (M6/M12/M20/M30). 9. `g_indepDone` was originally set `true` even when T2 was skipped → T2 now returns bool and is only marked done when it actually runs (auto-retry). ### Next checklist - [x] Phase 3: Engine 3 — display like the original 75466 code (reads Engine 1–2 output). - [x] Phase 4: ML LSTM + Regime (MSM/GARCH) — COMPLETE with GATE No-Go. - [x] Phase 5: final Algo Forge verification & publication (75466 attribution). ## Phase 2 Publication (complete) - Commit `cfcdf23` "Algo Forge Phase 2: Engine 2 (4 independent signal agents N/C/E/P + dynamic-weight fuzzy + separate aggregator)" — already in the local publish repo. ## Session 2026-08-21 (this session) — Phase 3 COMPLETE ✅ ### Phase 3 — Engine 3 (Display: pure render, reads Engine 1 & 2 output only) - New files: - `MQL5\Include\AlgoForge\AF_Engine2_Display.mqh` — display-computation layer in Engine 2: struct `AFDisplayData`, `AFDispZone`, `AFDispLine`, `AFDispPivot` + `AF_BuildDisplayData()` (structure, swing points, OB, FVG, premium/discount, MTF PDH/PDL/PWH/PWL levels, agent bias) — uses the same analysis helpers as the agents. - `MQL5\Include\AlgoForge\AF_Engine3_Render.mqh` — pure renderer: `AFRenderCfg`, `AFR_DefaultCfg`, `AFR_DrawAll()`, `AFR_Clear()`; lines/text/boxes/panel; dashboard mimicking the original 75466 layout (bias, structure, liquidity, context, key levels, N/C/E/P alignment, trade setup, legend). NO analysis computation. - `MQL5\Indicators\AlgoForge\AF_Engine3_Display.mq5` — consumer indicator (0 buffers, object-only): draws only when a new closed bar appears on the display TF (`g_lastDispBar`). - `MQL5\Experts\AlgoForge_Engine3_UnitTest.mq5` — unit tests T1–T5 (prefix `AFTEST3`). - `AF_Defines.mqh` extended: Engine-3 constants (AF_E3_*: window, zone counts, min OB/FVG sizes, PD bands, D1/W1 capacity, object prefix `AF3_`). - Principle: Engine 3 READS ONLY Engine 1 & 2 output; anti-freeze stays in Engine 1. ### Compilation - `AlgoForge_Engine3_UnitTest.mq5` → **0 errors, 0 warnings** (build 6093, AVX512). - `AF_Engine3_Display.mq5` (indicator) → **0 errors, 0 warnings**. - Regression: `AlgoForge_Engine1_UnitTest.mq5` & `AlgoForge_Engine2_UnitTest.mq5` (AF_Defines.mqh changed) → **0/0**. ### Unit Test (Strategy Tester, XAUUSD M15, every tick) | Run | Range | Result | |---|---|---| | Full | 2026.08.01–08.20 | **PASS=21528 FAIL=0** (finished 10:14:42; 506 AFTEST3 log lines) | - **T1**: display data valid (structure/eqPos/range/PDH-PDL-PWH-PWL must be >0 when the D1/W1 slots are ready). - **T2**: non-repaint — identical display data within the same bar. - **T3**: closed-bar lock on all slots (N/C/E/P/DISP). - **T4**: render creates 214–232 objects per bar (prefix `AF3_`) + the `P_BIAS` dashboard panel exists. - **T5**: render deterministic — re-rendering the same bar → same object count. - Real examples: structure Range/Bullish/Bearish; eqPos Premium/Equilibrium/Discount; HH/HL/LH/LL varying (21–22 / 17–18 / 20–21 / 17–18); piv=32, lines=43–45, ob=5, fvg=6. ### Constraints & Solutions (this session) 10. `ObjectCreate` in the Strategy Tester: on this build, chart objects are created even with `Visual=0` (object probe succeeds → T4/T5 run, 232 objects); when unsupported, the probe marks an automatic SKIP (not a failure) — see DESIGN.md §17. ## Phase 3 Publication (complete) - Commit `92f1fbc` "Algo Forge Phase 3: Engine 3 (pure display render - structure/zones/ signals/dashboard, reads Engine 1-2 output only) + AF_Engine3_Display indicator + unit test (PASS=21528 FAIL=0)" — 9 files, +1681/−79 — pushed to https://forge.mql5.io/chiki2bum2/SniperGold_ML (main). ## Phase 4 — Hybrid ML (SUB-SESSION 1: DATA + SURVEY ✅, 2026-08-21) - Handoff + initial prompt: `Shared Projects\SniperGold_ML\REFACTOR_ALGOFORGE.md` §14. - Baseline: `MQL5\Include\SniperGold_ML.mqh` (MLP 19→12→2, AUC 0.627/0.621) — deploy freeze, do not change during testing. - Binding protocol: `Shared Projects\SniperGold_ML\FASE3_BATASAN_UJI.md` (G1–G7). - Estimate: ≥150–250k tokens, multi-session; split into sub-sessions (data → regime → LSTM → validation → export → integration), each ending with a checkpoint + handoff. ### Sub-session 1 (COMPLETE) — Survey + Audit + ~19 years of data **Old pipeline audit** (Shared Projects\SniperGold_ML\): train_model.py (19 SMC features, 24×0.75ATR label, MLP, walk-forward, .mqh export), train_lstm.py (1-layer numpy LSTM, W=32 H=32, bootstrap CI), train_regime.py (GARCH-t + 2-state HMM + causal RV24), train_regime_gate.py (regime as gate — proven to lose), train_ensemble.py (5-seed), retrain_deploy.py (deploy+rollback+audit). Details: `docs\TAHAP4_PLAN.md` §2. **Data (downloaded + continuity verified):** - `ml\download_bars.py` → `Files\AlgoForge\Data\XAUUSD_{M15,H1,H4,D1}.{csv,npz}`. - Verification: 0 duplicates, 0 anomalies (>4 days), longest gap 96 hours (long weekend). | TF | Bars | Range | Decision | |---|---|---|---| | D1 | 4.842 | 2007-11-29 → 2026-08-21 (18.73 y) | ✅ fully dense — HTF context | | M15 | 199.386 total | 2007–2016 ≈ 1 bar/day | ⚠️ NOT usable for an M15 model | | M15 | ~186.500 | 2018-01-01 → now (8.6 y) | ✅ DENSE window (≈3x the 60k baseline) | **Binding decisions (honest):** 1. The "20-year M15" target is NOT available on the HFM feed → M15 uses the dense 2018+ window (8.6 years); D1 18.7 years for HTF bias. Dukascopy is only an experiment (different feed). 2. Label stays `24 bars × 0.75 ATR`; note: the new density ~65 bars/day → 24 bars ≈ 6 session hours; verified when building features (sub-session 2). 3. Baseline cache `features_XAUUSDc.npz` read-only (like-for-like gate vs 0.627/0.621). 4. Symbol: XAUUSD (2-digit) — XAUUSDc no longer exists in the terminal. 5. Architecture: 19 SMC features + causal regime (GARCH/HMM/RV) → 2-layer LSTM → 2-head P(long)/P(short); Engine-2 features (N/C/E/P) as backlog candidates, only if they win validation. Gates G1–G7 binding; test touched once. **New files:** `ml\download_bars.py`, `ml\probe_density.py`, `ml\probe_byyear.py`, `docs\TAHAP4_PLAN.md`. ### Sub-session 2 (COMPLETE) — Features + Regime + Label **`ml\build_features.py`** (reuses `train_model.py`/`train_regime.py` — runtime–training consistency): - 19 SMC features identical to baseline + `24×0.75ATR` label; dense 2018+ window (2017 warmup for structure/ATR/HTF lookbacks). - Causal regime (strict anti-lookahead): - **t-GARCH(1,1)** interior with classic bounds + 0.5% return winsor: `alpha=0.0615 beta=0.9230 persist=0.9844 nu=7.12` (without winsor the MLE falls to an ARCH-heavy boundary solution; evidence `diag_garch.py`/`diag_garch2.py`). - causal sigma (unconditional init — improvement over `train_regime.py` which used `np.var(r)` over the whole array = mild lookahead at bar 0). - z standardization train-only; RV24 from RAW returns. - **2-state Gaussian HMM (MSM) COLLAPSES on this feed** (P(high)=1 on z, |z|, log(RV24), RV24 — evidence `diag_hmm.py`; consistent with the Phase-2 lesson). Replaced by a **causal logistic regime score** (RV24 vs train median/MAD): mean 0.585, std 0.229, test mean 0.701 (2024+ is indeed volatile). The honest deviation is recorded in TAHAP4_PLAN §7.6; the final effect is decided by the AUC gate. - **Cache** `Files\AlgoForge\Data\features_XAUUSD.npz`: F(197032,19), F2(197032,6), label, ATR, close, time, split_bar=148470, split_pos=115631, meta JSON. 0 NaN/Inf; 0 constant columns. - Label: 154.175 samples (bull 80.939 / bear 73.236, ratio 1.105); train bull 0.516, test bull 0.551. **24-bar forward = median 6.00 session hours** (p10=6, p90=7). - **Unit tests B1–B5 PASS** (numerical ATR, streaming-feature anti-lookahead prefix==full, forward label, causal GARCH sigma, causal HMM filter + valid probs). - Verification: `ml\verify_cache.py` (shape/NaN/variation/label density) OK. - Baseline `features_XAUUSDc.npz` & `SniperGold_ML.mqh` NOT touched. **Next (sub-session 3):** `train_hybrid.py` — 2-layer LSTM (W=32, H=32) + 2-target long/short head; walk-forward 75/25 (+70/30, 80/20); 5 seeds [42,7,123,2024,999]; 2000× bootstrap CI; gates G1–G7 vs baseline 0.627/0.621 (like-for-like on the XAUUSDc cache) — test touched once. ### Sub-session 3 (COMPLETE) — 2-layer LSTM + Gate → **No-Go** **`ml\train_hybrid.py`** (pure numpy): - **2-layer** LSTM (W=32, H1=H2=32) + **2-target** long/short head (BCE mean B×2); 2-layer BPTT (L2 → dX → L1). Unit tests **B1–B5 PASS**: forward == 2×1-layer (err 0), gradcheck rel=2.7e-5, determinism, anti-lookahead, overfit AUC=1.0. - Calibration: 6.6 s/epoch (5.1k train) → gate estimate ~53 min. **LIKE-FOR-LIKE GATE (XAUUSDc 60k cache, 75/25 split, 5 seeds, 2000× bootstrap)** (log: `ml\TAHAP4_HYBRID.log`): - Hybrid ens: LONG=0.6074 (mean 0.5898±0.0066) · SHORT=0.6073 (mean 0.5900±0.0056) - vs target 0.6270/0.6207 → dAUC **−0.0196 / −0.0134** → **G3 FAILED** - vs MLP reproduction (0.6795/0.6560) → dAUC **−0.0723 / −0.0487**, 95% CI all NEGATIVE → **G4 FAILED (significantly worse)**; **G6 0/5 seeds** win. - **DECISION: GATE No-Go → protocol complete (one No-Go = done, no negotiation).** Consistent with Phase 3 (LSTM lost on 60k bars). Full walk-forward on new data NOT run (gate decisive; stop rule). - **Hybrid .mqh export CANCELLED** (no model passed the gate). The baseline `SniperGold_ML.mqh` (MLP 0.627/0.621) REMAINS the deployed model (freeze kept). **Phase-4 status:** data ✅ → regime/features/label ✅ → LSTM ✅ → validation/gate ✅ (No-Go) → export ❌ (cancelled) → integration — baseline stays. Proceed to **Phase 5** (baseline verification & publication / Algo Forge release) — see REFACTOR §20. ## Phase 3 Publication (complete, reference) - Commit `92f1fbc` "Algo Forge Phase 3: Engine 3 (pure display render - structure/zones/ signals/dashboard, reads Engine 1-2 output only) + AF_Engine3_Display indicator + unit test (PASS=21528 FAIL=0)" — 9 files, +1681/−79 — pushed to https://forge.mql5.io/chiki2bum2/SniperGold_ML (main). ## Phase 4 Sub-session 1 Publication (complete) - Commit `33fcb37` "Algo Forge Phase 4 sub-session 1: survey + old ML pipeline audit + XAUUSD 19y data ..." — 5 files, +471 — pushed (main). ## Phase 5 — BASELINE / ALGO FORGE VERIFICATION & PUBLICATION (COMPLETE, 2026-08-21) ### 1. 0/0 compile of the whole package + unit-test regression - Compile **0 errors / 0 warnings** (build 6093, AVX512): `AlgoForge_Backtest_Baseline.mq5`, `AlgoForge_Engine1/2/3_UnitTest.mq5`, `AF_Engine3_Display.mq5`. - Strategy Tester unit-test regression (XAUUSD M15, every tick, 2026.08.01–08.20): | Engine | PASS | FAIL | Status | |---|---|---|---| | Engine 1 | 115070 | 0 | identical to the Phase-1 baseline | | Engine 2 | 21534 | 0 | identical to the Phase-2 baseline | | Engine 3 | 21528 | 0 | identical to the Phase-3 baseline | ### 2. Net-of-spread baseline backtest (internal features + CSV, NOT iCustom) - New EA `MQL5\Experts\AlgoForge_Backtest_Baseline.mq5`: - 19 SMC features computed INTERNALLY (identical to v4.4 `ComputeMLFeatures` + dependencies), data source ONLY Engine 1 (`AFEngine1MTF`; closed-bar lock, anti-freeze). - Calls the frozen model `SniperGold_ML.mqh` (MLP 19→12→2, v20260821_2head). - Mode 0 = per-bar M15 prob CSV (parity/AUC via `ml\backtest_eval.py`); Mode 1 = net-of-spread trading in the tester (market OrderSend, actual spread). **Mode 0 (CSV/AUC)** — XAUUSD M15, 2026.01.01–08.20, every tick, **14.850 bars**: | Metric | Value | Baseline freeze | |---|---|---| | AUC LONG | **0.5305** | 0.6270 | | AUC SHORT | **0.5487** | 0.6207 | | Precision LONG @0.60 | 0.5352 (n=4454) | — | | Precision SHORT @0.60 | 0.5292 (n=7010) | — | - Poor calibration: prob long 0.65+ → actual frequency only 0.545. - **HONEST CONCLUSION**: on the XAUUSD feed (not the XAUUSDc training feed), the baseline model probabilities are NOT calibrated and the AUC is far lower. Evidence that the 0.627/0.621 metrics apply only to the training feed/split → **do not claim an edge on another feed without a like-for-like gate**. **Mode 1 (net-of-spread trade)** — XAUUSD M15, 2026.05.01–08.20, every tick, lot 0.01, deposit 10k, thresh L/S=0.60, SL=1.0ATR, TP=1.5ATR, maxHold=24: | Stat | Value | |---|---| | Trades | 308 | | Net profit | +1125.32 | | Max drawdown | 1188.18 | | Profit factor | 1.32 | - Honest note: some signals failed to execute (requote ret=10018, no retry); the positive result is NOT significant (runtime AUC 0.53/0.55; small sample; model not calibrated on this feed) → reported as-is, not an edge claim. ### 3. Documentation & publication - README/DESIGN/PROGRESS updated; `AlgoForge_Backtest_Baseline.mq5` + tester config + `ml\backtest_eval.py` added to the repo; mandatory 75466/Waseem Shahrukh attribution kept in all files. - Pushed to https://forge.mql5.io/chiki2bum2/SniperGold_ML (main) — see `REFACTOR_ALGOFORGE.md` §21. ## Agent base-TF change — HARDCODED H4/M30/M15/M3 (COMPLETE, 2026-08-21) - **Decision:** manual inputs `InpHtfS1..S4` **removed**; agent TFs hardcoded via new macros in `AF_Defines.mqh`: `AF_E2_TF_S1=H4` (Narrative), `AF_E2_TF_S2=M30` (Context), `AF_E2_TF_S3=M15` (Entry), `AF_E2_TF_S4=M3` (PriceAction). Top-down analysis basis: **H4→M30→M15→M3**. - Wording correction: S4 was originally mistyped M5 → corrected to **M3** (user request). - Files changed: `AF_Defines.mqh`, `AF_Engine3_Display.mq5` (indicator), `AlgoForge_Engine2_UnitTest.mq5`, `AlgoForge_Engine3_UnitTest.mq5` (independence-test tamper slots: H1/M20/M12/M6 — different from the base), Engine-2/3 tester configs. - **Compile 0/0** (indicator + 2 unit tests + Engine-1 regression). - **Re-test** (XAUUSD M15, every tick, 2026.08.01–08.20, new base): - Engine 2: **PASS=35886 FAIL=0** (final: N=H4 hist=79 · C=M30 hist=599 · E=M15 hist=1197 · P=M3 hist=5981). - Engine 3: **PASS=35880 FAIL=0** (DISP=M15, D1/W1 OK). - (PASS rose vs 21534/21528 because M3 adds more new bars per range; FAIL stays 0, no SKIP.) - Anti-freeze maintained: `historyCalls` = 1 per new bar per TF.