SniperGold_ML/docs/DESIGN.md

# Algo Forge — DESIGN

> Source of inspiration: SniperGold SMC Pro+ (c) Waseem Shahrukh — https://www.mql5.com/en/code/75466

Per-engine technical specification document. Project structure & status: see `PROGRESS.md` and `README.md`.

---

# Engine 1 — MTF Bar Data Collector + Cache

File: `MQL5\Include\AlgoForge\AF_Engine1_MTFData.mqh` — class `AFEngine1MTF`, struct `AFBar`.

## 1. Responsibilities

- Collect OHLCV bars from several explicit timeframes, as required by the
  N/C/E/P agents (Engine 2) and the display (Engine 3).
- Provide an **index-based internal cache** so data can be reused repeatedly without
  repeated History reads (anti-freeze). This pattern is proven in SniperGold v4.5
  (`RefreshMTFCache` / `BuildMTFStruct`).
- **Chart-TF independent**: all access uses an explicit symbol + explicit
  `ENUM_TIMEFRAMES`. `PERIOD_CURRENT` is rejected at `Register()`.
- **Closed-bar lock**: the cache contains ONLY closed bars (non-repainting).

## 2. API

### Lifecycle
```cpp
AFEngine1MTF e1;
int s1 = e1.Register(PERIOD_M15, 600);          // slot index or AF_E1_ERR_SLOT
bool changed = e1.Refresh();                     // call each OnCalculate/OnTick
```

### Read (idxFromRight: 0 = NEWEST closed bar)
```cpp
bool     e1.IsReady(slot);
int      e1.Count(slot);                         // number of closed bars in the cache
bool     e1.GetBar(slot, idx, AFBar &out);
bool     e1.GetBarByTime(slot, datetime, AFBar &out);
int      e1.FindBarIndex(slot, datetime);
double   e1.Open/High/Low/Close(slot, idx);
datetime e1.Time(slot, idx);
long     e1.TickVolume(slot, idx);
double   e1.ATR(slot, period=14);
```

### Diagnostics
```cpp
int  e1.HistoryCalls(slot);   // number of CopyRates executed (unit test anti-freeze)
int  e1.RefreshCount(slot);
int  e1.TotalHistoryCalls();
long e1.LastBars(slot);
```

## 3. Cache Mechanism

1. `Refresh()` checks `Bars(symbol, tf)` for each slot.
2. **Only if `Bars()` changed** (or the cache is not ready + throttle `AF_E1_RETRY_SEC`)
   → `Build(slot)` is called → `CopyRates` executes.
3. `Build`: `CopyRates(symbol, tf, 0, maxBars+1, rates)` with as-series array;
   the forming bar (not yet closed) is **dropped** based on `IsBarClosed()`:
   `barTime + PeriodSeconds(tf) <= TimeCurrent()`.
4. The cache is stored in series: `bars[0]` = newest closed bar. Capacity `maxBars`.

### Invariants (verified by unit tests)
- `Count(slot) <= Bars(symbol, tf)`.
- All cache bars are closed (T4).
- `HistoryCalls` increases by **exactly 1** per new bar per TF (T5 anti-freeze).
- Strictly decreasing time order from index 0 (T3); valid OHLC (T2).

## 4. Limits & Result Codes

- `AF_E1_MAX_SLOTS = 8`, `AF_E1_MAX_BARS = 5000`, `AF_E1_RETRY_SEC = 5`.
- `AF_E1_OK / AF_E1_ERR_SLOT / AF_E1_ERR_NOTREADY / AF_E1_ERR_RANGE`.

## 5. MQL5 Implementation Notes

- **No array-element references** in MQL5 (`T &x = arr[i]` = error).
  All access uses direct index (`m_slots[i].field`).
- Slots use a dynamic `m_slots[]` array (a struct containing dynamic arrays is safe).
- `#property version` must be `X.YY` format (e.g. "1.00") to avoid warnings.

## 6. Unit Test

- Harness: `MQL5\Experts\AlgoForge_Engine1_UnitTest.mq5` (EA, log prefix `AFTEST`).
- Run in the Strategy Tester (XAUUSD, model **every tick**; the "1-min OHLC" model
  rejects sub-chart TF requests).
- Result 2026-08-21: PASS=115070 FAIL=0 (20 days) · PASS=8870 FAIL=0 (verification).

---

# Engine 2 — 4 Independent Signal Agents (N/C/E/P) + Fuzzy + Aggregator

Files:
- `MQL5\Include\AlgoForge\AF_Engine2_Agents.mqh` — 4 agents + fuzzy logic (classes
  `AFAgentNarrative`, `AFAgentContext`, `AFAgentEntry`, `AFAgentPriceAction`).
- `MQL5\Include\AlgoForge\AF_Engine2_Aggregator.mqh` — separate aggregator
  (class `AFAggregator`) + facade `AFEngine2Signals`.

## 7. Key Principles

1. **4 INDEPENDENT agents** — each agent reads Engine 1 only through **one slot**
   (timeframe) of its own. No inter-agent calls/state; all methods are **stateless**
   (pure functions of Engine-1 closed bars).
2. **No mutual knowledge** — the final composition is done by the **separate
   aggregator**, not between agents.
3. **Dynamic-weight fuzzy logic** — each agent uses membership functions
   (`AF_MF_Tri` / `AF_MF_Trap`) + a light Mamdani evaluator (`AFFuzzyEval`:
   `buyAcc/sellAcc/wTot`, rules `Rule(buySide, fire, weight)`); weights adapt
   to market conditions measured **from the agent's own data** (trend strength,
   volatility, ranging).
4. **Non-repainting** — all inputs are Engine-1 closed bars (closed-bar lock
   guaranteed by Engine 1; Engine 2 never reads History directly).

## 8. Input Timeframe

Agent TFs are **HARDCODED** (no manual `InpHtfS1..S4` inputs — removed) — see
macros `AF_E2_TF_S1..S4` in `AF_Defines.mqh`:
```
AF_E2_TF_S1 = H4   (S1 = Narrative  / N)
AF_E2_TF_S2 = M30  (S2 = Context    / C)
AF_E2_TF_S3 = M15  (S3 = Entry      / E)
AF_E2_TF_S4 = M3   (S4 = PriceAction/P)
```
Consumers (indicator `AF_Engine3_Display`, Engine-2/3 unit tests) use these macros
directly at `Register()`. Analysis basis: **H4→M30→M15→M3** (top-down to the chart).

## 9. Agents

| Agent | Question | Assessed elements (from its own slot) | Dynamic weights |
|---|---|---|---|
| **N (Narrative)** | "Which way is the market?" | HH/HL/LL/LH pivots → trend (+clarity), CHoCH/MSS, BOS, liquidity sweep, premium/discount | Clear trend → structure dominates; flat → zones/liquidity up |
| **C (Context)** | "Which zone is price in?" | OB (opposite bar before a strong move), FVG/imbalance, S/R (pivots), premium/discount | High volatility → S/R & premium/discount down, OB/FVG up |
| **E (Entry)** | "Is there entry confirmation?" | Sweep, CHoCH, displacement, OB/FVG zones; rule **ZONE + CONFIRMATION = setup** | Strong displacement → confirmation weight up |
| **P (Price Action)** | "When to open?" | Engulfing, pin bar, inside bar, 2-bar momentum, close position in range | Ranging → reversal patterns up; trending → continuation up |

Per-agent output: `AFSignalOut { buy, sell, bias, confidence, dir, reason }`.

## 10. Aggregator (separate)

`AFAggregator::Compute(e1, slotE, n, c, e, p, out)`:
1. **Pass 1**: initial bias = Σ (base weight × confidence × bias) / Σ (base weight × confidence).
   Base weights: N=0.30, C=0.30, E=0.25, P=0.15 (`AF_AGG_W_*`).
2. **Pass 2 (dynamic weights)**: agents aligned with the majority get a 1.5× boost;
   aggregate `buy/sell` = Σ (effective weight × buy/sell) / Σ weights.
3. **Final signal**: `dir = BUY/SELL/WAIT` with threshold `AF_AGG_BUY_TH=0.20` and
   minimum support `AF_AGG_MIN_SUP=0.50`.
4. **Levels**: `entry` = close of the closed slot-E bar; `sl/tp` based on Engine-1 ATR
   (`AF_AGG_SL_ATR=1.5`, `AF_AGG_TP_ATR=2.5`) — only on BUY/SELL signals.
5. `confidence` aggregate = support × (0.7 + 0.1 × number of aligned agents).

Facade `AFEngine2Signals::Compute(e1, sN, sC, sE, sP, oN, oC, oE, oP, agg)`:
one call runs the 4 agents + aggregator (used by Engine 3 / consumers).

## 11. Unit Test

- Harness: `MQL5\Experts\AlgoForge_Engine2_UnitTest.mq5` (log prefix `AFTEST2`).
- Strategy Tester (XAUUSD, **every tick**):
  - T1 output validity per agent · T2 independence (changing agent X's input slot →
    other agents unchanged) + determinism · T3 closed-bar lock · T4 non-repaint
    (identical output within the same bar) · T5 aggregator validity · T6 synthetic aggregator.
- Results 2026-08-21: **PASS=21534 FAIL=0** (20 days) · PASS=9942 FAIL=0 (10-day verbose).
- Note: agents use tamper slots (different TFs) for the independence test; T2
  auto-retries when a tamper slot is not ready.

## 12. Implementation Constraints (additional, Phase-2 sessions)

- `replace_text_in_file` multi-line edits often fail → use single-line edits.
- The independence test needs a tamper slot with ≥ `AF_E2_MIN_BARS` (80) bars; choose
  a dense TF (M6/M12/M20/M30) so it is ready even on short runs.

---

# Engine 3 — Display (reads Engine 1 & 2 output only)

Files:
- `MQL5\Include\AlgoForge\AF_Engine2_Display.mqh` — **display-context builder**
  (Engine-2 layer): struct `AFDisplayData`, `AFDispZone`, `AFDispLine`,
  `AFDispPivot` + `AF_BuildDisplayData()`. **All display computation** (structure,
  swing points, OB, FVG, premium/discount, MTF levels, agent bias) happens HERE,
  using the same pure analysis helpers as the N/C/E/P agents
  (`AF_BuildSwing`, `AF_DetectSweep`, `AF_RangeStat`, etc.).
- `MQL5\Include\AlgoForge\AF_Engine3_Render.mqh` — **pure renderer** (Engine-3
  layer): struct `AFRenderCfg` + `AFR_DrawAll()` / `AFR_Clear()`. NO analysis
  computation; draws only from `AFDisplayData` + `AFSignalOut` + `AFAggOut`
  (structure, zones, signals, dashboard like the original 75466 code).
- `MQL5\Indicators\AlgoForge\AF_Engine3_Display.mq5` — indicator (chart window):
  inputs + OnInit/OnCalculate/OnDeinit; 0 buffers (object-only).
- `MQL5\Experts\AlgoForge_Engine3_UnitTest.mq5` — unit test (Strategy Tester).

## 13. Engine-3 Principles

1. **READS ONLY** Engine-1 (`AFEngine1MTF`) & Engine-2 output (`AFEngine2Signals` /
   `AFAggOut` / `AFDisplayData`). Engine 3 computes no structure/zones itself.
2. Signal–display consistency: `AF_BuildDisplayData` (Engine 2) uses the same
   helpers as the agents.
3. **Non-repainting**: the indicator draws only when a new closed bar appears on the
   display TF (`e1.Time(sDisp,0)` changes); no redraw within the same bar.
4. Anti-freeze: Engine 1 remains the only History reader (index cache).

## 14. Display elements (equivalent to original code 75466)

| Element | Source | Description |
|---|---|---|
| BOS/CHoCH structure lines | `AFDispLine` (AF_BuildStructLines) | fractal pivots, BOS/CHoCH labels |
| HH/HL/LH/LL swing points | `AFDispPivot` (AF_ClassifyPivots) | bull/bear colors |
| Order Block | `AFDispZone` (AF_CollectOBs) | MIT filter + dedupe + size ≥ 0.15×ATR |
| FVG | `AFDispZone` (AF_CollectFVG) | MIT filter + size ≥ 0.02×ATR |
| Premium/Discount | box from `swHigh/swLow` + position | premium/equilibrium/discount bands |
| MTF PDH/PDL levels | Engine-1 D1 slot (index 1) | solid lines |
| MTF PWH/PWL levels | Engine-1 W1 slot (index 1) | dashed lines |
| Entry/SL/TP signals + arrows | `AFAggOut` | only when dir BUY/SELL |
| Dashboard | `AFDisplayData` + `AFAggOut` | bias, structure, liquidity, context, levels, N/C/E/P alignment, trade setup, legend |

## 15. Indicator inputs

- `InpHtfS1..S4` = agent N/C/E/P TFs (Engine 2) — default H1/H1/M15/M15.
- `InpDispTF` = structure/zone display TF — default `PERIOD_CURRENT` (= chart).
- Toggles: structure, swing points, OB (+count), FVG (+count), premium/discount,
  MTF levels, signals, dashboard.
- LuxAlgo-style colors (same defaults as original code 75466) + transparent panel.

## 16. Unit Test

- Harness: `MQL5\Experts\AlgoForge_Engine3_UnitTest.mq5` (log prefix `AFTEST3`).
- Strategy Tester (XAUUSD, **every tick**, **Visual=1** — chart objects are only
  created in visual mode).
- T1 display-data validity (structure/eqPos/levels/bias) · T2 non-repaint (identical
  data within the same bar) · T3 closed-bar lock · T4 render creates chart objects
  (prefix `AF3_`) · T5 render determinism (same object count).
- Results 2026-08-21: see `PROGRESS.md`.

## 17. Implementation Constraints (Phase 3)

- `ObjectCreate` in the Strategy Tester only works in **visual mode**; the test EA
  probes once at startup and T4/T5 are auto-skipped (not failed) when objects are
  unsupported.
- Visual mode slows the tester (20 days every tick ≈ 10 minutes) → for quick
  verification run a short range (e.g. 3–5 days).

---

# Backtest Baseline (Phase 5) — verification & publication

File: `MQL5\Experts\AlgoForge_Backtest_Baseline.mq5`
+ config `MQL5\Profiles\Tester\AlgoForge_Backtest_Baseline.XAUUSD.M15.*.ini`
+ evaluation `ml\backtest_eval.py`.

## 18. Objective & Principles

- **Honest backtest** of the baseline strategy (MLP freeze `SniperGold_ML.mqh`,
  AUC long 0.627 / short 0.621) **net of spread**.
- **NOT a fragile iCustom**: 19 SMC features computed internally in the EA
  (identical to `SniperGold_SMC_ProPlus_v4_4.mq5` `ComputeMLFeatures` + its
  dependencies), data source ONLY Engine 1 (`AFEngine1MTF`) — closed-bar lock,
  anti-freeze, consistent with the Algo Forge architecture.
- Feature parameters hardcoded (`AF_BT_*` = v4.4 training defaults):
  SwingLen=50, InternalLen=5, Lookback=600, GrabWindow=8, EQ thr=0.10/3 bars,
  DeltaBars=10, HTF=D1/H4/H1, ConfluenceFilter=true.

## 19. Modes

| Mode | Function | Output |
|---|---|---|
| 0 | CSV long/short prob per closed M15 bar | `AlgoForge_bt_prob_*.csv` → `backtest_eval.py` (AUC/precision/calibration) |
| 1 | Net-of-spread trading in the Strategy Tester | OrderSend market (ATR SL/TP, max hold); actual tester spread; OnTester summary |

Inputs: `InpMode`, `InpThreshLong/Short` (default 0.60/0.60), `InpSL_ATR=1.0`,
`InpTP_ATR=1.5`, `InpMaxHoldBars=24` (= baseline label horizon),
`InpLot=0.01`, `InpMaxBars=700`.

## 20. Results (2026-08-21, XAUUSD M15, every tick) — HONEST

**Mode 0 (CSV/AUC)**, 2026.01.01–08.20, 14.850 bars:

| Metric | Value | Baseline freeze |
|---|---|---|
| AUC LONG | 0.5305 | 0.6270 |
| AUC SHORT | 0.5487 | 0.6207 |
| Precision LONG @0.60 | 0.5352 (n=4454) | — |
| Precision SHORT @0.60 | 0.5292 (n=7010) | — |

- Poor calibration (prob 0.65+ → long frequency 0.545) → on the XAUUSD feed (not the
  XAUUSDc training feed) the baseline model is NOT calibrated. **Do not claim an edge
  on another feed without a like-for-like gate.**

**Mode 1 (trade)**, 2026.05.01–08.20: trades=308, net=+1125.32, maxDD=1188.18,
PF=1.32. Note: some signals failed to execute (requote 10018, no retry); the positive
result is not significant (runtime AUC 0.53/0.55, small sample) — not an edge claim.

## 21. Constraints (Phase 5)

- Tester CSV is written to the agent sandbox (`Tester\Agent-*\MQL5\Files\`) and
  **reset per run** → read/copy results immediately after the run (Python evaluation
  directly from the sandbox path).
- MQL5 `FILE_CSV` delimiter in this terminal = **TAB** (not `;`/`,`) →
  `backtest_eval.py` uses `delimiter="\t"`.
- `PositionSelect` overload ambiguity on build 6093 (string vs ulong) → helper
  `BTSelectTicket()` (loop `PositionsTotal`/`PositionGetTicket`).
- OrderSend requotes (ret=10018) appear in the tester → recorded, no retry (honest).