299 lines
14 KiB
Markdown
299 lines
14 KiB
Markdown
# Algo Forge — DESIGN
|
|
|
|
> Source of inspiration: SniperGold SMC Pro+ (c) Waseem Shahrukh — https://www.mql5.com/en/code/75466
|
|
|
|
Per-engine technical specification document. Project structure & status: see `PROGRESS.md` and `README.md`.
|
|
|
|
---
|
|
|
|
# Engine 1 — MTF Bar Data Collector + Cache
|
|
|
|
File: `MQL5\Include\AlgoForge\AF_Engine1_MTFData.mqh` — class `AFEngine1MTF`, struct `AFBar`.
|
|
|
|
## 1. Responsibilities
|
|
|
|
- Collect OHLCV bars from several explicit timeframes, as required by the
|
|
N/C/E/P agents (Engine 2) and the display (Engine 3).
|
|
- Provide an **index-based internal cache** so data can be reused repeatedly without
|
|
repeated History reads (anti-freeze). This pattern is proven in SniperGold v4.5
|
|
(`RefreshMTFCache` / `BuildMTFStruct`).
|
|
- **Chart-TF independent**: all access uses an explicit symbol + explicit
|
|
`ENUM_TIMEFRAMES`. `PERIOD_CURRENT` is rejected at `Register()`.
|
|
- **Closed-bar lock**: the cache contains ONLY closed bars (non-repainting).
|
|
|
|
## 2. API
|
|
|
|
### Lifecycle
|
|
```cpp
|
|
AFEngine1MTF e1;
|
|
int s1 = e1.Register(PERIOD_M15, 600); // slot index or AF_E1_ERR_SLOT
|
|
bool changed = e1.Refresh(); // call each OnCalculate/OnTick
|
|
```
|
|
|
|
### Read (idxFromRight: 0 = NEWEST closed bar)
|
|
```cpp
|
|
bool e1.IsReady(slot);
|
|
int e1.Count(slot); // number of closed bars in the cache
|
|
bool e1.GetBar(slot, idx, AFBar &out);
|
|
bool e1.GetBarByTime(slot, datetime, AFBar &out);
|
|
int e1.FindBarIndex(slot, datetime);
|
|
double e1.Open/High/Low/Close(slot, idx);
|
|
datetime e1.Time(slot, idx);
|
|
long e1.TickVolume(slot, idx);
|
|
double e1.ATR(slot, period=14);
|
|
```
|
|
|
|
### Diagnostics
|
|
```cpp
|
|
int e1.HistoryCalls(slot); // number of CopyRates executed (unit test anti-freeze)
|
|
int e1.RefreshCount(slot);
|
|
int e1.TotalHistoryCalls();
|
|
long e1.LastBars(slot);
|
|
```
|
|
|
|
## 3. Cache Mechanism
|
|
|
|
1. `Refresh()` checks `Bars(symbol, tf)` for each slot.
|
|
2. **Only if `Bars()` changed** (or the cache is not ready + throttle `AF_E1_RETRY_SEC`)
|
|
→ `Build(slot)` is called → `CopyRates` executes.
|
|
3. `Build`: `CopyRates(symbol, tf, 0, maxBars+1, rates)` with as-series array;
|
|
the forming bar (not yet closed) is **dropped** based on `IsBarClosed()`:
|
|
`barTime + PeriodSeconds(tf) <= TimeCurrent()`.
|
|
4. The cache is stored in series: `bars[0]` = newest closed bar. Capacity `maxBars`.
|
|
|
|
### Invariants (verified by unit tests)
|
|
- `Count(slot) <= Bars(symbol, tf)`.
|
|
- All cache bars are closed (T4).
|
|
- `HistoryCalls` increases by **exactly 1** per new bar per TF (T5 anti-freeze).
|
|
- Strictly decreasing time order from index 0 (T3); valid OHLC (T2).
|
|
|
|
## 4. Limits & Result Codes
|
|
|
|
- `AF_E1_MAX_SLOTS = 8`, `AF_E1_MAX_BARS = 5000`, `AF_E1_RETRY_SEC = 5`.
|
|
- `AF_E1_OK / AF_E1_ERR_SLOT / AF_E1_ERR_NOTREADY / AF_E1_ERR_RANGE`.
|
|
|
|
## 5. MQL5 Implementation Notes
|
|
|
|
- **No array-element references** in MQL5 (`T &x = arr[i]` = error).
|
|
All access uses direct index (`m_slots[i].field`).
|
|
- Slots use a dynamic `m_slots[]` array (a struct containing dynamic arrays is safe).
|
|
- `#property version` must be `X.YY` format (e.g. "1.00") to avoid warnings.
|
|
|
|
## 6. Unit Test
|
|
|
|
- Harness: `MQL5\Experts\AlgoForge_Engine1_UnitTest.mq5` (EA, log prefix `AFTEST`).
|
|
- Run in the Strategy Tester (XAUUSD, model **every tick**; the "1-min OHLC" model
|
|
rejects sub-chart TF requests).
|
|
- Result 2026-08-21: PASS=115070 FAIL=0 (20 days) · PASS=8870 FAIL=0 (verification).
|
|
|
|
---
|
|
|
|
# Engine 2 — 4 Independent Signal Agents (N/C/E/P) + Fuzzy + Aggregator
|
|
|
|
Files:
|
|
- `MQL5\Include\AlgoForge\AF_Engine2_Agents.mqh` — 4 agents + fuzzy logic (classes
|
|
`AFAgentNarrative`, `AFAgentContext`, `AFAgentEntry`, `AFAgentPriceAction`).
|
|
- `MQL5\Include\AlgoForge\AF_Engine2_Aggregator.mqh` — separate aggregator
|
|
(class `AFAggregator`) + facade `AFEngine2Signals`.
|
|
|
|
## 7. Key Principles
|
|
|
|
1. **4 INDEPENDENT agents** — each agent reads Engine 1 only through **one slot**
|
|
(timeframe) of its own. No inter-agent calls/state; all methods are **stateless**
|
|
(pure functions of Engine-1 closed bars).
|
|
2. **No mutual knowledge** — the final composition is done by the **separate
|
|
aggregator**, not between agents.
|
|
3. **Dynamic-weight fuzzy logic** — each agent uses membership functions
|
|
(`AF_MF_Tri` / `AF_MF_Trap`) + a light Mamdani evaluator (`AFFuzzyEval`:
|
|
`buyAcc/sellAcc/wTot`, rules `Rule(buySide, fire, weight)`); weights adapt
|
|
to market conditions measured **from the agent's own data** (trend strength,
|
|
volatility, ranging).
|
|
4. **Non-repainting** — all inputs are Engine-1 closed bars (closed-bar lock
|
|
guaranteed by Engine 1; Engine 2 never reads History directly).
|
|
|
|
## 8. Input Timeframe
|
|
|
|
Agent TFs are **HARDCODED** (no manual `InpHtfS1..S4` inputs — removed) — see
|
|
macros `AF_E2_TF_S1..S4` in `AF_Defines.mqh`:
|
|
```
|
|
AF_E2_TF_S1 = H4 (S1 = Narrative / N)
|
|
AF_E2_TF_S2 = M30 (S2 = Context / C)
|
|
AF_E2_TF_S3 = M15 (S3 = Entry / E)
|
|
AF_E2_TF_S4 = M3 (S4 = PriceAction/P)
|
|
```
|
|
Consumers (indicator `AF_Engine3_Display`, Engine-2/3 unit tests) use these macros
|
|
directly at `Register()`. Analysis basis: **H4→M30→M15→M3** (top-down to the chart).
|
|
|
|
## 9. Agents
|
|
|
|
| Agent | Question | Assessed elements (from its own slot) | Dynamic weights |
|
|
|---|---|---|---|
|
|
| **N (Narrative)** | "Which way is the market?" | HH/HL/LL/LH pivots → trend (+clarity), CHoCH/MSS, BOS, liquidity sweep, premium/discount | Clear trend → structure dominates; flat → zones/liquidity up |
|
|
| **C (Context)** | "Which zone is price in?" | OB (opposite bar before a strong move), FVG/imbalance, S/R (pivots), premium/discount | High volatility → S/R & premium/discount down, OB/FVG up |
|
|
| **E (Entry)** | "Is there entry confirmation?" | Sweep, CHoCH, displacement, OB/FVG zones; rule **ZONE + CONFIRMATION = setup** | Strong displacement → confirmation weight up |
|
|
| **P (Price Action)** | "When to open?" | Engulfing, pin bar, inside bar, 2-bar momentum, close position in range | Ranging → reversal patterns up; trending → continuation up |
|
|
|
|
Per-agent output: `AFSignalOut { buy, sell, bias, confidence, dir, reason }`.
|
|
|
|
## 10. Aggregator (separate)
|
|
|
|
`AFAggregator::Compute(e1, slotE, n, c, e, p, out)`:
|
|
1. **Pass 1**: initial bias = Σ (base weight × confidence × bias) / Σ (base weight × confidence).
|
|
Base weights: N=0.30, C=0.30, E=0.25, P=0.15 (`AF_AGG_W_*`).
|
|
2. **Pass 2 (dynamic weights)**: agents aligned with the majority get a 1.5× boost;
|
|
aggregate `buy/sell` = Σ (effective weight × buy/sell) / Σ weights.
|
|
3. **Final signal**: `dir = BUY/SELL/WAIT` with threshold `AF_AGG_BUY_TH=0.20` and
|
|
minimum support `AF_AGG_MIN_SUP=0.50`.
|
|
4. **Levels**: `entry` = close of the closed slot-E bar; `sl/tp` based on Engine-1 ATR
|
|
(`AF_AGG_SL_ATR=1.5`, `AF_AGG_TP_ATR=2.5`) — only on BUY/SELL signals.
|
|
5. `confidence` aggregate = support × (0.7 + 0.1 × number of aligned agents).
|
|
|
|
Facade `AFEngine2Signals::Compute(e1, sN, sC, sE, sP, oN, oC, oE, oP, agg)`:
|
|
one call runs the 4 agents + aggregator (used by Engine 3 / consumers).
|
|
|
|
## 11. Unit Test
|
|
|
|
- Harness: `MQL5\Experts\AlgoForge_Engine2_UnitTest.mq5` (log prefix `AFTEST2`).
|
|
- Strategy Tester (XAUUSD, **every tick**):
|
|
- T1 output validity per agent · T2 independence (changing agent X's input slot →
|
|
other agents unchanged) + determinism · T3 closed-bar lock · T4 non-repaint
|
|
(identical output within the same bar) · T5 aggregator validity · T6 synthetic aggregator.
|
|
- Results 2026-08-21: **PASS=21534 FAIL=0** (20 days) · PASS=9942 FAIL=0 (10-day verbose).
|
|
- Note: agents use tamper slots (different TFs) for the independence test; T2
|
|
auto-retries when a tamper slot is not ready.
|
|
|
|
## 12. Implementation Constraints (additional, Phase-2 sessions)
|
|
|
|
- `replace_text_in_file` multi-line edits often fail → use single-line edits.
|
|
- The independence test needs a tamper slot with ≥ `AF_E2_MIN_BARS` (80) bars; choose
|
|
a dense TF (M6/M12/M20/M30) so it is ready even on short runs.
|
|
|
|
---
|
|
|
|
# Engine 3 — Display (reads Engine 1 & 2 output only)
|
|
|
|
Files:
|
|
- `MQL5\Include\AlgoForge\AF_Engine2_Display.mqh` — **display-context builder**
|
|
(Engine-2 layer): struct `AFDisplayData`, `AFDispZone`, `AFDispLine`,
|
|
`AFDispPivot` + `AF_BuildDisplayData()`. **All display computation** (structure,
|
|
swing points, OB, FVG, premium/discount, MTF levels, agent bias) happens HERE,
|
|
using the same pure analysis helpers as the N/C/E/P agents
|
|
(`AF_BuildSwing`, `AF_DetectSweep`, `AF_RangeStat`, etc.).
|
|
- `MQL5\Include\AlgoForge\AF_Engine3_Render.mqh` — **pure renderer** (Engine-3
|
|
layer): struct `AFRenderCfg` + `AFR_DrawAll()` / `AFR_Clear()`. NO analysis
|
|
computation; draws only from `AFDisplayData` + `AFSignalOut` + `AFAggOut`
|
|
(structure, zones, signals, dashboard like the original 75466 code).
|
|
- `MQL5\Indicators\AlgoForge\AF_Engine3_Display.mq5` — indicator (chart window):
|
|
inputs + OnInit/OnCalculate/OnDeinit; 0 buffers (object-only).
|
|
- `MQL5\Experts\AlgoForge_Engine3_UnitTest.mq5` — unit test (Strategy Tester).
|
|
|
|
## 13. Engine-3 Principles
|
|
|
|
1. **READS ONLY** Engine-1 (`AFEngine1MTF`) & Engine-2 output (`AFEngine2Signals` /
|
|
`AFAggOut` / `AFDisplayData`). Engine 3 computes no structure/zones itself.
|
|
2. Signal–display consistency: `AF_BuildDisplayData` (Engine 2) uses the same
|
|
helpers as the agents.
|
|
3. **Non-repainting**: the indicator draws only when a new closed bar appears on the
|
|
display TF (`e1.Time(sDisp,0)` changes); no redraw within the same bar.
|
|
4. Anti-freeze: Engine 1 remains the only History reader (index cache).
|
|
|
|
## 14. Display elements (equivalent to original code 75466)
|
|
|
|
| Element | Source | Description |
|
|
|---|---|---|
|
|
| BOS/CHoCH structure lines | `AFDispLine` (AF_BuildStructLines) | fractal pivots, BOS/CHoCH labels |
|
|
| HH/HL/LH/LL swing points | `AFDispPivot` (AF_ClassifyPivots) | bull/bear colors |
|
|
| Order Block | `AFDispZone` (AF_CollectOBs) | MIT filter + dedupe + size ≥ 0.15×ATR |
|
|
| FVG | `AFDispZone` (AF_CollectFVG) | MIT filter + size ≥ 0.02×ATR |
|
|
| Premium/Discount | box from `swHigh/swLow` + position | premium/equilibrium/discount bands |
|
|
| MTF PDH/PDL levels | Engine-1 D1 slot (index 1) | solid lines |
|
|
| MTF PWH/PWL levels | Engine-1 W1 slot (index 1) | dashed lines |
|
|
| Entry/SL/TP signals + arrows | `AFAggOut` | only when dir BUY/SELL |
|
|
| Dashboard | `AFDisplayData` + `AFAggOut` | bias, structure, liquidity, context, levels, N/C/E/P alignment, trade setup, legend |
|
|
|
|
## 15. Indicator inputs
|
|
|
|
- `InpHtfS1..S4` = agent N/C/E/P TFs (Engine 2) — default H1/H1/M15/M15.
|
|
- `InpDispTF` = structure/zone display TF — default `PERIOD_CURRENT` (= chart).
|
|
- Toggles: structure, swing points, OB (+count), FVG (+count), premium/discount,
|
|
MTF levels, signals, dashboard.
|
|
- LuxAlgo-style colors (same defaults as original code 75466) + transparent panel.
|
|
|
|
## 16. Unit Test
|
|
|
|
- Harness: `MQL5\Experts\AlgoForge_Engine3_UnitTest.mq5` (log prefix `AFTEST3`).
|
|
- Strategy Tester (XAUUSD, **every tick**, **Visual=1** — chart objects are only
|
|
created in visual mode).
|
|
- T1 display-data validity (structure/eqPos/levels/bias) · T2 non-repaint (identical
|
|
data within the same bar) · T3 closed-bar lock · T4 render creates chart objects
|
|
(prefix `AF3_`) · T5 render determinism (same object count).
|
|
- Results 2026-08-21: see `PROGRESS.md`.
|
|
|
|
## 17. Implementation Constraints (Phase 3)
|
|
|
|
- `ObjectCreate` in the Strategy Tester only works in **visual mode**; the test EA
|
|
probes once at startup and T4/T5 are auto-skipped (not failed) when objects are
|
|
unsupported.
|
|
- Visual mode slows the tester (20 days every tick ≈ 10 minutes) → for quick
|
|
verification run a short range (e.g. 3–5 days).
|
|
|
|
---
|
|
|
|
# Backtest Baseline (Phase 5) — verification & publication
|
|
|
|
File: `MQL5\Experts\AlgoForge_Backtest_Baseline.mq5`
|
|
+ config `MQL5\Profiles\Tester\AlgoForge_Backtest_Baseline.XAUUSD.M15.*.ini`
|
|
+ evaluation `ml\backtest_eval.py`.
|
|
|
|
## 18. Objective & Principles
|
|
|
|
- **Honest backtest** of the baseline strategy (MLP freeze `SniperGold_ML.mqh`,
|
|
AUC long 0.627 / short 0.621) **net of spread**.
|
|
- **NOT a fragile iCustom**: 19 SMC features computed internally in the EA
|
|
(identical to `SniperGold_SMC_ProPlus_v4_4.mq5` `ComputeMLFeatures` + its
|
|
dependencies), data source ONLY Engine 1 (`AFEngine1MTF`) — closed-bar lock,
|
|
anti-freeze, consistent with the Algo Forge architecture.
|
|
- Feature parameters hardcoded (`AF_BT_*` = v4.4 training defaults):
|
|
SwingLen=50, InternalLen=5, Lookback=600, GrabWindow=8, EQ thr=0.10/3 bars,
|
|
DeltaBars=10, HTF=D1/H4/H1, ConfluenceFilter=true.
|
|
|
|
## 19. Modes
|
|
|
|
| Mode | Function | Output |
|
|
|---|---|---|
|
|
| 0 | CSV long/short prob per closed M15 bar | `AlgoForge_bt_prob_*.csv` → `backtest_eval.py` (AUC/precision/calibration) |
|
|
| 1 | Net-of-spread trading in the Strategy Tester | OrderSend market (ATR SL/TP, max hold); actual tester spread; OnTester summary |
|
|
|
|
Inputs: `InpMode`, `InpThreshLong/Short` (default 0.60/0.60), `InpSL_ATR=1.0`,
|
|
`InpTP_ATR=1.5`, `InpMaxHoldBars=24` (= baseline label horizon),
|
|
`InpLot=0.01`, `InpMaxBars=700`.
|
|
|
|
## 20. Results (2026-08-21, XAUUSD M15, every tick) — HONEST
|
|
|
|
**Mode 0 (CSV/AUC)**, 2026.01.01–08.20, 14.850 bars:
|
|
|
|
| Metric | Value | Baseline freeze |
|
|
|---|---|---|
|
|
| AUC LONG | 0.5305 | 0.6270 |
|
|
| AUC SHORT | 0.5487 | 0.6207 |
|
|
| Precision LONG @0.60 | 0.5352 (n=4454) | — |
|
|
| Precision SHORT @0.60 | 0.5292 (n=7010) | — |
|
|
|
|
- Poor calibration (prob 0.65+ → long frequency 0.545) → on the XAUUSD feed (not the
|
|
XAUUSDc training feed) the baseline model is NOT calibrated. **Do not claim an edge
|
|
on another feed without a like-for-like gate.**
|
|
|
|
**Mode 1 (trade)**, 2026.05.01–08.20: trades=308, net=+1125.32, maxDD=1188.18,
|
|
PF=1.32. Note: some signals failed to execute (requote 10018, no retry); the positive
|
|
result is not significant (runtime AUC 0.53/0.55, small sample) — not an edge claim.
|
|
|
|
## 21. Constraints (Phase 5)
|
|
|
|
- Tester CSV is written to the agent sandbox (`Tester\Agent-*\MQL5\Files\`) and
|
|
**reset per run** → read/copy results immediately after the run (Python evaluation
|
|
directly from the sandbox path).
|
|
- MQL5 `FILE_CSV` delimiter in this terminal = **TAB** (not `;`/`,`) →
|
|
`backtest_eval.py` uses `delimiter="\t"`.
|
|
- `PositionSelect` overload ambiguity on build 6093 (string vs ulong) → helper
|
|
`BTSelectTicket()` (loop `PositionsTotal`/`PositionGetTicket`).
|
|
- OrderSend requotes (ret=10018) appear in the tester → recorded, no retry (honest).
|