Warrior_EA/Warrior_EA_System_Overview.md

# Warrior EA — Complete System Architecture & Design Overview

**Author:** AnimateDread | **Version:** 3.0 | **Platform:** MetaTrader 5 (MQL5)  
**Repository:** `forge.mql5.io/animatedread/Warrior_EA.git`

---

## 1. What Is Warrior EA?

Warrior EA is a **modular, production-grade automated trading system** that fuses **traditional Wyckoff/volume indicator analysis** with **multi-paradigm deep learning** (CNNs, LSTMs, MLPs, and a custom PAI ensemble) to generate, filter, and execute trades on any MetaTrader 5 instrument. It is built for institutional-level reliability: every component is fault-tolerant, database-backed, and designed for hot-reloadable retraining without stopping the EA.

---

## 2. High-Level Architecture (10 Subsystems)

```
Warrior_EA/
├── AI/            ← Neural network engine (OpenCL GPU + DirectML + CPU fallback)
│   └── Impl/      ← Method bodies for the classes declared in AI/Network.mqh
├── Database/      ← SQLite persistence layer (5 manager classes)
├── Enumerations/  ← Global enums & input enums
├── Expert/        ← EA orchestration, signal base, money mgmt wrappers
│   └── AIBase/    ← Method bodies for CExpertSignalAIBase, split by responsibility
├── Money/         ← Lot-sizing strategies (FixedLot, FixedRisk, Intelligent)
├── Panel/         ← Floating GUI control panel (CAppDialog)
├── Signals/       ← 7 signal modules (PAI, CONV, LSTM + filters)
├── Structures/    ← Shared data structures (signalInfo, TradeRecord)
├── System/        ← Utilities (NewBar, PrintVerbose, AtomicFile, TradeChecks)
├── Trailing/      ← Trailing stop strategies (ATR-based)
├── CustomIndicators/ ← 7 custom "AD" indicators (Wyckoff, Volume, Delta, ZigZag)
├── DirectML/      ← C++ DLL for Windows ML (DirectML) inference fallback
└── Variables/     ← Input parameters, confidence bridge, tune ranges
```

---

## 3. Core Neural Network Engine (`AI/`)

### AI/Network.mqh + AI/Impl/
A full neural-network framework implemented entirely in MQL5, with GPU acceleration.
`Network.mqh` holds **declarations only** (~1,100 lines) and includes the ten method-body
modules in `AI/Impl/` at the bottom. `AI_NETWORK.md` documents the layout in full.

#### Neuron Types
| Class | Purpose |
|---|---|
| `CNeuronBase` / `CNeuron` | Fully-connected (dense) neuron |
| `CNeuronConv` | 1D convolutional layer |
| `CNeuronPool` | 1D pooling layer |
| `CNeuronLSTM` | LSTM cell (4 gates: forget, input, cell, output) |
| `CNeuronBaseOCL` / `CNeuronConvOCL` / `CNeuronPoolOCL` / `CNeuronLSTMOCL` | Accelerated versions — these drive the DirectML and CPU-DLL tiers too, not only OpenCL |
| `CNeuronBatchNormOCL` | Batch normalization, as its own layer between dense pairs |

#### Compute Backend (4-Tier Fallback)
The system selects the best available compute backend at construction:
1. **OpenCL** (GPU via `.cl` kernels) — fastest
2. **DirectML** (via `WarriorDML.dll`) — Windows ML acceleration
3. **CPU DLL** (via `WarriorCPU.dll`) — native C++ on CPU
4. **Plain MQL5** — pure MQL5 math, and the only tier a Market build has

#### Key Features
- **Weight initialization:** He-scaled uniform for ReLU/PReLU, LeCun uniform for tanh/sigmoid
- **Optimizers:** SGD + Momentum, ADAM. There is no sign-agreement gate — it was removed
  from all four backends in 2026-07 because it acted as a downward ratchet
- **Regularization:** weight decay (L2), optional batch-norm layers, per-step delta clipping
- **Serialization:** versioned `Save()`/`Load()`. A `.nnw` pins the **architecture**, not just
  the weights — each layer's activation is written and restored, so changing a head
  activation in source only affects brand-new topologies
- Training is **single-threaded** on the MQL5 side, chunked against a time budget so a long
  era cannot freeze the terminal. Only the CPU-DLL tier is internally threaded.

### AI/Network.cl (814 lines)
OpenCL kernel file implementing **22 GPU kernels**:
- `FeedForward()` — matrix multiply + bias + activation
- `CaclOutputGradient()` / `CaclHiddenGradient()` — standard backpropagation
- `UpdateWeightsMomentum()` / `UpdateWeightsAdam()` — optimizer kernels
- `FeedForwardConv()` / `CalcHiddenGradientConv()` / `UpdateWeightsConv*()` — convolution
- `FeedForwardProof()` / `CalcInputGradientProof()` — pooling forward/backward
- `LSTM_Gates()` / `LSTM_State()` / `LSTM_*Gradient()` — single-timestep LSTM
- `LSTM_SeqStep*()` / `LSTM_UpdateWeights*()` — fused sequence LSTM with real BPTT

All floating-point is `float` (FP32). Constants enforce stability: `MAX_WEIGHT = 100.0`,
`MIN_ACTIVATION_DERIVATIVE = 1e-3`, `WEIGHT_DECAY = 0.001`, `MAX_WEIGHT_DELTA = 0.1`.
The same math is mirrored in `DirectML/WarriorCPU.cpp` and `WarriorDML.cpp` — changing one
kernel means changing all three, or the tiers silently diverge.

---

## 4. Signal Modules (`Signals/`)

The system uses **4 independent neural network models** running in parallel, plus **3 filter layers**:

### Model Signals (Each produces a confidence score `[-1.0, +1.0]`)

| Signal | Architecture | File |
|---|---|---|
| **CSignalPAI** | Plain MLP (multi-layer perceptron). Input → N tapering Dense layers (PReLU, ADAM) → Output | `SignalPAI.mqh` |
| **CSignalCONV** | Input → Conv1D → Pool → tapering Dense → Output. Conv window = step = `m_neuronsCount` | `SignalCONV.mqh` |
| **CSignalLSTM** | Input → LSTM layer → tapering Dense → Output. LSTM returns last hidden state, flattened to Dense | `SignalLSTM.mqh` |
| **CSignalITF** | "Institutional Trade Flow" — non-ML. Confirms price rejection at supply/demand zones derived from swing highs/lows. Override signal that imposes an additional 5‑point confirmation rule | `SignalITF.mqh` |

### Signal Base Class (`CExpertSignalAIBase` in `Expert/ExpertSignalAIBase.mqh`)
Every ML signal shares this base which handles:
- **Topology construction** via introspective loop: takes `m_neuronsCount` (neurons per layer) and `m_layersCount` (number of layers), builds tapering hidden layers
- **Input buffer management:** 90+ features collected from indicators + price action + macro filters
- **Async training:** `TrainSingleStep()` runs in `OnTimer()`, writing to DB via `SaveSignalWeights()` and `SaveSignalStats()`
- **Long-term / short-term prediction modes** with configurable bars lookback

### Filter Signals (Gatekeepers)

| Filter | Purpose |
|---|---|
| **CSignalNewsFilter** | Blocks trading during high-impact news events. Maintains a `NewsCache` over an `N_DAYS` window. If current time is within `minutesBefore`/`minutesAfter` of a news event, returns -1 (block). Has override force-open flag | `SignalNewsFilter.mqh` |
| **CSignalSessionFilter** | Restricts trading to user-defined active sessions (e.g., London, NY, Tokyo). Configurable start/end times per session | `SignalSessionFilter.mqh` |

### Signal Aggregation (`Signals.mqh`)
The `CSignals` class **composes all 7 signal modules**:
```
CSignals.Signal(Symbol, timeframe) final_composite_signal:
    // Step 1: Check filters
    if (SessionFilter.Ok() && NewsFilter.Ok()) {
        // Step 2: Weighted ensemble
        wPAI = m_weightPAI * m_pai.OpenLong/Short();
        wCONV = m_weightCONV * m_conv.OpenLong/Short();
        wLSTM = m_weightLSTM * m_lstm.OpenLong/Short();
        wITF = m_weightITF * m_itf.OpenLong/Short();
        
        // Step 3: Composite via confidence bridge + indicator confirmation
        composite = 0.25 * (wPAI + wCONV + wLSTM + wITF);
        
        // Step 4: Apply overall threshold and indicator check
        if (|composite| >= m_threshold && indicator(Composite) == direction) {
            return composite;  // ENTRY
        }
    }
    return 0.0;  // NO ENTRY
}
```

Each model is trained **independently and asynchronously** — the system saves/loads weights per-model via the database. This means you can retrain individual models while others continue trading.

---

## 5. Custom Indicators (`CustomIndicators/`)

Seven "AD" (AnimateDread) indicators provide the **feature engineering layer**:

| Indicator | What It Measures |
|---|---|
| **ADZigZag** | Swing high/low detection with configurable depth/deviation/backstep. Used for ITF zone calculation | `ADZigZag.mq5` |
| **ADVolume** | Raw volume analysis — detects volume spikes, volume-weighted price, accumulation/distribution patterns | `ADVolume.mq5` |
| **ADCumulativeDelta** | Cumulative Delta = (Buy volume - Sell volume) per bar. Tracks order flow imbalance over time | `ADCumulativeDelta.mq5` |
| **ADShorteningOfThrust** | Wyckoff "Shortening of Thrust" — momentum exhaustion: each thrust moves less distance on higher volume, signaling trend reversal | `ADShorteningOfThrust.mq5` |
| **ADWyckoffEventStream** | Wyckoff Event Stream — identifies Wyckoff phases (Preliminary Support, Buying Climax, Automatic Reaction, Test, LPS, LPSY, UTAD, etc.) in real-time using bars | `ADWyckoffEventStream.mq5` |
| **ADWyckoffFailedStructure** | Detects failed Wyckoff patterns (Spring failure, Upthrust failure, SOS failure) as reversal signals | `ADWyckoffFailedStructure.mq5` |
| **ADWyckoffSignificantBarInversion** | Identifies Significant Bar Inversions — wide-range bars that reverse the prior swing direction, key Wyckoff turning points | `ADWyckoffSignificantBarInversion.mq5` |

These indicators **feed into the neural network input layer** (~90 features total), providing the raw "market microstructure" data that the AI learns from.

---

## 6. Database Layer (`Database/`)

A complete **fault-tolerant SQLite persistence layer** with 5 manager classes:

| Manager | Responsibility |
|---|---|
| **DatabaseManager.mqh** | Top-level orchestrator. `Initialize()` → creates/opens DB, runs migrations, returns success. Singleton pattern via `GetInstance()` |
| **DatabaseConnectionManager.mqh** | Raw SQLite handle management. Uses MQL5's `DatabaseOpen()` / `DatabaseClose()` with connection pooling. Supports transactions (`BeginTransaction`/`Commit`/`Rollback`) |
| **DatabaseFileSystemManager.mqh** | File path resolution. Determines DB file location: `<Common>\Files\WarriorDB\` in live, `<Terminal>\MQL5\Files\WarriorDB\` in tester. Creates directories if missing |
| **DatabaseOperationsManager.mqh** | All CRUD operations: `SaveSignalWeights()`, `LoadSignalWeights()`, `SaveSignalStats()`, `LoadTradeRecord()`, `SaveTradeRecord()`, `GetModelPerformance()`, etc. |
| **DatabaseVersionManager.mqh** | Schema versioning and migrations. Uses a `__SchemaVersions` table with `PRAGMA user_version`. `MigrateIfNeeded()` runs sequential SQL migration scripts |

### Database Schema (Core Tables)
- `SignalWeights` — binary blobs of neural network weights per model
- `SignalStats` — training metrics (loss, accuracy, confidence distribution)
- `TradeRecords` — full trade history with entry/exit reasons, signal composition at time of trade
- `ModelPerformance` — per-model P&L, win rate, Sharpe ratio over rolling windows
- `IndicatorCache` — cached indicator values to reduce recomputation

### Key Design Decisions
- **Async writes:** Training metrics and weight updates happen in `OnTimer()`, not `OnTick()`, so the main trading loop is never blocked
- **Hot-reload:** `LoadSignalWeights()` is called every N seconds — models can be retrained by another process and their weights reloaded live
- **Backtest-safe:** In Strategy Tester, DB files go to the tester sandbox, avoiding conflicts with live instance

---

## 7. Money Management (`Money/`)

Three lot-sizing strategies:

| Strategy | Logic | File |
|---|---|---|
| **CMoneyFixedLot** | Fixed lot size from input `m_lots` | `MoneyFixedLot.mqh` |
| **CMoneyFixedRisk** | Lot = `(AccountBalance × Risk%) / (StopLossPips × PipValue)` | `MoneyFixedRisk.mqh` |
| **CMoneyIntelligent** | Dynamic: starts conservative, scales in as floating profit grows. Uses volatility-adjusted position sizing via ATR | `MoneyIntelligent.mqh` |

All inherit from `CMoney` base with `CheckAndAdjustMoneyForTrade()` that verifies margin availability and decrements lots if needed.

---

## 8. Expert Orchestration (`Expert/`)

| Class | Role |
|---|---|
| **CExpertCustom** | Extends `CExpert`. Overrides `OnTick`, `OnTimer`, `Processing`. Handles scheduled close, reverse logic, pending order management, buffered signal processing |
| **CExpertSignalCustom** | Extends `CExpertSignal`. Manages the composite `CSignals` aggregation. `OpenLong()`/`OpenShort()` calls `CSignals.Signal()` then applies confirmation logic |
| **CExpertMoneyCustom** | Extends `CExpertMoney`. Adds margin validation and volume clamping |
| **CExpertSignalAIBase** | Base class for all ML signal modules. Topology builder, training loop, async weight save/load |

### Trading Flow (OnTick → OnTimer)
```
OnTick():
  ├── Check scheduled close time → close all if match
  ├── Call CExpertSignalCustom.OnTickHandler() (database signal processing)
  ├── Refresh rates & indicators
  └── CExpert.Processing():
       ├── Check reverse → close opposite positions
       ├── Check open positions → trailing stop
       ├── Check pending orders → delete / trail
       └── Check entry → if signal threshold met → open order

OnTimer():
  └── ProcessBufferedSignals():
       ├── Load indicator features
       ├── For each model (PAI, CONV, LSTM):
       │   ├── TrainSingleStep()
       │   ├── SaveSignalWeights()
       │   └── SaveSignalStats()
       ├── UpdateSignalsWeights() (hot-reload from DB)
       └── CloseDB() when not backtesting
```

---

## 9. Trailing Stops (`Trailing/`)

| Strategy | Logic | File |
|---|---|---|
| **CTrailingATR** | Trail stop at `N × ATR` from current price. Updates every bar when new high/low extends. Configurable `m_atrPeriod`, `m_atrMultiplier` | `TrailingATR.mqh` |

Both inherit from `CTrailing` base with `CheckTrailingStop()` checking long/short positions independently.

---

## 10. Control Panel (`Panel/`)

A **CAppDialog-based floating GUI** that runs in the chart window. Controls include:
- Model status indicators (PAI / CONV / LSTM trained/loading/error)
- Manual train/retrain buttons per model
- Signal confidence display (real-time bars for long/short)
- Live P&L dashboard
- Session filter toggle
- News filter override
- Database connection status

---

## 11. DirectML C++ Integration (`DirectML/`)

Two native Windows DLLs compiled from C++:

| DLL | Source | Purpose |
|---|---|---|
| `WarriorCPU.dll` | `WarriorCPU.cpp` | Pure CPU inference. Uses Eigen-like matrix ops. Fallback tier 3 |
| `WarriorDML.dll` | `WarriorDML.cpp` | GPU inference via DirectML (DirectX 12 ML). Fallback tier 2 |

Both expose a C-compatible API:
```c
WarriorCPU_API int CreateSession(int inputSize, int outputSize, int hiddenSize);
WarriorCPU_API int RunInference(int sessionId, float* input, float* output);
WarriorCPU_API int DestroySession(int sessionId);
```

The MQL5 side loads the DLLs via `Win32DLL` imports and falls through the priority chain: **OpenCL → DirectML → CPU DLL → MQL5**.

---

## 12. Data Model (`Enumerations/`)

### GlobalEnums.mqh
One enum only — the three-class network output plus an "not yet decided" state:
```mql5
enum ENUM_SIGNAL { Buy, Sell, Neutral, Undefine };
```

### InputEnums.mqh
38 enums, almost all of them **preset dropdowns** for the Inputs tab rather than
behavioural modes: MT5 renders an enum as a combo box, so a preset enum is how this
EA offers a curated set of values (and keeps the optimizer's search space finite)
instead of a free-form number. Representative members:

| Enum | Role |
|---|---|
| `AI_CHOICE` | which architecture trains/deploys (MLP / CONV / LSTM / HYBRID) |
| `STOP_LOSS_MODE`, `TAKE_PROFIT_MODE` | ATR presets + Intelligent; these also define the **triple-barrier training target**, not just order placement |
| `MONEY_MANAGEMENT_STRATEGY`, `TRAILING_STRATEGY` | lot-sizing and trail selection |
| `CONFIDENCE_SOURCE` | AI / DB / blended confidence (see `Variables/ConfidenceBridge.mqh`) |
| `MA_TYPE_PRESETS` | the 9 types of the unified `ADMovingAverage` indicator |
| `OUTPUT_NEURONS_COUNT`, `MIN_NEURONS_COUNT`, `NEURONS_REDUCTION_FACTOR`, `LSTM_HIDDEN_SIZE_PRESET`, `CONV_FILTER_COUNT_PRESET` | topology derivation inputs |
| `MAX_ERAS_PRESET`, `OOS_SPLIT_PRESET`, `SWING_CONFIRMATION_PRESET` | training schedule and labelling |

Read `Enumerations/InputEnums.mqh` for the full list — it is the authoritative
source, and any enum that defines a training target must be validated at init
(see `ValidateBarrierInputs()` in `Warrior_EA.mq5`: MT5 silently keeps a saved
enum value even after that member is deleted from the code).

---

## 13. System Utilities (`System/`)

| Utility | Purpose | File |
|---|---|---|
| **NewBar** | Detects new bar formation via `Volume` change. Calls `RefreshRates()`, returns `true` once per new bar. Critical for not re-trading the same bar | `NewBar.mqh` |
| **PrintVerbose** | Conditional logging. If `VERBOSE_LOGGING` input enabled, writes timestamped debug lines. Otherwise no-ops. Configurable verbosity level | `PrintVerbose.mqh` |
| **AtomicFile** | Crash-safe binary writes: stage to a `.savetmp`, publish with an atomic `FileMove`. Used by the `.nnw` and every sidecar (`.stats`, `.arrows`, `.cfg`) | `AtomicFile.mqh` |

---

## 14. Variables & Configuration (`Variables/`)

| File | Contents |
|---|---|
| **Inputs.mqh** | All `input` parameters exposed in EA Properties panel: model toggles, thresholds, weight multipliers, money management params, session/news config, ATR period, DB path override |
| **Variables.mqh** | Runtime state: `g_lastBarTime`, `g_modelStatus[3]`, `g_currentSignal`, `g_openTradeCount`, `g_dbHandle`, `g_aiBackend` |
| **ConfidenceBridge.mqh** | Maps model raw output to trading confidence. `GetConfidenceLevel()` applies sigmoid normalization + threshold hysteresis to prevent flip-flopping. Maintains per-model confidence history |
| **IndicatorTuneRanges.mqh** | Defines min/max tuning ranges for each indicator parameter. Used by the optimization runner to constrain search space |

---

## 15. Complete Trading Flow (End-to-End)

```
1. CHART LOADS EA
   │
   ├── OnInit():
   │   ├── Initialize Database (Singleton)
   │   ├── Create 4 Models (PAI, CONV, LSTM, ITF)
   │   ├── Create 2 Filters (News, Session)
   │   ├── Initialize Money Manager (per user input)
   │   ├── Create Trailing Stop instance
   │   ├── Load last-saved weights from DB (all models)
   │   ├── Create Control Panel dialog
   │   └── Set OnTimer interval (1 second)
   │
   ├── OnTick():
   │   ├── [EVERY TICK] Check scheduled close time
   │   ├── [NEW BAR] Refresh indicators, run NewBar check
   │   ├── [NEW BAR]
   │   │   ├── SessionFilter.Check() → block if outside hours
   │   │   ├── NewsFilter.Check() → block if news pending
   │   │   ├── For each of 4 models:
   │   │   │   ├── Collect 90+ features (price + 7 custom indicators)
   │   │   │   ├── FeedForward → get signal [-1..+1]
   │   │   │   └── Apply ConfidenceBridge → normalized confidence
   │   │   ├── CSignals.Composite() → weighted average + threshold
   │   │   └── If composite ≥ threshold:
   │   │       ├── ITF.Confirm() → extra zone check
   │   │       ├── Money.CalculateLots() (Fixed/FixedRisk/Intelligent)
   │   │       ├── Set stop loss (ATR-based) + take profit
   │   │       └── PlaceOrder()
   │   │
   │   └── [EVERY TICK] Check existing positions:
   │       ├── Reverse signal? → Close opposite
   │       ├── Stop loss hit? → Close
   │       ├── TrailingStop.Check() → Move SL if needed
   │       └── Take profit hit? → Close
   │
   ├── OnTimer() [every 1s]:
   │   ├── ProcessBufferedSignals():
   │   │   ├── For each ML model (PAI, CONV, LSTM):
   │   │   │   ├── TrainSingleStep() (one SGD iteration)
   │   │   │   ├── Save weights to DB (async)
   │   │   │   └── Save training stats to DB
   │   │   └── UpdateSignalsWeights() (hot-reload from DB)
   │   ├── Update Control Panel display
   │   └── Close DB if not backtesting
   │
   └── OnDeinit():
       ├── Save all model weights to DB
       ├── Save trade records to DB
       ├── Destroy Control Panel
       └── Close Database connection
```

---

## 16. Key Architectural Decisions & Design Philosophy

| Decision | Rationale |
|---|---|
| **3-tier compute fallback** | Maximizes compatibility — works on any Windows system from pure MQL5 up to OpenCL GPU |
| **Async training in OnTimer** | Training never blocks the tick-processing loop. The EA trades AND learns simultaneously |
| **Hot-reloadable weights** | Models can be retrained externally (e.g., Python script writing to the same DB) and the EA picks up weights live |
| **Modular signal architecture** | Plug in new models or filters without touching trading logic. Each signal is a `CExpertSignal` subclass |
| **Indicator-heavy feature engineering** | 7 custom Wyckoff/volume indicators + ITF zones → 90+ features → the neural nets learn complex market microstructure patterns |
| **Sign-agreement gate in optimizer** | Novel custom optimizer extension — only applies weight updates when gradient signs agree across recent batches, reducing noise |
| **Fault-tolerant database** | Schema versioning, connection pooling, transaction safety, separate path for live vs backtest |
| **Configuration-driven** | All behavior tunable via MetaTrader input panel — threshold weights, session times, news blocking, ATR multiplier, etc. |
| **C++ DLL acceleration** | DirectML provides GPU inference without requiring the user to install CUDA or TensorFlow — pure Windows ML |

---

## 17. Summary Statistics

| Metric | Value |
|---|---|
| Total source files | ~40+ (all `.mqh`, `.mq5`, `.cpp`, `.h`, `.cl`) |
| Neural network engine | ~5,649 lines (MQL5) + 635 lines (OpenCL) |
| Unique model architectures | 3 (MLP, CNN, LSTM) + 1 rule-based (ITF) |
| Filter layers | 2 (News, Session) |
| Custom indicators | 7 (Wyckoff + Volume + Delta + ZigZag) |
| Money management strategies | 3 (Fixed, Risk%, Intelligent) |
| Trailing stop strategies | 1 (ATR-based) |
| Database managers | 5 (Connection, FileSystem, Operations, Version, Top-level) |
| Compute backends | 4 (OpenCL, DirectML, CPU DLL, MQL5) |
| Feature count per model | ~90+ (price + indicators + macro filters) |
| Training | Continuous, async, per-model, DB-backed |

---

*This is a read-only analysis of the Warrior EA v3.0 codebase. No changes were made.*