mql5/Experts/Advisors/DualEA/docs/Execution-Pipeline.md

# Execution Pipeline — DualEA System

**Complete trade execution flow from signal to position**

---

## Pipeline Overview

```
OnTick/OnTimer
    ↓
[1. Early Safety & Filters] ← circuit breaker, memory guard, news filter (PaperEA); risk/spread/session/news/etc (LiveEA)
    ↓ PASS
[2. Strategy Selection] ← selector scoring (PaperEA) or external orchestration (LiveEA)
    ↓ SELECTED
[3. Signal Generation] ← PaperEA: indicator signal generator registry; LiveEA: strategy bridge is available but not called from OnTick
    ↓ SIGNAL
[4. 8-Stage Gates] ← GateManager (PaperEA) / LiveEA early + risk4 gates
    ↓ PASS
[5. Policy & Insights Gating] ← LiveEA: per-slice policy + insights gating + exploration; PaperEA: minimal policy load + min_conf gate
    ↓ PASS
[6. Trade Execution] ← TradeManager.ExecuteOrder() (both EAs)
    ↓ SUCCESS
[7. Post-Execution] ← KB logging, features export, telemetry, learning
```

---

## Stage 1: Signal Generation

**Entry Points:**
- `OnTick()`: Real-time tick processing
- `OnTimer()`: Used today for maintenance and reload polling (PaperEA); LiveEA uses a timer for insights auto-reload polling

**PaperEA_v2 Signal Generation (actual):**
- Strategy selection chooses from a **23-strategy list** (e.g., `ADXStrategy`, `RSIStrategy`, `MACDStrategy`, `IchimokuStrategy`, etc.).
- Signal generation is executed through `Include/StrategySignalGenerators.mqh` via `GenerateSignalFromStrategy()`.
- Output is a `TradingSignal` (from `Include/GateManager.mqh`) with `price`, `sl`, `tp`, `volume`, and `confidence` seeded.

**LiveEA Signal Generation (actual):**
- `LiveEA.mq5` provides the gating and execution pipeline but **does not call the strategy bridge from `OnTick()`**.
- Strategy generation exists as `LiveEA_StrategyBridge.mqh` (21 default `IStrategy` implementations) and can be used by an external orchestrator or future wiring.

**Signal Structure:**
- `strategy_name`, `direction` (1=Buy, -1=Sell)
- `confidence` (0.0-1.0), `entry_price`, `stop_loss`, `take_profit`, `lot_size`
- `reason` (human-readable), `timestamp`
- Strategy-specific indicator values

---

## Stage 2: Early Phase Validation

Quick rejection before expensive gate processing.

**Checks:**
1. **Circuit breakers / risk limits**
2. **News blackout** (Common Files CSV `DualEA\news_blackouts.csv`)
3. **Session/trading hours caps** (where enabled)
4. **Spread and margin checks** (LiveEA)

**NoConstraintsMode**:
- PaperEA default is `true` and bypasses many constraints, but **circuit breaker + memory guard remain enforced**.
- LiveEA default is `false`; when `true`, many gates become **shadow/diagnostic only** (it logs but does not block on early gates).

---

## Stage 3: 8-Stage Gate Processing

Progressive filtering through configurable gates.

**Important reality check:** In `Include/GateManager.mqh`, several gates are currently implemented as simplified/deterministic checks (for repeatable Strategy Tester runs), not a full production-grade market simulation.

### Gate 1: Signal Rinse
- **Purpose**: Basic validation
- **Checks**: `confidence ≥ G1_MinConfidence`, `strength ≥ G1_MinStrength`

### Gate 2: Market Soap
- **Purpose**: Market context
- **Checks**: ATR within `[G2_MinVolatility, G2_MaxVolatility]`, portfolio correlation ≤ `G2_MaxCorrelation`, regime tradeable

### Gate 3: Strategy Scrub
- **Purpose**: Strategy-specific validation
- **Checks**: History bars ≥ `G3_MinHistoryBars`, strategy.ValidateSignal()

### Gate 4: Risk Wash
- **Purpose**: Risk validation
- **Checks**: Risk/trade ≤ `G4_MaxRiskPct`, portfolio risk ≤ `G4_MaxPortfolioRisk`, RR ≥ `G4_MinRiskReward`

### Gate 5: Performance Wax
- **Purpose**: Backtest performance
- **Checks**: Backtest WR ≥ `G5_MinBacktestWinRate`, trades ≥ `G5_MinBacktestTrades`, R ≥ `G5_MinBacktestRMultiple`

### Gate 6: ML Polish
- **Purpose**: ML confidence
- **Checks**: ML confidence ≥ `G6_MinMLConfidence`, ML available if `G6_RequireMLAvailable`

### Gate 7: Live Clean
- **Purpose**: Live market conditions
- **Checks**: Spread, market open, liquidity

### Gate 8: Final Verify
- **Purpose**: Pre-execution validation
- **Checks**: Broker constraints (min/max lots), duplicate signals

**Learning Integration**: Each gate decision recorded via `LearningBridge` for threshold optimization.

---

## Stage 4: Strategy Selection

### StrategySelector Scoring (PaperEA)

```cpp
score = (winRate × W1) + (normRMultiple × W2) + (normSharpe × W3) + (normTrades × W4)
if(SelectorUseRecency) score ×= recencyDecay^daysSinceLastTrade
```

**Weights (actual PaperEA inputs):** `SelW_PF`, `SelW_Exp`, `SelW_WR`, `SelW_DD` with optional recency overlay.

### Insights Gating

Load insights.json slice for `strategy|symbol|timeframe`:

- **No slice found**: Enter exploration mode if enabled
- **Slice exists**: Check `winRate ≥ InsightsMinWinRate`, `totalTrades ≥ InsightsMinTotalTrades`, `avgR ≥ InsightsMinAvgR`

### Exploration Mode

**Triggered when**: No insights slice exists

**Caps (current defaults in code):**
- Daily: `ExploreMaxPerSlicePerDay` (default 100)
- Weekly: `ExploreMaxPerSlice` (default 100)

**Counters**: Persist in `explore_counts.csv` and `explore_counts_day.csv`

**NoConstraintsMode**: Bypasses insights gating and exploration caps entirely.

---

## Stage 5: Policy Application

### Policy Loading

**LiveEA (actual):**
- Loads `DualEA\policy.json` from **Common Files**.
- Parses per-slice fields including `p_win`, and optional scaling keys `sl_scale`, `tp_scale`, `trail_scale`.
- Applies a slice lookup order:
  - Exact `strategy+symbol+timeframe`
  - Aggregate `strategy+symbol` with `timeframe=-1`
  - Aggregate `strategy` with `symbol="*"` and `timeframe=-1`

**PaperEA_v2 (actual):**
- Loads `DualEA\policy.json` from Common Files and treats the policy as “loaded” if it can find the string `min_confidence`.
- `g_policy_min_conf` defaults to `0.5`.
- Per-slice parsing/scaling is **not currently wired** in PaperEA’s `Policy_Load()`.
- HTTP polling exists (see `PolicyServerUrl`, `PolicyHttpPollPercent`) but still only checks for `min_confidence`.

### Policy Gating Logic

```
if policy.slices == 0:
    if FallbackWhenNoPolicy && (FallbackDemoOnly→demo OR !FallbackDemoOnly):
        → FALLBACK: neutral scaling, bypass insights/exploration
    else:
        → BLOCK
        
if slice missing:
    if FallbackWhenSliceMissing && (demo check as above):
        → FALLBACK: neutral scaling
    else:
        → BLOCK
        
if slice.probability < policy.min_confidence:
    → BLOCK
    
else:
    → PASS: Apply scaling
```

### Policy Scaling

**LiveEA (actual):**
- Adjusts SL/TP distance from entry using `sl_scale` / `tp_scale`.
- Adjusts trailing distance using `trail_scale`.

**PaperEA_v2 (current):**
- Policy gating is applied as a **minimum confidence check** (and the scaling path is currently neutral/placeholder in `ApplyPolicyGating()`).

---

## Stage 6: Risk Management

### Position Sizing

**Base Sizing**:
```
pipRisk = |entry - SL| / Point / 10
riskAmount = Balance × (G4_MaxRiskPct / 100)
lots = riskAmount / (pipRisk × tickValue)
lots = NormalizeLots(lots)  // Broker constraints
```

**Correlation Adjustment** (LiveEA with PositionManager):
```
avgCorr = CorrelationManager.GetAverageCorrelation(symbol)
dampening = 1.0 - (avgCorr × 0.5)
adjustedLots = lots × dampening
adjustedLots = max(adjustedLots, max(lots × PM_CorrMinMult, PM_CorrMinLots))
```

### Circuit Breakers

- **Daily Loss**: Block if daily PnL < `-MaxDailyLossPct`
- **Drawdown**: Block if drawdown from peak > `MaxDrawdownPct`
- **Consecutive Losses**: Block after `MaxConsecutiveLosses` losses
- **Cooldown**: After breaker trips, wait `CircuitCooldownSec` seconds

---

## Stage 7: Trade Execution

### TradeManager.Execute()

**PaperEA_v2 (actual):** Places **real MT5 orders on demo accounts** via `CTradeManager::ExecuteOrder()`.

**LiveEA (actual):** Places **real MT5 orders** via `CTradeManager::ExecuteOrder()`.
- Market orders: Immediate execution
- Pending orders: BuyStop, SellStop, BuyLimit, SellLimit
- SL/TP normalization to broker tick size
- Volume normalization (min/max/step)

**Trailing Stops**: Initialized if `TrailEnabled=true`
- Fixed-points: `TrailActivationPoints`, `TrailDistancePoints`, `TrailStepPoints`
- ATR-based: `TrailATRMultiplier × ATR` for distance

---

## Stage 8: Post-Execution

### Knowledge Base Logging

**knowledge_base.csv**:
```
timestamp,symbol,type,entry_price,stop_loss,take_profit,close_price,profit,strategy_id
```

**knowledge_base_events.csv**:
```
timestamp,strategy,retcode,deal,order
```

### Features Export

**features.csv** (long format):
```
timestamp,symbol,timeframe,strategy,direction,confidence,...,indicator_1,indicator_2,...
```
- 50+ features per trade
- Automatic ~100MB rotation (rename to `*.bak`); no compression is performed in MQL5

### Telemetry & Logging

**Telemetry Events**:
- `gate_decision`: Pass/block with reason, latency
- `trade_execution`: Order placement, retcode
- `policy_load`, `insights_rebuild`, `threshold_adjust`
- `risk4_*`: Risk gate events (spread, margin, drawdown)
- `pm_*`: PositionManager events (correlation sizing, scale-ins)
- `p5_*`: Selector events (refresh, rescore, tune)

**UnifiedTradeLogger**: Daily JSON logs with full lifecycle
```json
{
  "timestamp": "2025-01-15T10:30:00Z",
  "strategy": "ADXStrategy",
  "symbol": "EURUSD",
  "timeframe": 60,
  "direction": 1,
  "entry": 1.0850,
  "sl": 1.0800,
  "tp": 1.0950,
  "lots": 0.10,
  "gates": ["signal_rinse:pass", "market_soap:pass", ...],
  "policy": {"confidence": 0.75, "sl_mult": 1.0, "tp_mult": 1.2},
  "outcome": "closed",
  "profit": 15.50,
  "r_multiple": 1.2
}
```

### Learning System Update

**LearningBridge**: Records `CSignalDecision` with:
- All 8 gate decisions (pass/fail, reason, latency)
- Final outcome (executed, blocked, reason)
- Trade result (if executed): profit, R-multiple, duration

**GateLearningSystem**: Updates gate thresholds
- Hybrid learning: immediate (5% rate) + batch (every 20 trades)
- Target success rate per gate configurable
- Adjusts thresholds to optimize pass rate vs trade quality

---

## Live Trading Flow (LiveEA)

**Differences from PaperEA_v2 (as wired today):**
1. **Insights auto-reload loop**: LiveEA requests rebuild by creating `DualEA\insights.reload` and waits for `DualEA\insights.ready`.
2. **Per-slice policy scaling**: LiveEA parses per-slice policy entries and applies SL/TP/trailing scaling.
3. **Risk gates**: LiveEA includes spread/session/margin/consecutive-loss gating in the EA itself.

**Safety Features**:
- Shadow mode possible (log decisions, don't execute)
- Policy fallback with demo-only restriction
- Multiple circuit breakers
- One-execution-per-slice-per-minute de-dup (Phase 6 - planned)

---

## Error Handling

### Broker Errors

**Retcode Handling**:
- `TRADE_RETCODE_DONE`: Success → proceed to post-execution
- `TRADE_RETCODE_REQUOTE`: Retry with new price (max 3 attempts)
- `TRADE_RETCODE_REJECT`: Log + telemetry, don't retry
- `TRADE_RETCODE_NO_MONEY`: Trip circuit breaker, halt trading
- Other errors: Log, telemetry, optional cooldown

### File I/O Errors

**KB Write Failures**:
- Retry with exponential backoff (3 attempts)
- Create directory if missing
- Log to Journal on persistent failure
- Continue execution (non-fatal)

**Policy/Insights Load Failures**:
- Use last-known-good cached version
- Log warning + telemetry event
- Fallback to neutral scaling if allowed
- Reload attempt on next timer cycle

### Learning System Failures

- Non-fatal: Log errors, continue trading
- Disable learning if consecutive failures > 5
- Telemetry event for monitoring

---

## Performance Characteristics

### Latency Targets

- **Signal → Gate entry**: <5ms
- **8-stage gates**: <50ms total (target <10ms per gate)
- **Policy/insights lookup**: <2ms (in-memory cache)
- **KB write**: <10ms (non-blocking preferred)
- **Telemetry emit**: <1ms
- **Total pipeline**: <100ms (OnTick → OrderSend)

### Memory Footprint

- **Strategies (21)**: ~500KB
- **Gate Manager**: ~100KB
- **Policy cache**: ~50KB
- **Insights cache**: ~200KB
- **Telemetry buffer**: ~500KB
- **Total**: ~1.5MB per EA instance

### File I/O

- **KB writes**: Append-only, buffered every 10 trades
- **Features export**: Buffered every 50 trades
- **Telemetry flush**: Every 5 minutes or 1000 events
- **Policy/insights reload**: On timer (every 60 minutes) or .reload trigger

---

**See Also:**
- [Configuration-Reference.md](Configuration-Reference.md) - All input parameters
- [Observability-Guide.md](Observability-Guide.md) - Telemetry and monitoring
- [Policy-Exploration-Guide.md](Policy-Exploration-Guide.md) - Policy fallback and exploration details