Princeec13/mql5
0
0
Fork 1
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
mql5/Experts/Advisors/DualEA/docs/Execution-Pipeline.md
Princeec13 0b557e494f
2026-02-05 01:22:42 -05:00

12 KiB
Raw Permalink Blame History

Execution Pipeline — DualEA System

Complete trade execution flow from signal to position

Pipeline Overview

OnTick/OnTimer
    ↓
[1. Early Safety & Filters] ← circuit breaker, memory guard, news filter (PaperEA); risk/spread/session/news/etc (LiveEA)
    ↓ PASS
[2. Strategy Selection] ← selector scoring (PaperEA) or external orchestration (LiveEA)
    ↓ SELECTED
[3. Signal Generation] ← PaperEA: indicator signal generator registry; LiveEA: strategy bridge is available but not called from OnTick
    ↓ SIGNAL
[4. 8-Stage Gates] ← GateManager (PaperEA) / LiveEA early + risk4 gates
    ↓ PASS
[5. Policy & Insights Gating] ← LiveEA: per-slice policy + insights gating + exploration; PaperEA: minimal policy load + min_conf gate
    ↓ PASS
[6. Trade Execution] ← TradeManager.ExecuteOrder() (both EAs)
    ↓ SUCCESS
[7. Post-Execution] ← KB logging, features export, telemetry, learning

Stage 1: Signal Generation

Entry Points:

  • OnTick(): Real-time tick processing
  • OnTimer(): Used today for maintenance and reload polling (PaperEA); LiveEA uses a timer for insights auto-reload polling

PaperEA_v2 Signal Generation (actual):

  • Strategy selection chooses from a 23-strategy list (e.g., ADXStrategy, RSIStrategy, MACDStrategy, IchimokuStrategy, etc.).
  • Signal generation is executed through Include/StrategySignalGenerators.mqh via GenerateSignalFromStrategy().
  • Output is a TradingSignal (from Include/GateManager.mqh) with price, sl, tp, volume, and confidence seeded.

LiveEA Signal Generation (actual):

  • LiveEA.mq5 provides the gating and execution pipeline but does not call the strategy bridge from OnTick().
  • Strategy generation exists as LiveEA_StrategyBridge.mqh (21 default IStrategy implementations) and can be used by an external orchestrator or future wiring.

Signal Structure:

  • strategy_name, direction (1=Buy, -1=Sell)
  • confidence (0.0-1.0), entry_price, stop_loss, take_profit, lot_size
  • reason (human-readable), timestamp
  • Strategy-specific indicator values

Stage 2: Early Phase Validation

Quick rejection before expensive gate processing.

Checks:

  1. Circuit breakers / risk limits
  2. News blackout (Common Files CSV DualEA\news_blackouts.csv)
  3. Session/trading hours caps (where enabled)
  4. Spread and margin checks (LiveEA)

NoConstraintsMode:

  • PaperEA default is true and bypasses many constraints, but circuit breaker + memory guard remain enforced.
  • LiveEA default is false; when true, many gates become shadow/diagnostic only (it logs but does not block on early gates).

Stage 3: 8-Stage Gate Processing

Progressive filtering through configurable gates.

Important reality check: In Include/GateManager.mqh, several gates are currently implemented as simplified/deterministic checks (for repeatable Strategy Tester runs), not a full production-grade market simulation.

Gate 1: Signal Rinse

  • Purpose: Basic validation
  • Checks: confidence ≥ G1_MinConfidence, strength ≥ G1_MinStrength

Gate 2: Market Soap

  • Purpose: Market context
  • Checks: ATR within [G2_MinVolatility, G2_MaxVolatility], portfolio correlation ≤ G2_MaxCorrelation, regime tradeable

Gate 3: Strategy Scrub

  • Purpose: Strategy-specific validation
  • Checks: History bars ≥ G3_MinHistoryBars, strategy.ValidateSignal()

Gate 4: Risk Wash

  • Purpose: Risk validation
  • Checks: Risk/trade ≤ G4_MaxRiskPct, portfolio risk ≤ G4_MaxPortfolioRisk, RR ≥ G4_MinRiskReward

Gate 5: Performance Wax

  • Purpose: Backtest performance
  • Checks: Backtest WR ≥ G5_MinBacktestWinRate, trades ≥ G5_MinBacktestTrades, R ≥ G5_MinBacktestRMultiple

Gate 6: ML Polish

  • Purpose: ML confidence
  • Checks: ML confidence ≥ G6_MinMLConfidence, ML available if G6_RequireMLAvailable

Gate 7: Live Clean

  • Purpose: Live market conditions
  • Checks: Spread, market open, liquidity

Gate 8: Final Verify

  • Purpose: Pre-execution validation
  • Checks: Broker constraints (min/max lots), duplicate signals

Learning Integration: Each gate decision recorded via LearningBridge for threshold optimization.

Stage 4: Strategy Selection

StrategySelector Scoring (PaperEA)

score = (winRate × W1) + (normRMultiple × W2) + (normSharpe × W3) + (normTrades × W4)
if(SelectorUseRecency) score ×= recencyDecay^daysSinceLastTrade

Weights (actual PaperEA inputs): SelW_PF, SelW_Exp, SelW_WR, SelW_DD with optional recency overlay.

Insights Gating

Load insights.json slice for strategy|symbol|timeframe:

  • No slice found: Enter exploration mode if enabled
  • Slice exists: Check winRate ≥ InsightsMinWinRate, totalTrades ≥ InsightsMinTotalTrades, avgR ≥ InsightsMinAvgR

Exploration Mode

Triggered when: No insights slice exists

Caps (current defaults in code):

  • Daily: ExploreMaxPerSlicePerDay (default 100)
  • Weekly: ExploreMaxPerSlice (default 100)

Counters: Persist in explore_counts.csv and explore_counts_day.csv

NoConstraintsMode: Bypasses insights gating and exploration caps entirely.

Stage 5: Policy Application

Policy Loading

LiveEA (actual):

  • Loads DualEA\policy.json from Common Files.
  • Parses per-slice fields including p_win, and optional scaling keys sl_scale, tp_scale, trail_scale.
  • Applies a slice lookup order:
    • Exact strategy+symbol+timeframe
    • Aggregate strategy+symbol with timeframe=-1
    • Aggregate strategy with symbol="*" and timeframe=-1

PaperEA_v2 (actual):

  • Loads DualEA\policy.json from Common Files and treats the policy as “loaded” if it can find the string min_confidence.
  • g_policy_min_conf defaults to 0.5.
  • Per-slice parsing/scaling is not currently wired in PaperEA’s Policy_Load().
  • HTTP polling exists (see PolicyServerUrl, PolicyHttpPollPercent) but still only checks for min_confidence.

Policy Gating Logic

if policy.slices == 0:
    if FallbackWhenNoPolicy && (FallbackDemoOnly→demo OR !FallbackDemoOnly):
        → FALLBACK: neutral scaling, bypass insights/exploration
    else:
        → BLOCK
        
if slice missing:
    if FallbackWhenSliceMissing && (demo check as above):
        → FALLBACK: neutral scaling
    else:
        → BLOCK
        
if slice.probability < policy.min_confidence:
    → BLOCK
    
else:
    → PASS: Apply scaling

Policy Scaling

LiveEA (actual):

  • Adjusts SL/TP distance from entry using sl_scale / tp_scale.
  • Adjusts trailing distance using trail_scale.

PaperEA_v2 (current):

  • Policy gating is applied as a minimum confidence check (and the scaling path is currently neutral/placeholder in ApplyPolicyGating()).

Stage 6: Risk Management

Position Sizing

Base Sizing:

pipRisk = |entry - SL| / Point / 10
riskAmount = Balance × (G4_MaxRiskPct / 100)
lots = riskAmount / (pipRisk × tickValue)
lots = NormalizeLots(lots)  // Broker constraints

Correlation Adjustment (LiveEA with PositionManager):

avgCorr = CorrelationManager.GetAverageCorrelation(symbol)
dampening = 1.0 - (avgCorr × 0.5)
adjustedLots = lots × dampening
adjustedLots = max(adjustedLots, max(lots × PM_CorrMinMult, PM_CorrMinLots))

Circuit Breakers

  • Daily Loss: Block if daily PnL < -MaxDailyLossPct
  • Drawdown: Block if drawdown from peak > MaxDrawdownPct
  • Consecutive Losses: Block after MaxConsecutiveLosses losses
  • Cooldown: After breaker trips, wait CircuitCooldownSec seconds

Stage 7: Trade Execution

TradeManager.Execute()

PaperEA_v2 (actual): Places real MT5 orders on demo accounts via CTradeManager::ExecuteOrder().

LiveEA (actual): Places real MT5 orders via CTradeManager::ExecuteOrder().

  • Market orders: Immediate execution
  • Pending orders: BuyStop, SellStop, BuyLimit, SellLimit
  • SL/TP normalization to broker tick size
  • Volume normalization (min/max/step)

Trailing Stops: Initialized if TrailEnabled=true

  • Fixed-points: TrailActivationPoints, TrailDistancePoints, TrailStepPoints
  • ATR-based: TrailATRMultiplier × ATR for distance

Stage 8: Post-Execution

Knowledge Base Logging

knowledge_base.csv:

timestamp,symbol,type,entry_price,stop_loss,take_profit,close_price,profit,strategy_id

knowledge_base_events.csv:

timestamp,strategy,retcode,deal,order

Features Export

features.csv (long format):

timestamp,symbol,timeframe,strategy,direction,confidence,...,indicator_1,indicator_2,...
  • 50+ features per trade
  • Automatic ~100MB rotation (rename to *.bak); no compression is performed in MQL5

Telemetry & Logging

Telemetry Events:

  • gate_decision: Pass/block with reason, latency
  • trade_execution: Order placement, retcode
  • policy_load, insights_rebuild, threshold_adjust
  • risk4_*: Risk gate events (spread, margin, drawdown)
  • pm_*: PositionManager events (correlation sizing, scale-ins)
  • p5_*: Selector events (refresh, rescore, tune)

UnifiedTradeLogger: Daily JSON logs with full lifecycle

{
  "timestamp": "2025-01-15T10:30:00Z",
  "strategy": "ADXStrategy",
  "symbol": "EURUSD",
  "timeframe": 60,
  "direction": 1,
  "entry": 1.0850,
  "sl": 1.0800,
  "tp": 1.0950,
  "lots": 0.10,
  "gates": ["signal_rinse:pass", "market_soap:pass", ...],
  "policy": {"confidence": 0.75, "sl_mult": 1.0, "tp_mult": 1.2},
  "outcome": "closed",
  "profit": 15.50,
  "r_multiple": 1.2
}

Learning System Update

LearningBridge: Records CSignalDecision with:

  • All 8 gate decisions (pass/fail, reason, latency)
  • Final outcome (executed, blocked, reason)
  • Trade result (if executed): profit, R-multiple, duration

GateLearningSystem: Updates gate thresholds

  • Hybrid learning: immediate (5% rate) + batch (every 20 trades)
  • Target success rate per gate configurable
  • Adjusts thresholds to optimize pass rate vs trade quality

Live Trading Flow (LiveEA)

Differences from PaperEA_v2 (as wired today):

  1. Insights auto-reload loop: LiveEA requests rebuild by creating DualEA\insights.reload and waits for DualEA\insights.ready.
  2. Per-slice policy scaling: LiveEA parses per-slice policy entries and applies SL/TP/trailing scaling.
  3. Risk gates: LiveEA includes spread/session/margin/consecutive-loss gating in the EA itself.

Safety Features:

  • Shadow mode possible (log decisions, don't execute)
  • Policy fallback with demo-only restriction
  • Multiple circuit breakers
  • One-execution-per-slice-per-minute de-dup (Phase 6 - planned)

Error Handling

Broker Errors

Retcode Handling:

  • TRADE_RETCODE_DONE: Success → proceed to post-execution
  • TRADE_RETCODE_REQUOTE: Retry with new price (max 3 attempts)
  • TRADE_RETCODE_REJECT: Log + telemetry, don't retry
  • TRADE_RETCODE_NO_MONEY: Trip circuit breaker, halt trading
  • Other errors: Log, telemetry, optional cooldown

File I/O Errors

KB Write Failures:

  • Retry with exponential backoff (3 attempts)
  • Create directory if missing
  • Log to Journal on persistent failure
  • Continue execution (non-fatal)

Policy/Insights Load Failures:

  • Use last-known-good cached version
  • Log warning + telemetry event
  • Fallback to neutral scaling if allowed
  • Reload attempt on next timer cycle

Learning System Failures

  • Non-fatal: Log errors, continue trading
  • Disable learning if consecutive failures > 5
  • Telemetry event for monitoring

Performance Characteristics

Latency Targets

  • Signal → Gate entry: <5ms
  • 8-stage gates: <50ms total (target <10ms per gate)
  • Policy/insights lookup: <2ms (in-memory cache)
  • KB write: <10ms (non-blocking preferred)
  • Telemetry emit: <1ms
  • Total pipeline: <100ms (OnTick → OrderSend)

Memory Footprint

  • Strategies (21): ~500KB
  • Gate Manager: ~100KB
  • Policy cache: ~50KB
  • Insights cache: ~200KB
  • Telemetry buffer: ~500KB
  • Total: ~1.5MB per EA instance

File I/O

  • KB writes: Append-only, buffered every 10 trades
  • Features export: Buffered every 50 trades
  • Telemetry flush: Every 5 minutes or 1000 events
  • Policy/insights reload: On timer (every 60 minutes) or .reload trigger

See Also: