Princeec13/mql5
0
0
Çatalla 1
Kod Konular Değişiklik istekleri 1 Projects Sürüm Paketler Wiki Aktivite İşlemler
mql5/Experts/Advisors/DualEA/docs/README.md
Princeec13 62037ac5b0 new files added
2026-02-24 12:47:37 -05:00

53 KiB
Ham Kalıcı Bağlantı Suçlama Geçmiş

DualEA — Production Multi-Strategy Trading System

Paper EA + Live EA with ML Policy Engine

System Overview

DualEA is a production-ready, multi-strategy Expert Advisor ecosystem for MetaTrader 5 that combines:

  1. PaperEA_v2 (2793 lines) - Executes real MT5 orders on demo accounts with 8-stage gate filtering, 23 signal generators, ML integration
  2. LiveEA (1702 lines) - Real trading implementation with insights/policy gating from PaperEA
  3. ML Pipeline - Python-based TensorFlow/Keras training with LSTM models and policy export
  4. Knowledge Base - Shared CSV/JSON data store in Common Files for cross-system communication

Compilation Status: ✅ 0 errors, 0 warnings (MQL5 Build 5572+ compatible)

Core Architecture

MQL5/
├── Experts/
│   └── Advisors/
│       └── DualEA/
│           ├── PaperEA/                # PaperEA_v2 System (Demo Execution)
│           │   ├── PaperEA_v2.mq5     # Enhanced Paper EA with 8-stage gates
│           │   └── PaperEA_backtest.ini
│           ├── LiveEA/                 # Live Trading System  
│           │   ├── LiveEA.mq5         # Live EA with advanced gating
│           │   ├── LiveEA_StrategyBridge.mqh
│           │   ├── Telemetry.mqh
│           │   └── build_liveea.bat
│           ├── Include/                # Core System Components
│           │   ├── IStrategy.mqh           # Strategy interface
│           │   ├── StrategySelector.mqh    # Strategy selection & scoring
│           │   ├── TradeManager.mqh        # Trade execution management
│           │   ├── PositionManager.mqh     # Position management
│           │   ├── KnowledgeBase.mqh       # Data persistence layer
│           │   ├── GateManager.mqh         # 8-stage gate system
│           │   ├── LearningBridge.mqh      # Learning data bridge
│           │   ├── Telemetry.mqh           # Telemetry system
│           │   ├── TelemetryStandard.mqh   # Standard telemetry
│           │   ├── InsightsLoader.mqh      # Insights JSON loader
│           │   ├── SessionManager.mqh      # Session management
│           │   ├── CorrelationManager.mqh  # Correlation analysis
│           │   ├── VolatilitySizer.mqh     # Volatility-based sizing
│           │   ├── Strategies/             # Strategy Implementations
│           │   │   ├── Registry.mqh        # Strategy registry
│           │   │   ├── ADXStrategy.mqh
│           │   │   ├── BollAveragesStrategy.mqh
│           │   │   ├── MeanReversionBBStrategy.mqh
│           │   │   ├── SuperTrendADXKamaStrategy.mqh
│           │   │   ├── RSI2BBReversionStrategy.mqh
│           │   │   ├── DonchianATRBreakoutStrategy.mqh
│           │   │   └── ... (17 more strategies)
│           │   └── Indicators/             # Technical Indicators
│           │       ├── ATR.mqh
│           │       ├── Donchian.mqh
│           │       ├── KAMA.mqh
│           │       └── ... (7 more indicators)
│           ├── ML/                      # Machine Learning Pipeline
│           │   ├── train.py            # Model training script
│           │   ├── policy_export.py    # Policy generation
│           │   ├── dataset.py          # Data loading utilities
│           │   ├── features.py         # Feature engineering
│           │   ├── model.py            # Model definitions
│           │   ├── policy.py           # Policy utilities
│           │   ├── artifacts/          # Model artifacts
│           │   │   ├── tf_model.keras  # Trained model
│           │   │   ├── scaler.pkl      # Feature scaler
│           │   │   └── features.json   # Feature definitions
│           │   └── requirements.txt    # Python dependencies
│           ├── Scripts/                # Utility Scripts
│           │   ├── InsightsRebuild.mq5 # Rebuild insights from data
│           │   └── PolicyDump.mq5      # Policy debugging
│           ├── docs/                   # Documentation
│           │   ├── README.md           # This file
│           │   ├── CORE_IMPLEMENTATION_PLAN.md
│           │   ├── DualEA_Handbook.md
│           │   └── ... (11 more docs)
│           ├── config/                 # Configuration
│           │   └── learning_config.json
│           ├── logs/                   # Log files
│           ├── results/                # Test results
│           └── build_*.bat             # Build scripts

Common Files Structure (Shared Data)

Terminal/Common/Files/DualEA/
├── features.csv           # Feature data from strategies (with rotation)
├── knowledge_base.csv     # Trade execution records
├── insights.json          # Performance analytics per strategy/symbol/timeframe
├── policy.json           # ML-generated trading policy with confidence thresholds
├── insights.reload       # Trigger file for insights reload (LiveEA uses this to request rebuilds)
└── policy.reload         # Trigger file for policy reload (PaperEA_v2 uses this; LiveEA requires restart)

Current Implementation Status

✅ Fully Implemented & Working

PaperEA v2 System (2793 lines)

  • 8-stage gate system with unified configuration (ConfigManager, EventBus, SystemMonitor)
  • 23 indicator-based signal generators (e.g., ADXStrategy, RSIStrategy, MACDStrategy) with asset-class registry
  • Real MT5 order execution on demo accounts via CTradeManager::ExecuteOrder()
  • Advanced optimization: AdaptiveSignalOptimizer, PolicyUpdater, PositionReviewer, GateLearningSystem
  • Comprehensive input parameters (180+ settings for fine-tuning)
  • Circuit breakers, news filtering, regime detection, session management
  • Knowledge Base with CFeaturesKB (features.csv) and CKnowledgeBase (knowledge_base.csv)
  • Real-time telemetry with TelemetryStandard wrapper
  • UnifiedTradeLogger for JSON lifecycle tracking
  • TradeManager with market/pending orders, ATR/fixed trailing stops

LiveEA System (1702 lines)

  • Complete insights gating from insights.json (strategy/symbol/TF performance metrics)
  • ML policy gating from policy.json with confidence thresholds and SL/TP/trail scaling
  • Risk gating: spread, session, daily loss, drawdown, margin, consecutive losses
  • News blackout filtering with CSV-based event calendar
  • Exploration mode for strategies with insufficient historical data
  • Position/Session/Correlation/Volatility managers fully integrated
  • InsightsLoader for DRY insights parsing
  • Strategy bridge for live execution

ML Pipeline (Python)

  • train.py: TensorFlow/Keras classifier + LSTM with TimeSeriesSplit validation
  • policy_export.py: Converts trained models to policy.json with slice-based probabilities
  • features.py: Yahoo Finance enrichment, technical indicators, scaling
  • dataset.py: CSV loading with label generation from R-multiples
  • File rotation and compression for large artifacts
  • artifacts/: Saved models (.keras), scalers (.pkl), metadata (.json)

Supporting Systems

  • GateManager: 8 gates with learning-based threshold adjustment
  • LearningBridge: Bidirectional data flow between Paper/Live, CSignalDecision records
  • StrategySelector: Performance-based strategy scoring with recency weighting
  • InsightsRebuild.mq5: One-click insights.json generation from features.csv
  • Test suite: 8 test files for integration, gates, managers, system health

🔄 Implemented with Minor Gaps

  • Insights auto-reload: LiveEA watches for insights.reload request and polls for insights.ready to auto-reload. PaperEA's CheckInsightsReload() exists but only deletes the file (rebuild logic commented out).
  • LSTM sequence training: Code exists but needs min sequence validation
  • Multi-timeframe confirmation: P5_MTFConfirmEnable parameter exists, needs gate implementation

🔴 Not Implemented (Future Enhancements)

  • Real-time Insights Auto-Rebuild: Automatic background InsightsRebuild trigger when features.csv is stale
  • LiveEA Policy Hot-Reload: Watch policy.reload to refresh ML policy without EA restart (PaperEA already supports this)
  • Multi-timeframe MTF Gate: P5_MTFConfirmEnable parameter exists but gate logic needs implementation
  • Advanced Correlation Pruning: More sophisticated portfolio correlation algorithms beyond Pearson
  • Distributed Training: Multi-machine ML training for large datasets

<EFBFBD>️ P0-P5 Advanced Hardening & Intelligence (NEW)

The EA now includes a comprehensive hardening and intelligence upgrade with P0 through P5 priority levels:

P0 (Production Risk) - CRITICAL

  • Nuclear Risk Engine: VaR 95%, CVaR, Kelly Criterion, correlation matrix monitoring
  • SQLite Knowledge Base: Parallel CSV migration with 30-day transition period
  • Shadow Logging: Atomic CSV writes with checksums and file locking

P1 (Architectural Hardening)

  • Feature Cache: Pre-warmed indicator handles (ADX, ATR, RSI) to eliminate redundant calculations
  • System Monitor: Real-time CPU and memory tracking with automatic circuit breakers
  • Volatility Exits: Trailing stop system with volatility-adjusted sizing

P2 (Intelligence Enhancements)

  • Concept Drift Detector: Welford's Algorithm-based gate threshold auto-adjustment
  • Auto-Promotion System: Performance-based strategy promotion from PaperEA to LiveEA
  • Shadow Trading Bridge: Parallel demo execution to detect slippage
  • Multi-Timeframe Confirmation Gate: MTF entry validation to reduce false signals
  • Correlation-Aware Pruner: Prevents strategy over-concentration (0.70 max correlation)

P3 (Performance Optimizations)

  • CPU Budgeting: 10ms fail-fast with risk gate protection (G1/G4/G7/G8 never skipped)
  • Indicator Cache: Pre-warmed handles for all technical indicators
  • Lock-Free Ring Buffers: 8192-element capacity for zero-contention logging
  • Batch Telemetry Writes: Batched I/O to reduce disk pressure
  • Timer Strategy Scanner: Centralized low-tick strategy scanning

P4 (Strategic Evolution)

  • gRPC ML Bridge: Real-time ML inference with latency monitoring
  • Adversarial Fuzzer: Automated jailbreak testing for robustness validation
  • Nuclear Risk Engine (comprehensive): VaR, Kelly, correlation-based circuit breakers
  • Feature Drift Detector: Real-time distribution shift detection

Integration Architecture

// P0-P5 systems are managed by CDualEAController singleton
// Integrated via PaperEA_v2_P0P5_Integration.mqh

// OnInit:
InitializeP0P5Systems();  // Initializes all P0-P5 subsystems

// OnTick:
OnTickP0P5();  // Processes drift detection, CPU budgeting, risk checks

// OnTimer:
OnTimerP0P5();  // Periodic maintenance (telemetry flush, shadow trades)

// OnDeinit:
ShutdownP0P5Systems();  // Clean shutdown of all P0-P5 components

Key Design Decisions:

  • Risk gates (G1, G4, G7, G8) are never skipped under CPU pressure
  • Multi-symbol management remains chart-per-symbol with Redis coordination
  • SQLite runs parallel to CSV for 30-day migration period
  • All P0-P5 features have enable/disable input parameters for gradual rollout

<EFBFBD>🔧 Unified System Architecture (NEW)

The DualEA system features a Unified Architecture that eliminates redundancy by consolidating insights management and gating pipeline into three core components:

Core Components:

  • ConfigManager: Centralized configuration for gates, insights, and system settings
  • EventBus: Cross-component communication with priority-based event logging
  • SystemMonitor: Real-time health monitoring and performance metrics

Key Benefits:

  • Single source of truth for configuration and monitoring
  • Event-driven architecture with unified logging
  • Real-time performance tracking and health scoring
  • Backward compatibility with legacy mode support

Quick Start:

// Initialize unified system (enabled by default)
CGateManager* gateManager = new CGateManager(symbol, timeframe, learning, true);

// Access unified components for advanced configuration
CConfigManager* config = CConfigManager::GetInstance();
config->SetVerboseLogging(true);
config->SetNoConstraintsMode(false);

See UnifiedSystemGuide.md for detailed integration documentation.

Key Features

8-Stage Gate System (PaperEA v2)

  1. Signal Rinse: Basic signal validation and confidence filtering
  2. Market Soap: Market context analysis (volatility, correlation, regime)
  3. Strategy Scrub: Strategy-specific validation and parameter adjustment
  4. Risk Wash: Risk-based position sizing and validation
  5. Performance Wax: Backtest-based performance validation
  6. ML Polish: Machine learning confidence scoring
  7. Live Clean: Live market condition validation
  8. Final Verify: Final trade validation before execution

Machine Learning Pipeline

  • Feature Engineering: 50+ technical indicators and market features
  • Model Training: TensorFlow/Keras with LSTM and Dense model support
  • Policy Generation: Automated policy.json creation with confidence thresholds
  • Yahoo Finance Integration: Market data enrichment for training
  • Time Series Validation: Proper temporal splits for backtesting

Strategy Library (23 Signal Generators in PaperEA, 21 IStrategies in LiveEA)

PaperEA_v2 uses 23 indicator-based signal generators via StrategySignalGenerators.mqh:

  • ADX, RSI, MACD, Stochastic, CCI, Momentum, Williams %R, DeMarker, Force Index
  • Bears Power, Bulls Power, RVI, OsMA, TriX, Accelerator Oscillator, Awesome Oscillator
  • Alligator, Gator, Ichimoku, Fractals, ZigZag, Ultimate Oscillator, Moving Average

LiveEA includes 21 IStrategy implementations via LiveEA_StrategyBridge.mqh (available for external orchestration):

  • ADXStrategy, SuperTrendADXKamaStrategy, DonchianATRBreakoutStrategy, ForexTrendStrategy, AlligatorStrategy
  • BollAveragesStrategy, MeanReversionBBStrategy, RSI2BBReversionStrategy, VWAPReversionStrategy
  • AwesomeOscillatorStrategy, AcceleratorOscillatorStrategy, BearsPowerStrategy, BullsPowerStrategy, KeltnerMomentumStrategy, AroonStrategy
  • GoldVolatilityStrategy, IndicesEnergiesStrategy, OpeningRangeBreakoutStrategy
  • MultiIndicatorStrategy, EMAPullbackStrategy, StochasticOscillatorStrategy

Data Management

  • Knowledge Base: Centralized trade logging with CSV rotation
  • Features Export: Long-format feature data for ML training
  • Insights Generation: Performance analytics per strategy/symbol/timeframe
  • Common Files: Cross-EA data sharing via MT5 Common Files directory

System Configuration and Monitoring

Gate Configuration Parameters

The unified system supports comprehensive gate configuration through ConfigManager:

// News filtering
UseNewsFilter, NewsBufferBeforeMin, NewsBufferAfterMin, NewsImpactMin

// Trading windows  
UsePromotionGate, PromoStartHour, PromoEndHour

// Market regime detection
UseRegimeGate, RegimeATRPeriod, RegimeMinATRPct, RegimeMaxATRPct

// Circuit breakers and insights
CircuitCooldownSec, NoConstraintsMode
InsightsAutoReload, InsightsLiveFreshMinutes, InsightsStaleHours

Execution Pipeline

  1. Early Phase: Market condition validation and news filtering
  2. 8-Stage Gate Processing: Signal refinement through unified gate system
  3. Risk Management: Position sizing and portfolio risk assessment
  4. Execution: Trade placement with full audit trail and telemetry

Observability Features

  • Event-Driven Logging: All system events flow through EventBus with priority levels
  • Real-time Metrics: Gate success rates, processing times, and system health scoring
  • Performance Tracking: Automated threshold optimization based on historical success rates
  • Cross-EA Compatibility: Identical configuration and behavior between PaperEA and LiveEA

System Capabilities

PaperEA_v2 Core Features

  • Real MT5 Execution: Places real orders on demo accounts via CTradeManager::ExecuteOrder() (not simulation)

  • CTradeManager: Market/pending orders, trailing stops (fixed-points & ATR)

  • 23 Signal Generators: Indicator-based registry via StrategySignalGenerators.mqh

  • 8-Stage Gates: ConfigManager/EventBus/SystemMonitor unified architecture

  • Optimization: AdaptiveSignalOptimizer, PolicyUpdater, PositionReviewer, GateLearningSystem

  • Data Export: features.csv (CFeaturesKB), knowledge_base.csv (CKnowledgeBase), JSON logs (UnifiedTradeLogger)

  • Inputs (PaperEA):

    • Position and risk: LotSize, MagicNumber, StopLossPips, TakeProfitPips, MaxOpenPositions (0=unlimited)
    • Trailing defaults (used if a strategy doesn't set them): TrailEnabled, TrailType=0(fixed), TrailActivationPoints, TrailDistancePoints, TrailStepPoints
    • Mode toggles: NoConstraintsMode (bool, default true) — bypasses many constraints for maximum data collection; critical safety checks (circuit breaker, memory) remain enforced
    • Logging: KBDebugInit (writes INIT line), DebugTrailing (reserved)

  • Modules:

    • Position Manager (see Include/PositionManager.mqh) — available behind UsePositionManager. Defaults: PaperEA=true, LiveEA=false. Current integration initializes the manager and supports caps/correlation hooks; advanced exits/brackets will be integrated next.

MQL5 Compatibility Notes

All code has been verified MQL5-compatible with the following syntax adjustments:

C++ Syntax MQL5 Syntax Files Updated
-> operator . operator PaperEA_v2.mq5, AdaptiveSignalOptimizer.mqh, CInsightGateBridge.mqh
& references Direct array access CInsightGateBridge.mqh, CInsightsRealtime.mqh
IsTradeAllowed() Custom implementation PaperEA_v2.mq5 (lines 4663-4694)
Non-existent methods Commented out PaperEA_v2.mq5 (RecordSanitizationEvent, PerformBatchLearning)
Implicit type casts Explicit (double) casts CSQLiteKnowledgeBase.mqh, PaperEA_v2.mq5

See MQL5_Migration_Notes.md for detailed migration documentation.

Critical Runtime Fixes (February 2026)

Recent tester log analysis identified and resolved critical runtime issues:

Issue Root Cause Fix Location
V0.00 Volume Signal field mismatch: generators set signal.sl, gates read signal.stop_loss (0.0) EnhancedEfficientGateSystem.mqh lines 1006-1071
3+ Second Delays ShadowLogger reading 488MB file into memory for every write CShadowLogger.mqh - direct append mode
Signal Optimization Failed Invalid SL/TP (0.0) caused position sizing to return 0 volume Signal field fallback logic added

Key Fix - Signal Field Alignment:

// Fixed: Check both field names with fallback
decision.final_sl = (signal.stop_loss > 0) ? signal.stop_loss : signal.sl;
decision.final_tp = (signal.take_profit > 0) ? signal.take_profit : signal.tp;
decision.final_volume = (signal.volume > 0) ? signal.volume : 0.1;

Key Fix - ShadowLogger Efficiency:

// BEFORE: Read entire file + append + write back (3000ms+)
// AFTER: Direct append with FILE_WRITE|SEEK_END (<10ms)
// Added: Memory buffer fallback, progressive backoff, skip locking in tester

Latest Fixes (February 15, 2026)

Issue Root Cause Fix Location
No Sell Orders Overly restrictive MA crossover conditions PaperEA_v2.mq5 lines 3467-3508
Position Count 0 CPositionManager not synced with MT5 positions PaperEA_v2.mq5 lines 1791-1797
Memory Leaks Double deletion of MasterController objects PaperEA_v2.mq5 lines 1830-1854
Wrong Position Limit Counted all MT5 positions instead of EA's PaperEA_v2.mq5 lines 3085-3106
Insight Bridge Stats 0 Bridge never queried in trading flow PaperEA_v2.mq5 lines 4239-4250

Key Fix - Sell Signal Generation:

// OLD (broken):
if(fast_ma > slow_ma && fast_ma < current_price)      // Buy - worked
else if(fast_ma < slow_ma && fast_ma > current_price) // Sell - rarely triggered

// NEW (fixed):
if(fast_ma > slow_ma)      // Buy when fast > slow (uptrend)
else if(fast_ma < slow_ma) // Sell when fast < slow (downtrend)

Key Fix - Position Sync:

if(CheckPointer(g_position_manager) != POINTER_INVALID)
{
   g_position_manager.SyncPositions(MagicNumber);
   LOG(StringFormat("[PositionManager] Synced with %d positions for magic=%d", 
           g_position_manager.GetPositionCount(), MagicNumber));
}

See RECENT_MODIFICATIONS.md for detailed ongoing changes.

LiveEA Inputs (New)

  • Policy fallback and gating:
    • UsePolicyGating, DefaultPolicyFallback, FallbackWhenNoPolicy, FallbackDemoOnly.
    • Log reasons: fallback_no_policy (no slices loaded), fallback_policy_miss (slice not found). Both bypass exploration caps and use neutral scaling; demo-only unless FallbackDemoOnly=false.
  • Insights auto-reload controls:
    • InsightsAutoReload, InsightsLiveFreshMinutes, InsightsReadyPollSec (inputs present; wiring for automated reload is being expanded in LiveEA).
  • Position management:
    • UsePositionManager (default false) and PMMaxOpenPositions for portfolio caps; enables integration with Include/PositionManager.mqh where available.

Policy scaling keys

  • LiveEA expects trailing scaling under the key trail_scale.
  • PaperEA’s PolicyEngine and docs/PolicySchema.md use trail_atr_mult for ATR-based trailing. Exporters may emit both keys for compatibility until the schema is unified.

Where Files Are Written

We target the MT5 Common Files area so Strategy Tester, Demo/Paper, and Live share the same outputs.

  • Typical path: C:\Users\<you>\AppData\Roaming\MetaQuotes\Terminal\Common\Files\DualEA\
  • Alternative (some installs): C:\ProgramData\MetaQuotes\Terminal\Common\Files\DualEA\

Files produced:

  • knowledge_base.csv (header created at init)
  • knowledge_base_events.csv
  • features.csv (per-trade feature rows for ML)
  • explore_counts.csv (weekly exploration usage)
  • explore_counts_day.csv (daily exploration usage)
  • insights.json (aggregated performance by strategy/symbol/timeframe)
  • policy.json (exported by trainer) and policy.reload (touch file to trigger hot-reload)

Documentation

Knowledge Base Schemas

  • knowledge_base.csv
    • Columns: timestamp,symbol,type,entry_price,stop_loss,take_profit,close_price,profit,strategy_id
  • knowledge_base_events.csv
    • Columns: timestamp,strategy,retcode,deal,order

How To Build

  • Use the provided batch scripts:
    • PaperEA: open a terminal in MQL5/Experts/Advisors/DualEA/PaperEA/ and run build_paperea.bat
    • LiveEA: open a terminal in MQL5/Experts/Advisors/DualEA/LiveEA/ and run build_liveea.bat
  • Or compile from MetaEditor (PaperEA.mq5 / LiveEA.mq5).

How To Run (Strategy Tester)

  1. Open Strategy Tester and select PaperEA.
  2. On the Inputs tab, click Reset to load the latest defaults.
  3. Adjust LotSize, SL/TP, and trailing inputs as needed.
  4. Run a backtest; outputs will be in Common\Files\DualEA\ and visible in the Journal.

Trainer Workflow (Windows)

  • Requirements: Python 3, internet access (for pip + Yahoo enrichment).
  • Steps:
    1. Open a terminal in MQL5/Experts/DualEA/ML/.
    2. Run: run_train_and_export.bat
  • What it does:
    • Creates .venv/, installs ML/requirements.txt.
    • Runs train.py to produce ML/artifacts/tf_model.keras, scaler.pkl, features.json.
    • Runs policy_export.py --min_conf 0.45 to write Common\Files\DualEA\policy.json with per-slice p_win and scaling fields: sl_scale, tp_scale, trail_scale (LiveEA) or trail_atr_mult (PaperEA).
    • Touches Common\Files\DualEA\policy.reload.
  • EA behavior:
    • PaperEA calls CheckPolicyReload() in both OnTimer() and OnTick(); when policy.reload exists, it reloads policy.json and logs Policy gating: min_conf=..., slices=N.
    • On trade execution, if policy is loaded, the EA applies per-slice scaling before sending the order.

Rebuilding Insights (without rerunning trades)

  • Use the provided script to recompute insights.json from the current features.csv:
    1. In MT5 Navigator → Scripts → run InsightsRebuild.
    2. Check Common\Files\DualEA\insights.json and Journal for status.

Troubleshooting

  • “No CSVs in MQL5/Files”: Expected. Tester and Live now write to Common\Files\DualEA using FILE_COMMON.
  • “Error 5004 opening file”: First‑run read attempts can fail if the file doesn’t exist; the constructor now creates the file and header without logging an error.
  • “Inputs not showing in Tester”: Click Inputs → Reset; MT5 caches previous inputs.

PaperEA

  • Purpose: primary data-collection EA for backtests and demo/paper runs. Logs features, events, and trades to the shared Knowledge Base for ML and insights.
  • Execution model:
    • OnInit/OnTick drive the scan→gate→execute path; logs to KB on every decision and execution.
    • Policy reload via DualEA/policy.reload checked in CheckPolicyReload(); telemetry flush after scans.
    • Timer-based scanning per Phase 6 design may be enabled as parity with LiveEA (see Phase 6) where implemented.
  • Gating and exploration caps:
    • NoConstraintsMode (default true) bypasses many gates/caps for unrestricted exploration; critical safety checks (circuit breaker, memory guard, news filter) remain enforced.
    • Exploration caps persist in Common Files: DualEA/explore_counts_day.csv, DualEA/explore_counts.csv.
    • No‑slice‑only bypass: exploration only bypasses when a slice truly doesn’t exist; existing under‑threshold slices remain gated.
    • Reset by deleting the above CSVs in Common\Files\DualEA.
  • Policy integration and fallback:
    • Inputs: UsePolicyGating, DefaultPolicyFallback, FallbackWhenNoPolicy, FallbackDemoOnly.
    • When policy is loaded and slice exists, ApplyPolicyGating() checks confidence (per-slice scaling currently minimal in PaperEA; LiveEA has full scaling).
    • On fallback (no policy or slice missing), trades use neutral scaling, do not consume exploration quotas, and bypass caps (demo‑only by default).
  • Persistence paths (MT5 Common Files):
    • knowledge_base.csv, knowledge_base_events.csv, features.csv, insights.json, exploration CSVs under Common\Files\DualEA.
  • CI hooks (insights freshness + coverage):
    • Use kb_check.bat to rebuild and validate insights.json headlessly; fails on stale/empty slices and on missing required coverage.
    • Example: 
      kb_check.bat --build-insights --attempt-run-script --run-validate --fail-on-stale --fail-on-empty-slices --require-symbols=USDCNH --require-timeframes=60 --require-strategies=BollAverages,DonchianATRBreakout,RSI2BBReversion --fail-on-missing-required --show-report

Roadmap (Phased)

This roadmap aligns with the implementation plan.

For detailed Phase 3 guidance (docs, hygiene, scoring, roadmap), see:

  • docs/Phase3.md

Action Framework — 3 Phases × 3 Cycles (Merged)

  • Source: docs/DualEA_Action_Framework.md (full details and templates)

  • Phase 1 — Codebase & Requirements Deep Dive

    • Cycle 1: Structural Audit (jailbreak permitted)
      • Inventory gating paths (strategy → policy → insights → execution)
      • Trace file I/O and timers: CheckPolicyReload(), CheckInsightsReload(), staleness checks
      • Identify insertion points: circuit breakers, session/news filters, PositionManager, correlation, vol sizing, regime
      • Deliverables: audit map + risk log
    • Cycle 2: Requirements Mapping
      • Spec one‑pagers per feature with inputs, data flow, telemetry, failure/rollback
    • Cycle 3: Risk & Gaps Closure
      • Threat model: file races, stale artifacts, partial reloads; decide LiveEA fail‑safe defaults
      • Deliverables: mitigations + rollback levers; updated lifecycle gates

  • Phase 2 — Design, Build, Validate

    • Cycle 1: Task Decomposition
      • Backlog with effort/risk/owner/sequence for FR‑01..FR‑10
    • Cycle 2: Code + Peer Review
      • Definition of Ready: inputs defined, telemetry standardized, rollback behavior set
      • Guidelines: feature flags (Use*), shared includes under Include/
    • Cycle 3: QA + Fuzzing
      • Unit‑like stubs, deterministic sims, log assertions; fuzz missing files/reload storms/high‑tick

  • Phase 3 — Polish, Lockdown, Final Scoring

    • Cycle 1: Docs & architecture alignment (README + lifecycle + operator runbooks)
    • Cycle 2: Code hygiene & consistency (inputs/enums/log tags; dead‑code purge)
    • Cycle 3: Final scoring (correctness/safety/observability/perf) + next roadmap

Action Backlog (Pre‑Implementation Specs)

  • FR‑01 Circuit Breakers — Inputs: UseCircuitBreakers, DailyLossLimit, DailyDrawdownLimit, CooldownMinutes; block entries, allow exits; telemetry cb:*; DoD: day rollover sims
  • FR‑02 Session/Window Filter — Inputs: UseSessionFilter, SessionTZ, SessionWindows; pre‑gate block; DoD: DST/tz edges
  • FR‑03 News Filter — Inputs: UseNewsFilter, ImpactLevelMin, PreLockoutMin, PostLockoutMin; offline‑safe fallback; DoD: replay test
  • FR‑04 Position Manager (v1) — Inputs: UsePositionManager, ExitProfile, ClusterMinPts, PerSymbolMax; centralized exits/scaling hooks; DoD: backtest diffs. (Core module implemented in Include/PositionManager.mqh; integration behind flag pending.)
  • FR‑05 Correlation Exposure Caps — Inputs: UseCorrelationCaps, CorrLookbackDays, MaxGroupExposure; cap grouped exposure; DoD: deterministic calc
  • FR‑06 Volatility Position Sizing — Inputs: UseVolSizing, ATRPeriod, RiskPerTrade; ATR‑adaptive lots; DoD: regime consistency, caps respected
  • FR‑07 Regime Detector Stub — Inputs: UseRegime, RegimeMethod; tag regime; optional gating modifier; DoD: stable tags
  • FR‑08 Promotion Gate & Evaluator — Inputs: min trades/hit‑rate/expectancy/DD/Sharpe; artifact: Scripts/DualEA/AssessPromotion.mq5; DoD: reproducible
  • FR‑09 Telemetry Standardization & Ops Views — Event schema/fields/severity; rotate/size bounds; optional ops notebooks
  • FR‑10 Performance — Event batching, log throttling, timer cadence review; DoD: CPU/mem profile under stress

Control Gates (Before Coding)

  • Gate A: Specs complete (inputs, flow, telemetry, rollback) for FR‑01..FR‑10
  • Gate B: Test plans defined; simulators/log assertions ready
  • Gate C: Security + fail‑safe review; LiveEA defaults conservative

Ownership & Sequencing (Draft)

  • Wave 1 (risk first): FR‑01, FR‑02, FR‑08
  • Wave 2 (exposure/positioning): FR‑04, FR‑05, FR‑06
  • Wave 3 (context/perf): FR‑03, FR‑07, FR‑09, FR‑10
  • Note: Features default‑off in LiveEA until validated in PaperEA

Status Dashboard

  • Completed:
    • Phase 1: Data & Insights Foundation (insights builder, headless rebuild, validator, CI flags, Common Files outputs)
    • Phase 2: ML/LSTM Pipeline (trainer + policy export + batch runner; PaperEA policy hot‑reload integrated)
  • Phase 2a: PaperEA Execution & Telemetry (baseline complete; timer-scan parity tracked under Phase 6)
  • Phase 3 (core): LiveEA core loop with policy gating/fallbacks and scaling
    • LiveEA/LiveEA.mq5: policy loaded from file on init; ApplyPolicyScaling() applies SL/TP/trailing (trailing via trail_scale), fallback gating (fallback_no_policy, fallback_policy_miss, demo‑only by default), heartbeat with fallback flags
  • In Progress:
    • Phase 3: LiveEA hardening (risk gates, stricter caps, optional shadow‑mode, per‑minute de‑dup)
    • Phase 6: Low‑Latency Minutely Scanner & ML‑Enriched Orchestration
    • Phase 6: Low‑Latency Minutely Scanner & Orchestration (central ScanStrategies, minute de‑dup, parity with OnTick)
    • Phase 2a → P1C1 (ACTIVE): Structural Audit for PaperEA (gating/caps persistence, telemetry, KB writes, insights staleness, CI validators)
    • Phase 4/7/8: Risk/Telemetry/Adversarial hardening — partial primitives present
    • Ops/CI TODOs (remaining):
      • Add kb_check.bat for headless insights rebuild+validate and wire into CI (fail on stale/coverage).
      • Add scripts/RunInsightsValidation.ps1 and ScheduleInsightsValidation.ps1 for scheduled validation.
      • Optional CI trainer job: run ML/run_train_and_export.bat on schedule/data‑change; publish policy.json and touch policy.reload.
      • Operator runbook: KB rebuild/validate flows and artifact paths under Common\\Files\\DualEA.
      • Telemetry: concise summaries for insights rebuild/validation outcomes.
  • Not Implemented:
    • Phase 5, 9, 10, 11

Phase 1: Data & Insights Foundation

  • TODOs (hardening and ops):
    • CI integration: add kb_check.bat and wire Scripts/ValidateInsights.mq5 to CI to fail on stale insights.json, empty/missing slices, and unmet required coverage (symbols/TFs/strategies).
    • Headless rebuild wrapper: provide a batch/PowerShell entry point to run Scripts/InsightsRebuild.mq5 non‑interactively (for CI/scheduled tasks).
    • Schema versioning: version insights.json and add compatibility checks in Include/StrategySelector.mqh; document version in docs/KB-Schemas.md.
    • I/O hardening: ensure FolderCreate("DualEA", FILE_COMMON) is called by all writers; prefer robust readers (FILE_TXT|FILE_ANSI), skip malformed rows, and log/continue.
    • Validator enhancements: expand ValidateInsights to assert non‑empty required metrics per slice and configurable thresholds; surface a concise report.
    • Ops scheduling: optionally add scripts/RunInsightsValidation.ps1 and ScheduleInsightsValidation.ps1 to run validation on a cadence.

Phase 2: ML/LSTM Pipeline

  • TODOs (ops/hardening):
    • CI trainer job: run ML/run_train_and_export.bat on schedule or data‑change; publish policy.json to Common Files and touch policy.reload.
    • Provenance & signing: include model hash, train window, metrics in policy.json; sign/hash for integrity; keep last‑known‑good for rollback.
    • Shadow eval & A/B: support dual policies in PaperEA (shadow mode) prior to promotion to LiveEA.
    • Calibration & thresholds: reliability diagrams, Brier, per‑slice min_conf calibration; document acceptance bands.
    • Monitoring: log SHAP/feature importance drift signals; alert on material drift.
    • Rollback guardrails: auto‑rollback to last‑known‑good on parse/plausibility failure during hot‑reload.

Note (sequencing): core ML training/export and PaperEA policy load are implemented. The structured 3×3 audit for Phase 2 will begin after Phase 3 — Cycle 3.

  • Suggestions:
    • Use walk‑forward splits; track ROC‑AUC, Brier, Expected‑R; calibrate probabilities and thresholds per slice.
    • Export policy.json with provenance (model hash, train window, metrics); sign/hash for integrity.
    • Run A/B policies and shadow evaluation in PaperEA before promotion to LiveEA.
    • Log SHAP/feature importance to monitor drift and potential leakage.

Phase 2a: PaperEA Execution & Telemetry

  • TODOs (remaining):
    • CI wiring: integrate Scripts/ValidateInsights.mq5 into CI and add kb_check.bat to fail on stale insights.json, empty/missing slices, and missing required coverage.
    • Headless operations: add scripts/RunInsightsValidation.ps1 and ScheduleInsightsValidation.ps1 for scheduled rebuild/validation.
    • Ops docs: add a short operator runbook covering rebuild/validate flows and artifact locations in Common\\Files\\DualEA.
    • Explore caps ops: provide a lightweight reset helper and a smoke test that verifies daily/weekly counters persist and increment as expected.
    • Telemetry polish: emit concise summaries for insights rebuild/validation outcomes.
  • Status:
    • Baseline implemented (modular strategies, exploration caps/gating, NoConstraintsMode, policy hot‑reload/fallbacks, Common Files persistence).
    • Timer‑based scanning parity with LiveEA is tracked under Phase 6.

Phase 3: LiveEA & Feedback Loop

  • Implement LiveEA.mq5 with the same strategies but stricter gating (policy + risk limits).
  • LiveEA logs to the same KB; trainer merges Paper+Live data.
  • Feedback loop: Paper adapts based on Live outcomes via retrained policy.
  • TODOs (remaining):
    • LiveEA shadow‑mode: log candidate decisions; execute only top ML‑ranked passing min_conf and risk budget.
    • Tighter gates vs PaperEA: stricter min_conf, spread/news/session caps, lower exploration caps.
    • Enforce risk/circuit‑breakers: max daily loss, session drawdown, stricter spread/news/session caps.
    • One‑execution‑per‑slice‑per‑minute de‑dup; deterministic tie‑breakers.
    • Full timer‑scan parity (Phase 6): centralize ScanStrategies; consistent OnTick/Timer paths.
    • Policy hot‑reload guardrails with rollback to last‑known‑good on parse/plausibility failure.
    • Integrate Include/PositionManager.mqh behind UsePositionManager; route sizing/exits through it.
    • Prep hooks for Phase 6 parity: shared scanner entrypoints and per‑minute dedup state.

    Red‑Team Execution Plan (3×3) — Current focus

  • Phase 1: Codebase & Requirements Deep Dive
    • Cycle 1 Structural Audit for LiveEA
    • Cycle 2: Requirements Mapping to concrete code paths and artifacts; close gaps
    • Cycle 3: Risk & Gaps Closure; redesign fragile modules as needed
  • Phase 2: Design, Build, Validate
    • Cycle 1: Task decomposition for LiveEA hardening; define acceptance criteria
    • Cycle 2: Implement fixes; peer review and adversarial edge‑case tests
    • Cycle 3: QA + fuzzing of IO/races; enforce CI gates
  • Phase 3: Polish, Lockdown, Final Scoring
    • Cycle 1: Docs & architecture alignment (README + docs/Phase3.md)
    • Cycle 2: Code hygiene & consistency; refactors; dead‑code purge
    • Cycle 3: Final scoring + roadmap

Next: Upon completing Phase 3 — Cycle 3 above, begin Phase 2 — Cycle 1 (ML/LSTM Pipeline) as the next 3×3 plan.

Phase 4: Risk & Safety Systems

Phase 4 TODO checklist
  • Run Strategy Tester to trigger each Phase 4 gate and confirm risk4_* telemetry + margin_level reason

Phase 4 Telemetry: risk4_* events and reason strings

  • risk4_spread: spread cap gating.
    • Shadow (NoConstraintsMode=true): LogGatingShadow(strategy, symbol, tf, "risk4_spread", ok, reason, true)
    • Enforced: [RISK4] blocked <strategy> on <symbol>/<tf> reason=<spread_reason> and LogGating(symbol, tf, strategy, "risk4_spread", allow, reason)

Phase 5: Dynamic Strategy/Indicator Selection

  • Status: In progress — merged into LiveEA: cooldown + auto re-enable + dynamic auto‑tuning with telemetry (underperf_auto_disabled, p5_cooldown, p5_auto_reenabled, p5_auto_tune).
    • New (LiveEA): Phase 5 telemetry completed for p5_refresh, p5_rescore, and p5_selector_cfg.
  • TODO (prioritized)
    1. Selector wiring parity in both EAs (OnInit): thresholds, recency overlay, strict thresholds, and per‑symbol/timeframe strategy registration.
      • Acceptance: strategies register per symbol/TF; logs/telemetry confirm thresholds/recency overlays applied.
    2. Correlation‑aware pruning within selector/gating to retain a diversified subset per symbol/timeframe.
      • Emit p5_corr telemetry; cap per symbol/TF.
    3. Multi‑timeframe confirmations in selector/gating with inputs (e.g., EnableMTFConfirmations, MTFConfirmTF, MTFConfirmMinScore) and telemetry (p5_mtf).
    4. Parameter stability tracking in selector; compute stability score and gate via (P5_StabilityGateEnable, P5_StabilityWindowDays, P5_StabilityMaxStdR) with telemetry (p5_stability).
    5. Policy integration for subset selection/weighting from ML policy (beyond neutral scaling); document and gate safely.
    6. Feature/telemetry export parity after exec in both EAs (ATR, spread, TF, trailing params, strategy features).
    7. Telemetry consistency across both EAs: standardize remaining p5_* event keys and shadow logging under NoConstraintsMode (PaperEA parity).
    8. Test plan: backtests (walk‑forward), forward/demo runs, and monitoring to validate selector and gating behavior.

Phase 6: Low‑Latency Minutely Scanner & ML‑Enriched Orchestration

  • Status: Partial — OnTimer heartbeat/reload present; centralized ScanStrategies with minute de‑dup not yet implemented.

  • Guarantee both EAs "look" for trades at least every minute, independent of tick arrival.

    • Add inputs: TimerScanEnabled (bool, default true), TimerScanSeconds (int, default 60).
    • OnInit(): configure EventSetTimer(TimerScanSeconds) when TimerScanEnabled.
    • OnTimer(): run a ScanStrategies() routine that performs the same Refresh → CheckSignal → gating (selector + insights + policy + caps) path as OnTick().
    • Deduplicate within a minute per symbol|timeframe|strategy using a rolling LastScanMinute map to avoid duplicate triggers if ticks also arrive.
    • Preserve existing behavior: always CheckPolicyReload() and Telemetry.Flush() on timer.

  • Performance & reliability hardening

    • Pre‑create and reuse indicator handles for all registered strategies/timeframes to avoid reallocation overhead per scan.
    • Cache policy and insights in memory; throttle file I/O to timer ticks; batch telemetry writes.
    • Optional: EventSetMillisecondTimer(250–1000ms) for ultra‑low‑latency scanning in LiveEA; gate by CPU budget.

  • LiveEA enrichment path

    • Consume trained ML outputs for per‑slice min_conf, expected‑R, and dynamic sizing; apply via ApplyPolicyScaling().
    • Maintain exploration caps and detailed LogGating/LogGatingShadow for full auditability.

  • Test & validation plan

    • Backtests: walk‑forward with OOS across 3–5y per symbol/TF, including realistic spread/slippage/commission; parameter stability and FDR controls.
    • Forward tests: parallel demo/real, M1 granularity logs, latency SLO (<250ms scan→gate→exec where broker allows), missed‑opportunity audits.
    • Monitoring: 60s heartbeat, exploration cap summaries, and per‑slice gating counters.

  • Suggestions:

    • Guard OnTimer() with a re‑entrancy flag to prevent overlapping scans; prefer non‑blocking work.
    • Expose TimerScanSeconds overrides (e.g., 30s) and optional EventSetMillisecondTimer for LiveEA with CPU budget guard.
    • Record scan→gate latency metrics; emit to telemetry for SLO tracking.
    • Pre‑warm indicator handles in OnInit(); refresh/rebuild on symbol/period change; cache policy/insights in memory.
    • Centralize gating in a reusable EvaluateAndMaybeExecute() path used by both OnTick() and timer scans.

Phase 7: Monitoring & Telemetry

  • Emit telemetry CSV/JSON (tick latency, SL moves, trades/sec, error rates).

  • Optional lightweight dashboard (HTML/JS) or notebook to visualize metrics.

  • Suggestions:

    • Define SLOs: scan→gate latency, missed‑opportunity rate, per‑reason gating counts.
    • Alerts for policy/insights read failures and staleness; log rotation and retention policy.
    • Optional dashboard/notebook; per‑slice counters and trend lines for drift/anomaly detection.

Phase 8: Adversarial/Jailbreak Hardening

  • Fuzz file I/O (empty/corrupt/locked policy.json/insights.json/CSV) and assert safe fallbacks, halts, or rollbacks.

  • Timing/race tests: low-tick periods, bar-boundary races, clock skew; verify de-dup and single-exec semantics.

  • Resource pressure: very large CSVs, slow disk; ensure OnTick() remains non-blocking, heavy I/O on timer.

  • Produce Q&A logs, expert rebuttals, and jailbreak traces mapped to files/modules per @/greep.

  • Suggestions:

    • Add SafeMode input to force conservative gating on anomalies; persist last‑known‑good policy.json/insights.json caches.
    • Create a fuzz harness (script/test) that mutates JSON/CSV and asserts expected EA behavior and logs.
    • Store jailbreak trace logs in Common\Files\DualEA\jailbreak\ with timestamps and module references.
    • Automate adversarial tests in CI; fail the pipeline on missing fallbacks or unsafe behavior.

Phase 9: MLOps & Continuous Training

  • Data/versioning (DVC/Git LFS), model registry, and feature/label drift detection with alerts.

  • Scheduled retraining and policy export with provenance (model hash, data window, metrics).

  • Canary deployment of policy.json and automatic rollback on degradation or plausibility failures.

  • Suggestions:

    • Define acceptance criteria (min ROC‑AUC/Brier, uplift vs baseline) before policy promotion to LiveEA.
    • Build reproducible training pipelines (seeded, environment‑pinned) and artifact signing.
    • Add drift detectors on features/labels; trigger retrain or quarantine when thresholds are exceeded.
    • Maintain a data catalog of features.csv schema versions and TTL/retention policies.

Phase 10: Multi‑Symbol/TF Orchestration (Optional)

  • Option A: one EA per symbol/TF (simple, robust). Option B: single orchestrator scanning a configured portfolio.

  • Central registry of symbols/TFs; per-slice de‑dup and CPU budget/load shedding to maintain latency SLOs.

  • Consistent telemetry across all slices; portfolio‑level risk budget enforcement.

  • Suggestions:

    • For orchestrator mode: implement per‑slice schedule, backpressure, and fairness to avoid starving symbols.
    • Throttle scanning and degrade gracefully under CPU load; prioritize LiveEA executions.
    • Guard broker API calls; exponential backoff on transient errors; per‑symbol error budgets.
    • Keep robust chart‑per‑slice mode as default until orchestrator proves reliability.

Phase 11: Performance & Resilience

  • Avoid heavy file I/O in OnTick(); defer to timer/batched writes; use FILE_SHARE_* and rotate logs.

  • Pre‑allocate buffers, reuse indicator handles; cap log verbosity on hot paths.

  • Watchdog for timer and safe‑mode fallbacks on parse errors (use last‑known‑good caches).

  • Suggestions:

    • Implement log rotation (daily/size‑based) and retention; compress archives.
    • Add a timer watchdog that re‑initializes the timer if missed deadlines are detected.
    • Use re‑entrancy guards around critical sections (order placement, persistence writes).
    • Ensure OnDeinit() flushes telemetry and persists counters atomically.

Next Milestones

  • Add features.csv export and insights.json aggregation.
  • Provide Python ML trainer and batch launcher (writes policy.json).
  • Add policy.json loader in PaperEA and gating/scaling application.
  • Skeleton LiveEA.mq5 with circuit breaker checks.

LiveEA: PositionManager — Correlation Sizing + Dynamic Risk Caps

This section documents the LiveEA wiring of Include/PositionManager.mqh for correlation‑aware sizing and dynamic risk caps. All features are behind UsePositionManager and new PM inputs.

Inputs (LiveEA)

  • PositionManager switches
    • UsePositionManager (bool)
    • PM_EnableCorrelationSizing (bool, default true)
    • PM_EnableAdaptiveSizing (bool), PM_MinSizeMult (double), PM_MaxSizeMult (double)
    • PM_ScalingProfile (int: 0=Aggressive, 1=Moderate, 2=Conservative)
    • PM_EnableVolatilityExit (bool), PM_VolatilityExitATRMult (double)
  • Dynamic risk caps (optional)
    • PM_EnableDynamicRiskCaps (bool)
    • PM_MaxDailyDDPct (double), PM_MaxPosRiskPct (double), PM_MaxPortfolioRiskPct (double), PM_RiskDecay (double 0..1)
  • Correlation sizing floors
    • PM_CorrMinMult (double) — minimum fraction of base lots allowed after correlation dampening
    • PM_CorrMinLots (double) — absolute minimum lots floor after dampening

Behavior

  • New entries: after policy scaling (ApplyPolicyScaling), LiveEA applies correlation‑adjusted sizing via CPositionManager.CalculateCorrelationAdjustedVolume(symbol, base_lots).
    • Floors: PM_CorrMinMult * base_lots and PM_CorrMinLots are enforced to avoid over‑dampening.
  • Scale‑ins: when PositionManager proposes vol_scaled, LiveEA applies the same correlation dampening and floors before overriding order.lots.
  • Dynamic risk caps: configured in OnInit() via CPositionManager.SetDynamicRisk(...). Internally caps additional size growth (implementation advisory, subject to future expansion).

Telemetry & Logs

  • Events: pm_corr_sizing, pm_corr_floor, pm_corr_scale_in, pm_corr_floor_scale_in with correlation value and before/after lots.
  • Init log: [PM] Initialized (...) includes adaptive, profile, volatility exit, and dynamic risk params.

Validation

  1. Enable UsePositionManager=true, PM_EnableCorrelationSizing=true.
  2. Open a correlated position (e.g., EURUSD) and then trade a related pair (e.g., GBPUSD). Expect reduced lots and pm_corr_sizing event.
  3. Trigger a scale‑in via PositionManager; expect pm_corr_scale_in and floor events if dampening is severe.
  4. Toggle PM_CorrMinMult / PM_CorrMinLots to observe floors applying (logs show before/after + floor values).

License

Copyright 2025, Windsurf Engineering.

If you truly need low-latency sync later, consider:

  • Named pipes or local TCP with a lightweight Python service
  • A minimal REST loopback server

Note: These add failure modes and deployment friction; file-based I/O is simpler and reliable for MT5. Prefer file-based until a concrete latency SLO requires otherwise.

Appendix A: Red‑Team Artifacts (Q&A, Rebuttals, Jailbreak Traces)

To operationalize the 3×3 red‑team process, store artifacts in MT5 Common Files for both Tester and Live to access.

  • Base path: Common\\Files\\DualEA\\jailbreak\\
  • Run layout:
    • jailbreak\\runs\\YYYYMMDD_HHMMSS/
      • qna.md — Q&A log for the session
      • rebuttals.jsonl — Expert rebuttals (JSONL)
      • traces.jsonl — Jailbreak traces (JSONL)
      • summary.md — Short session summary with outcomes and follow‑ups
    • jailbreak\\latest/ — optional symlink/copy of most recent run for dashboards

A.1 Q&A Log (Markdown)

Minimal, structured template capturing prompts, answers, and references to code.

# Red‑Team Q&A — <YYYY‑MM‑DD HH:MM:SS>

## Q1: <question>
- Context: <module(s) e.g., `Experts/Advisors/DualEA/PaperEA/PaperEA.mq5`>
- Hypothesis: <expected behavior>
- Answer: <observed behavior/decision>
- Evidence: <file refs, line ranges, logs>
- Outcome: pass | fail | needs‑followup

## Q2: ...

A.2 Expert Rebuttals (JSONL)

One JSON per line with explicit links to files/modules and decisions.

{"id":"rb_0001","claim":"Exploration bypass triggers with existing slice","rebuttal":"Bypass is no‑slice‑only; existing under‑threshold slices are gated","evidence_files":["Experts/Advisors/DualEA/PaperEA/PaperEA.mq5"],"decision_paths":["Gate->Insights->ExploreCaps"],"status":"resolved","timestamp":"2025-08-17T00:00:00Z"}

Fields:

  • id: stable id
  • claim: red‑team claim
  • rebuttal: concise counterargument
  • evidence_files: array of repository paths
  • decision_paths: e.g., Scan->Selector->Gate->Execute
  • status: open | resolved | needs‑work
  • timestamp: ISO8601

A.3 Jailbreak Traces (JSONL)

Record adversarial tests, input mutations, expected vs observed behavior, and severity.

{"ts":"2025-08-17T00:00:00Z","module":"PolicyLoader","file":"Experts/Advisors/DualEA/Include/StrategySelector.mqh","symbol":"EURUSD","timeframe":60,"case":"corrupt_policy_json","mutation":"truncated file","expected":"fallback to last‑known‑good policy with log and no crash","observed":"fallback engaged; neutral scaling applied","outcome":"safe","severity":"low","artifacts":["Common/Files/DualEA/policy.json","Common/Files/DualEA/policy.backup.json"],"notes":"Validated timer reload path and guardrails"}

Fields:

  • ts: timestamp
  • module, file: component and path under repo
  • symbol, timeframe: slice context when applicable
  • case, mutation: short names for cataloging
  • expected, observed: behaviors
  • outcome: safe | unsafe | inconclusive
  • severity: low | medium | high
  • artifacts: related files (Common Files paths allowed)
  • notes: free‑form

A.4 Traceability Rules

  • Always reference concrete files/paths (e.g., Experts/Advisors/DualEA/LiveEA/LiveEA.mq5).
  • Map each finding to a decision path: OnTick()->ScanStrategies()->Gating()->TradeManager.Execute().
  • Prefer JSONL for machine‑readable aggregation; keep Markdown summaries for humans.
  • Attach log snippets (Journal) and KB rows when relevant.

A.5 CI Hooks

  • Copy the latest run into jailbreak\\latest/ after each session.
  • Optional validator can assert that for each case seen in traces there is at least one outcome=safe or an open issue.
  • Include red‑team artifacts in reports alongside insights validation.

A.6 Operational Notes

  • PaperEA and LiveEA do not read from jailbreak/; it is a write‑only audit area.
  • Keep PII/broker identifiers out of logs; sanitize before sharing.
  • Large artifacts (CSV/JSON) should be rotated/compressed if size grows beyond operational limits.