mql5/Experts/Advisors/DualEA/docs/Operations.md

Operations Guide (DualEA)

Practical procedures for managing DualEA runtime, knowledge base, policy reloads, and interpreting logs. All paths are under MT5 Common Files.

• Typical path: C:\Users\<you>\AppData\Roaming\MetaQuotes\Terminal\Common\Files\DualEA\
• Alternate: C:\ProgramData\MetaQuotes\Terminal\Common\Files\DualEA\

Controls & Inputs (PaperEA)

• NoConstraintsMode — disable all gating and caps for maximum data collection (bypasses trading hours, selector gating, insights gating, exploration caps, and open-position limit). Default: true.
• UsePolicyGating — enable/disable ML policy gating
• DefaultPolicyFallback — allow neutral trading when slice/policy missing
• FallbackDemoOnly — restrict fallback to demo only (default true)
• FallbackWhenNoPolicy — allow fallback when no policy file is loaded (default true)
• ExploreMaxPerSlice — weekly cap (default 3; 0 = unlimited)
• ExploreMaxPerSlicePerDay — daily cap (default 2; 0 = unlimited)
• MaxOpenPositions — global open-position limit (default 0; 0 = unlimited)
• Verbosity — controls ShouldLog(level) (INFO/DEBUG)

Knowledge Base Files

• Trades: knowledge_base.csv (see docs/KB-Schemas.md)
• Events: knowledge_base_events.csv
• Features: features.csv (long format, includes r_multiple rows)
• Insights: insights.json (built via script or builder)
• Exploration counters: explore_counts.csv, explore_counts_day.csv

System Flow

flowchart LR
  %% PaperEA orchestration
  subgraph Paper
    PEA[PaperEA]
    STR[Strategies - IStrategy]
    SEL[Selector - insights + recency]
    TM[TradeManager - SL/TP/Trailing]
    TEL[Telemetry]
  end

  %% Common Files runtime artifacts
  subgraph KB["Knowledge Base - Common Files DualEA"]
    FEAT[features.csv]
    TRD[knowledge_base.csv]
    EVT[knowledge_base_events.csv]
    TLM[telemetry logs]
    INS[insights.json]
    POL[policy.json]
    EXPW[explore_counts.csv]
    EXPD[explore_counts_day.csv]
    RLDI[insights.reload]
    RLDP[policy.reload]
    VREP[insights_validation.txt]
  end

  %% Insights builder (can be called by EA or script)
  subgraph Builder[Insights Builder]
    IB[CInsightsBuilder.Build]
  end

  %% Scripts and CI helpers
  subgraph Scripts_CI[Scripts / CI]
    IRB[Scripts/InsightsRebuild.mq5]
    VAL[Scripts/ValidateInsights.mq5]
    KBC[kb_check.bat]
  end

  %% ML trainer
  subgraph ML[Trainer]
    TRN[train.py / policy_export.py]
  end

  %% LiveEA consumer
  subgraph Live
    LEA[LiveEA]
  end

  %% Paper data flows
  PEA --> STR
  STR -->|ExportFeatures| FEAT
  PEA --> TM
  PEA -->|logs| TRD
  PEA -->|events| EVT
  PEA -->|telemetry| TLM
  PEA <-->|caps read/write| EXPW
  PEA <-->|caps read/write| EXPD

  %% Insights lifecycle
  FEAT --> IB
  TRD --> IB
  RLDI -->|On-demand rebuild| IB
  PEA -. OnInit/OnTimer stale? .-> IB
  IB -->|write| INS
  INS -->|load| SEL
  SEL -->|gate decisions - bypassed by NoConstraintsMode| PEA

  %% Policy lifecycle
  FEAT --> TRN
  TRD --> TRN
  TRN -->|write| POL
  RLDP -->|reload trigger| PEA
  POL -->|load and scale| PEA

  %% Scripts / CI
  IRB -->|run builder| IB
  VAL -->|validate| VREP
  KBC -->|build & validate| IRB
  KBC -->|invoke| VAL
  KBC -->|report| VREP

  %% LiveEA consumption and feedback
  INS --> LEA
  POL --> LEA
  LEA -->|results| TRD
  LEA -->|events| EVT
  LEA -->|telemetry| TLM

Policy Lifecycle

• Location: DualEA/policy.json
• Reload trigger: touch DualEA/policy.reload
• The EA checks this in both OnTimer() and OnTick() (CheckPolicyReload()); when present, it reloads policy.json, logs Policy reload signal detected: reloaded|failed, and on success logs Policy gating: min_conf=..., slices=N.
• Loaded state: "loaded" iff one or more slices exist (slices > 0)
• Tracked via ArraySize(g_pol_strat) > 0 in PaperEA/PaperEA_v2.mq5

Fallback Behavior

• When UsePolicyGating=true:
  • fallback_no_policy: if no policy is loaded and FallbackWhenNoPolicy=true
  • fallback_policy_miss: if policy loaded but the exact slice is missing
• Both paths:
  • Demo-only unless FallbackDemoOnly=false
  • Bypass insights thresholds and exploration caps
  • Neutral scaling (no SL/TP/trailing multipliers applied)
• Journal examples:
  • FALLBACK: no policy loaded -> neutral scaling used for <strat> on <symbol>/<tf> demo=<true|false>
  • FALLBACK: policy slice missing -> neutral scaling used for <strat> on <symbol>/<tf> demo=<true|false>

Exploration Mode Management

• Purpose: allow limited trades to seed data for unseen slices.
• Counters:
  • Weekly file: explore_counts.csv (key,week_monday_yyyymmdd,count)
  • Daily file: explore_counts_day.csv (key,day_yyyymmdd,count)
• How counts advance:
  • On successful explore execution, EA increments and persists counters (SaveExploreCounts*())
• Caps and gating logs:
  • Allow: GATE: explore allow <strat> on <symbol>/<tf> reason=<...> (day=d/D, week=w/W) slice_exists=<true|false>
  • Block: GATE: blocked <strat> on <symbol>/<tf> reason=explore_cap_day|explore_cap_week ...
• Reset counters:
  • Delete explore_counts.csv and/or explore_counts_day.csv from the Common Files folder
• Notes:
  • Caps support 0 = unlimited for both daily and weekly limits.
  • When NoConstraintsMode=true, insights gating and exploration caps are bypassed entirely.

NoConstraintsMode (Data Collection Mode)

• Purpose: run PaperEA without constraints to maximize data collection for ML training.
• Behavior when true:
  • Bypasses: trading hours gate, selector gating, insights gating, exploration caps, and MaxOpenPositions guard.
  • Exploration caps accept 0 = unlimited, but are ignored in this mode.
  • Startup log (INFO): Mode: NoConstraintsMode=true -> bypass trading hours, selector gating, insights gating, exploration caps (0=unlimited), and MaxOpenPositions.
• Recommended usage: PaperEA on demo/test for aggressive data harvesting. Not recommended for LiveEA.

Rebuilding Insights

• Without re-running trades:
  • Run script Scripts/InsightsRebuild.mq5 from MT5 Navigator
  • Inputs: InFeaturesPath = "DualEA\\features.csv"
  • Output: DualEA/insights.json
  • Logs: InsightsRebuild: starting rebuild from features.csv ...

Troubleshooting & Common Messages

• Policy reload cannot open: Policy gating: cannot open DualEA\policy.json (Common). Err=<code>
• Policy reload detected: Policy reload signal detected: reloaded|failed
• Reload cleanup warning (non-fatal): Policy reload: cannot delete signal file DualEA\policy.reload (err=<code>)
• Cache loads:
  • Insights gating cache load: ok|fail
  • Policy gating cache load: ok|fail
  • Explore counters loaded: week=ok|fail day=ok|fail
• Execution blocks:
  • EXEC: blocked <strat> on <symbol>/<tf> due to MaxOpenPositions=<N> (only when MaxOpenPositions > 0; set to 0 to disable. Default: 0)
• Scan/gate summaries:
  • [SCAN] <symbol> tf=<tf> time=<ts> strats=<N> spread_pts=<x>
  • [STRAT] <name> p_win=<prob>
  • [GATE] <summary>

Maintenance Procedures

• Reset exploration only:
  • Delete explore_counts.csv and/or explore_counts_day.csv
• Full KB reset (dangerous, wipes history):
  • Delete knowledge_base.csv, knowledge_base_events.csv, features.csv, insights.json
• Force policy refresh from trainer:
  • Run ML/run_train_and_export.bat (writes policy.json, touches policy.reload)

Safety Notes

• Fallback trades are neutral and do not consume exploration quotas
• File I/O uses FILE_COMMON; ensure your MT5 has write permissions to the Common Files directory
• Some readers use FILE_ANSI and plain text parsing — avoid altering CSV delimiters/headers

Artifact Rotation & Compression

• All major artifacts (features.csv, insights.json, scaler.pkl, tf_model.keras, features.json) are automatically rotated if they exceed operational size limits.
• features.csv (MQL5): If file exceeds 100MB, it is renamed with a timestamped .bak suffix before new data is written. No compression (MQL5 limitation).
• ML artifacts (Python):
  • tf_model.keras: Rotated if above 100MB (no compression).
  • scaler.pkl, features.json: Rotated and compressed with gzip if above 20MB.
  • Old files are preserved with a timestamped .bak (and .gz for compressed).
• Rotation occurs before each new write session.

Trade Execution Retry Policy

• All trade executions (market, stop, limit) are guarded with a retry loop for transient broker errors (requote, busy, timeout, etc.).
• Up to 3 attempts are made, with a short sleep between retries.
• Each failure and retry is logged with error code, attempt number, and time.
• Non-retryable errors abort further attempts immediately.

Insights Rebuild Timeout

• The InsightsRebuild script now accepts a TimeoutMs input (default: 60,000 ms).
• The feature scan is chunked and logs progress every 1000 rows, with a brief sleep to avoid watchdog timeouts.
• If the timeout is exceeded, the rebuild aborts and logs the row/time boundary.