mql5/Experts/Advisors/DualEA/docs/Operations.md

# Operations Guide (DualEA)

Practical procedures for managing DualEA runtime, knowledge base, policy reloads, and interpreting logs. All paths are under MT5 Common Files.

- Typical path: `C:\Users\<you>\AppData\Roaming\MetaQuotes\Terminal\Common\Files\DualEA\`
- Alternate: `C:\ProgramData\MetaQuotes\Terminal\Common\Files\DualEA\`

## Controls & Inputs (PaperEA)
- `NoConstraintsMode` — disable all gating and caps for maximum data collection (bypasses trading hours, selector gating, insights gating, exploration caps, and open-position limit). Default: `true`.
- `UsePolicyGating` — enable/disable ML policy gating
- `DefaultPolicyFallback` — allow neutral trading when slice/policy missing
- `FallbackDemoOnly` — restrict fallback to demo only (default true)
- `FallbackWhenNoPolicy` — allow fallback when no policy file is loaded (default true)
- `ExploreMaxPerSlice` — weekly cap (default 3; `0` = unlimited)
- `ExploreMaxPerSlicePerDay` — daily cap (default 2; `0` = unlimited)
- `MaxOpenPositions` — global open-position limit (default `0`; `0` = unlimited)
- `Verbosity` — controls `ShouldLog(level)` (INFO/DEBUG)

## Knowledge Base Files
- Trades: `knowledge_base.csv` (see `docs/KB-Schemas.md`)
- Events: `knowledge_base_events.csv`
- Features: `features.csv` (long format, includes `r_multiple` rows)
- Insights: `insights.json` (built via script or builder)
- Exploration counters: `explore_counts.csv`, `explore_counts_day.csv`

## System Flow
```mermaid
flowchart LR
  %% PaperEA orchestration
  subgraph Paper
    PEA[PaperEA]
    STR[Strategies - IStrategy]
    SEL[Selector - insights + recency]
    TM[TradeManager - SL/TP/Trailing]
    TEL[Telemetry]
  end

  %% Common Files runtime artifacts
  subgraph KB["Knowledge Base - Common Files DualEA"]
    FEAT[features.csv]
    TRD[knowledge_base.csv]
    EVT[knowledge_base_events.csv]
    TLM[telemetry logs]
    INS[insights.json]
    POL[policy.json]
    EXPW[explore_counts.csv]
    EXPD[explore_counts_day.csv]
    RLDI[insights.reload]
    RLDP[policy.reload]
    VREP[insights_validation.txt]
  end

  %% Insights builder (can be called by EA or script)
  subgraph Builder[Insights Builder]
    IB[CInsightsBuilder.Build]
  end

  %% Scripts and CI helpers
  subgraph Scripts_CI[Scripts / CI]
    IRB[Scripts/InsightsRebuild.mq5]
    VAL[Scripts/ValidateInsights.mq5]
    KBC[kb_check.bat]
  end

  %% ML trainer
  subgraph ML[Trainer]
    TRN[train.py / policy_export.py]
  end

  %% LiveEA consumer
  subgraph Live
    LEA[LiveEA]
  end

  %% Paper data flows
  PEA --> STR
  STR -->|ExportFeatures| FEAT
  PEA --> TM
  PEA -->|logs| TRD
  PEA -->|events| EVT
  PEA -->|telemetry| TLM
  PEA <-->|caps read/write| EXPW
  PEA <-->|caps read/write| EXPD

  %% Insights lifecycle
  FEAT --> IB
  TRD --> IB
  RLDI -->|On-demand rebuild| IB
  PEA -. OnInit/OnTimer stale? .-> IB
  IB -->|write| INS
  INS -->|load| SEL
  SEL -->|gate decisions - bypassed by NoConstraintsMode| PEA

  %% Policy lifecycle
  FEAT --> TRN
  TRD --> TRN
  TRN -->|write| POL
  RLDP -->|reload trigger| PEA
  POL -->|load and scale| PEA

  %% Scripts / CI
  IRB -->|run builder| IB
  VAL -->|validate| VREP
  KBC -->|build & validate| IRB
  KBC -->|invoke| VAL
  KBC -->|report| VREP

  %% LiveEA consumption and feedback
  INS --> LEA
  POL --> LEA
  LEA -->|results| TRD
  LEA -->|events| EVT
  LEA -->|telemetry| TLM
```

## Policy Lifecycle
- Location: `DualEA/policy.json`
- Reload trigger: touch `DualEA/policy.reload`
- The EA checks this in both `OnTimer()` and `OnTick()` (`CheckPolicyReload()`); when present, it reloads `policy.json`, logs `Policy reload signal detected: reloaded|failed`, and on success logs `Policy gating: min_conf=..., slices=N`.
- Loaded state: "loaded" iff one or more slices exist (slices > 0)
- Tracked via `ArraySize(g_pol_strat) > 0` in `PaperEA/PaperEA_v2.mq5`

## Fallback Behavior
- When `UsePolicyGating=true`:
  - `fallback_no_policy`: if no policy is loaded and `FallbackWhenNoPolicy=true`
  - `fallback_policy_miss`: if policy loaded but the exact slice is missing
- Both paths:
  - Demo-only unless `FallbackDemoOnly=false`
  - Bypass insights thresholds and exploration caps
  - Neutral scaling (no SL/TP/trailing multipliers applied)
- Journal examples:
  - `FALLBACK: no policy loaded -> neutral scaling used for <strat> on <symbol>/<tf> demo=<true|false>`
  - `FALLBACK: policy slice missing -> neutral scaling used for <strat> on <symbol>/<tf> demo=<true|false>`

## Exploration Mode Management
- Purpose: allow limited trades to seed data for unseen slices.
- Counters:
  - Weekly file: `explore_counts.csv` (key,week_monday_yyyymmdd,count)
  - Daily file: `explore_counts_day.csv` (key,day_yyyymmdd,count)
- How counts advance:
  - On successful explore execution, EA increments and persists counters (`SaveExploreCounts*()`)
- Caps and gating logs:
  - Allow: `GATE: explore allow <strat> on <symbol>/<tf> reason=<...> (day=d/D, week=w/W) slice_exists=<true|false>`
  - Block: `GATE: blocked <strat> on <symbol>/<tf> reason=explore_cap_day|explore_cap_week ...`
- Reset counters:
  - Delete `explore_counts.csv` and/or `explore_counts_day.csv` from the Common Files folder
 - Notes:
   - Caps support `0 = unlimited` for both daily and weekly limits.
   - When `NoConstraintsMode=true`, insights gating and exploration caps are bypassed entirely.

## NoConstraintsMode (Data Collection Mode)
- Purpose: run PaperEA without constraints to maximize data collection for ML training.
- Behavior when `true`:
  - Bypasses: trading hours gate, selector gating, insights gating, exploration caps, and `MaxOpenPositions` guard.
  - Exploration caps accept `0 = unlimited`, but are ignored in this mode.
  - Startup log (INFO): `Mode: NoConstraintsMode=true -> bypass trading hours, selector gating, insights gating, exploration caps (0=unlimited), and MaxOpenPositions`.
- Recommended usage: PaperEA on demo/test for aggressive data harvesting. Not recommended for LiveEA.

## Rebuilding Insights
- Without re-running trades:
  - Run script `Scripts/InsightsRebuild.mq5` from MT5 Navigator
  - Inputs: `InFeaturesPath = "DualEA\\features.csv"`
  - Output: `DualEA/insights.json`
  - Logs: `InsightsRebuild: starting rebuild from features.csv ...`

## Troubleshooting & Common Messages
- Policy reload cannot open: `Policy gating: cannot open DualEA\policy.json (Common). Err=<code>`
- Policy reload detected: `Policy reload signal detected: reloaded|failed`
- Reload cleanup warning (non-fatal): `Policy reload: cannot delete signal file DualEA\policy.reload (err=<code>)`
- Cache loads:
  - `Insights gating cache load: ok|fail`
  - `Policy gating cache load: ok|fail`
  - `Explore counters loaded: week=ok|fail day=ok|fail`
- Execution blocks:
  - `EXEC: blocked <strat> on <symbol>/<tf> due to MaxOpenPositions=<N>` (only when `MaxOpenPositions > 0`; set to `0` to disable. Default: `0`)
- Scan/gate summaries:
  - `[SCAN] <symbol> tf=<tf> time=<ts> strats=<N> spread_pts=<x>`
  - `[STRAT] <name> p_win=<prob>`
  - `[GATE] <summary>`

## Maintenance Procedures
- Reset exploration only:
  - Delete `explore_counts.csv` and/or `explore_counts_day.csv`
- Full KB reset (dangerous, wipes history):
  - Delete `knowledge_base.csv`, `knowledge_base_events.csv`, `features.csv`, `insights.json`
- Force policy refresh from trainer:
  - Run `ML/run_train_and_export.bat` (writes `policy.json`, touches `policy.reload`)

## Safety Notes
- Fallback trades are neutral and do not consume exploration quotas
- File I/O uses `FILE_COMMON`; ensure your MT5 has write permissions to the Common Files directory
- Some readers use `FILE_ANSI` and plain text parsing — avoid altering CSV delimiters/headers

## Artifact Rotation & Compression
- All major artifacts (features.csv, insights.json, scaler.pkl, tf_model.keras, features.json) are automatically rotated if they exceed operational size limits.
- **features.csv** (MQL5): If file exceeds 100MB, it is renamed with a timestamped `.bak` suffix before new data is written. No compression (MQL5 limitation).
- **ML artifacts** (Python):
  - `tf_model.keras`: Rotated if above 100MB (no compression).
  - `scaler.pkl`, `features.json`: Rotated and compressed with gzip if above 20MB.
  - Old files are preserved with a timestamped `.bak` (and `.gz` for compressed).
- Rotation occurs before each new write session.

## Trade Execution Retry Policy
- All trade executions (market, stop, limit) are guarded with a retry loop for transient broker errors (requote, busy, timeout, etc.).
- Up to 3 attempts are made, with a short sleep between retries.
- Each failure and retry is logged with error code, attempt number, and time.
- Non-retryable errors abort further attempts immediately.

## Insights Rebuild Timeout
- The InsightsRebuild script now accepts a `TimeoutMs` input (default: 60,000 ms).
- The feature scan is chunked and logs progress every 1000 rows, with a brief sleep to avoid watchdog timeouts.
- If the timeout is exceeded, the rebuild aborts and logs the row/time boundary.