SniperGold_ML/ml/p3/p3_s25_external_data_validation/README.md

# P3-S25 — EXTERNAL DATA BASIS VALIDATION & INDEPENDENT POPULATION RECONCILIATION

```text
Session : P3-S25 — EXTERNAL DATA BASIS VALIDATION
Status  : RESEARCH-ONLY DATA VALIDATION (no ML, no tuning, no calibration)
Policy  : docs/CODE_VERIFICATION_AND_SILENT_BUG_POLICY_v1.md
Purpose : Determine whether the corrected SniperGold research population and
          UTC-clock M30 construction remain valid when independently checked
          against prepared external historical data.
Next    : P3-S26 = NOT STARTED (requires explicit owner authorization)
```

## Mission (abridged)

Answer whether the current absence of a demonstrated predictive edge is
genuine, or whether residual historical-data/basis differences could still
explain part of the result. Do **NOT** recover a positive ML result; do **NOT**
optimize AUC; do **NOT** modify the model. Eliminate or quantify remaining
data-basis uncertainty.

## External data candidates (authoritative locations)

| Source | Path | Nature |
|---|---|---|
| Tickstory → MT5 tick CSV | `MQL5\Files\XAUUSD_mt5_ticks.csv` | plain-text tick CSV, NO header, ~34.5 GB, 722,941,870 rows, 2003-05-05 → 2026-08-20 |
| Dukascopy via Tickstory | `MQL5\Files\XAUUSD\` | `YYYY/Mon/day/NNh_ticks.bi5` compressed container (zstd/lz4), 204,152 files (~3.2 GB) |

Each source is treated **independently** (never merged).

## Phases implemented

```text
0  Mandatory preflight (documented in docs/P3_S25_EXTERNAL_DATA_BASIS_VALIDATION.md)
1  External data inventory          -> s25_inventory.py
2  Data-quality gate                -> s25_quality.py
3  Tick -> M15 independent oracle   -> s25_oracle.py  (ticks->M15 UTC buckets)
4  M15 -> M30 independent oracle    -> s25_oracle.py  (M15->M30 UTC buckets; 2 impls 100% agree)
5  Compare with corrected basis     -> s25_parity.py
6  Data-basis classification        -> s25_classify.py
7  Candidate Setup population replay (research basis) -> s25_replay.py (PART A)
8  External population replay       -> s25_replay.py (PART B)
9  Identity-level reconciliation    -> s25_reconcile.py
10 Difference attribution           -> s25_attribution.py
11 Information-value test           -> s25_classify.py (Question set + hierarchy)
12 Silent-bug / mutation testing    -> s25_mutation.py
13 Reproducibility (run twice)      -> s25_repro.py
```

## Hard authorization boundary

Authorized: inventory, DQ validation, independent UTC M15/M30 construction,
broker vs external comparison, corrected population replay, identity
reconciliation, machine-readable evidence, mutation testing, documentation.
NOT authorized: ML retraining/tuning, calibration, feature engineering/removal,
label/TP/SL/horizon changes, nonlinear models, deployment, trading, production
MQL5 changes.

## Deliverables

- `docs/P3_S25_EXTERNAL_DATA_BASIS_VALIDATION.md`
- `docs/P3_S25_EXTERNAL_DATA_POPULATION_RECONCILIATION.md`
- `docs/P3_S25_EXTERNAL_DATA_DECISION.md`
- `docs/SESSION_HANDOVER_2026-08-26_P3_S25_EXTERNAL_DATA_BASIS_VALIDATION.md`
- this namespace (`ml/p3/p3_s25_external_data_validation/`)

## Determinism & reproducibility

Every script is deterministic; evidence is machine-readable (JSON/CSV) with a
SHA-256 manifest generated by `s25_manifest.py`. The runner
(`s25_run_main.py`) can be executed twice; substantive outputs are
byte-identical except generated-UCT timestamps and explicit environment
metadata.

## Hard stop

Commit only research artifacts, push to `origin/main`, verify clean tree, and
record the final SHA. `P3-S26 = NOT STARTED`.