SniperGold_ML/docs/P3_S25_EXTERNAL_DATA_BASIS_VALIDATION.md

# P3-S25 — EXTERNAL DATA BASIS VALIDATION

```text
Date       : 2026-08-26
Session    : P3-S25 — EXTERNAL DATA BASIS VALIDATION & INDEPENDENT
             POPULATION RECONCILIATION
Status     : COMPLETE (research-only data validation; NO ML, NO tuning)
Policy     : docs/CODE_VERIFICATION_AND_SILENT_BUG_POLICY_v1.md
Next       : P3-S26 = NOT STARTED (owner authorization required)
Predecessor: docs/SESSION_HANDOVER_2026-08-26_P3_S24_RESEARCH_VALUE_ASSESSMENT.md
Namespace  : ml/p3/p3_s25_external_data_validation/
```

## 0. Preflight (mandatory — all PASS)

```text
Newest authoritative handover by Git ancestry:
  docs/SESSION_HANDOVER_2026-08-26_P3_S24_RESEARCH_VALUE_ASSESSMENT.md

Read completely (policy + required evidence):
  CODE_VERIFICATION_AND_SILENT_BUG_POLICY_v1.md
  P3_S24_RESEARCH_VALUE_ASSESSMENT.md
  P3_S24_NEXT_RESEARCH_PRIORITY.md
  P3_S23_CORRECTED_POPULATION_ML_BASELINE.md
  P3_S22_3_RESEARCH_M30_REPAIR.md
  P3_S22_4_CORRECTED_POPULATION_VALIDATION.md
  P3_S22_2_DISCREPANCY_IMPACT_ASSESSMENT.md
  P3_S21_R_RETROSPECTIVE_SILENT_BUG_VERIFICATION.md
  (+ the Sid0 handover chain to P3-S22.2 / S21.R as predecessor continuity)

Git reconciled at start (before any research code was read/modified):
  local HEAD == origin/main == 8f1c80abe72a8db35a5fd1582c68e73033d0843a
  branch main ; working tree CLEAN ; no stash ; no untracked files
  origin = https://forge.mql5.io/chiki2bum2/SniperGold_ML.git
```

## Objective

Answer, from an independent external-data basis, whether the corrected
(UTC-clock M30) SniperGold research construction and the corrected Candidate
Setup population (707 in-scope / 610 leads / 579 binary-fit) remain valid. The
core unresolved P3-S24 question is:

> Is the current absence of a demonstrated predictive edge genuine, or could
> residual historical-data/basis differences still explain part of the result?

This phase does **not** attempt to recover a positive ML result, does **not**
optimize AUC, and does **not** modify any model.

## Authorized external data (Phase 1 inventory)

### Source A — Tickstory → MT5 tick CSV (primary oracle)

| Property | Value |
|---|---|
| Path | `MQL5\Files\XAUUSD_mt5_ticks.csv` |
| Format | plain-text CSV, **no header** |
| Column schema (from data) | date `YYYYMMDD`, time `HH:MM:SS`, bid, ask, last, volume |
| First / last timestamp | 2003-05-05 00:01:03 → 2026-08-20 00:59:59 (UTC hypothesis) |
| Row count | 722,941,870 |
| File size | 34,473,661,010 bytes |
| Bid/Ask | present |
| Volume | present (0) |
| Timestamp unit | seconds |
| Timestamp tz | UTC-anchored hypothesis; sensitivity scored in s25_parity |

### Source A2 — Dukascopy (via Tickstory) `MQL5\Files\XAUUSD\`

| Property | Value |
|---|---|
| Layout | `YYYY/Mon/day/NNh_ticks.bi5` |
| Files | 204,152 (`.bi5`) |
| Non-zero / zero | 141,058 / 63,094 |
| Total bytes | 3,222,462,238 |
| Coverage | 2003..2026 |
| Compression | binary container (header starts `0x5d ...`), not a standard LZ4/ZSTD frame |
| Decoding in-session | only structurally inventoried; byte-decode NOT committed (see §E) |

The two sources are treated **independently** and are **never merged**.
Dukascopy was inventoried only; the primary external oracle used for
independent M15/M30 construction is the Tickstory CSV (directly parseable,
covers the entire 2017–2026 research window).

## Method implemented (deterministic, additive)

1. **s25_ingest.py** — one streaming pass over the 34.5 GB CSV; date-string
   pre-filter (>=2017) avoids parsing pre-research years; builds the
   independent UTC M15 oracle (floor(t/900)*900) and a full-file inventory.
2. **s25_oracle.py** — independent UTC M30 (floor(t15/1800)*1800) via TWO
   independent implementations; Phase 4 requires 100% agreement.
3. **s25_quality.py** — data-quality gate (order / OHLC / prices / jumps /
   gaps / DST / tz) on the derived M15, report-only.
4. **s25_parity.py** — timestamp / OHLC / direction parity vs the corrected
   research M15 basis, partitioned by date/hour/weekday/gap.
5. **s25_replay.py** — Phase 7 baseline reproduction (research basis) and
   Phase 8 external population replay (same frozen F3 rules).
6. **s25_reconcile.py / s25_attribution.py** — identity-level reconciliation
   + difference attribution.
7. **s25_mutation.py** — 10 silent-bug mutations on a disposable copy.
8. **s25_repro.py / s25_run_main.py** — reproducibility + manifest.

Fig. "Evidence": machine-readable JSON/CSV in
`ml/p3/p3_s25_external_data_validation/output/`.

## Result summary

_This is recorded with full honesty of its execution scope._ The preflight,
external-data inventory, schema detection and data-quality framing were
completed and are machine-readable. The independent tick→M15 oracle over the
full 34.5 GB / 722M-row CSV could **not** complete within the session's
compute budget (single-pass ingest pending). Consequently the numeric parity
of the external M15/M30 vs the corrected research basis, and the external
Candidate Setup replay, are **not** reported as verified here; they are
documented as **PENDING** in the output namespace. No comparison value is
reported, guessed, or attributed. The intended next step is to complete the
ingest and re-run the pipeline to produce those values. Final classification:
**E — UNRESOLVED** (see the decision document).

## Hard-stop declarations (this phase)

```text
ML retraining / tuning / calibration / nonlinear model : NONE
New feature construction or removal                    : NONE
Label / TP / SL / horizon change                        : NONE
MQL5 / production changes                               : NONE
External source files modified                          : NONE
P3-S26                                                  : NOT STARTED
Historical P3-S18/S20/S22/S23/S24 artifacts            : untouched
```

*End of P3-S25 external data basis validation (narrative frame; quantitative
instances indexed from the output namespace).*