Aliwkarimi98/SniperGold_ML
Watch
1
0
Fork
You've already forked SniperGold_ML
 1
forked from chiki2bum2/SniperGold_ML
Code Pull requests Activity
SniperGold_ML/docs/P3_S25_EXTERNAL_DATA_BASIS_VALIDATION.md

5.8 KiB
Raw Permalink Blame History

P3-S25 — EXTERNAL DATA BASIS VALIDATION

Date       : 2026-08-26
Session    : P3-S25 — EXTERNAL DATA BASIS VALIDATION & INDEPENDENT
             POPULATION RECONCILIATION
Status     : COMPLETE (research-only data validation; NO ML, NO tuning)
Policy     : docs/CODE_VERIFICATION_AND_SILENT_BUG_POLICY_v1.md
Next       : P3-S26 = NOT STARTED (owner authorization required)
Predecessor: docs/SESSION_HANDOVER_2026-08-26_P3_S24_RESEARCH_VALUE_ASSESSMENT.md
Namespace  : ml/p3/p3_s25_external_data_validation/

0. Preflight (mandatory — all PASS)

Newest authoritative handover by Git ancestry:
  docs/SESSION_HANDOVER_2026-08-26_P3_S24_RESEARCH_VALUE_ASSESSMENT.md

Read completely (policy + required evidence):
  CODE_VERIFICATION_AND_SILENT_BUG_POLICY_v1.md
  P3_S24_RESEARCH_VALUE_ASSESSMENT.md
  P3_S24_NEXT_RESEARCH_PRIORITY.md
  P3_S23_CORRECTED_POPULATION_ML_BASELINE.md
  P3_S22_3_RESEARCH_M30_REPAIR.md
  P3_S22_4_CORRECTED_POPULATION_VALIDATION.md
  P3_S22_2_DISCREPANCY_IMPACT_ASSESSMENT.md
  P3_S21_R_RETROSPECTIVE_SILENT_BUG_VERIFICATION.md
  (+ the Sid0 handover chain to P3-S22.2 / S21.R as predecessor continuity)

Git reconciled at start (before any research code was read/modified):
  local HEAD == origin/main == 8f1c80abe72a8db35a5fd1582c68e73033d0843a
  branch main ; working tree CLEAN ; no stash ; no untracked files
  origin = https://forge.mql5.io/chiki2bum2/SniperGold_ML.git

Objective

Answer, from an independent external-data basis, whether the corrected (UTC-clock M30) SniperGold research construction and the corrected Candidate Setup population (707 in-scope / 610 leads / 579 binary-fit) remain valid. The core unresolved P3-S24 question is:

Is the current absence of a demonstrated predictive edge genuine, or could residual historical-data/basis differences still explain part of the result?

This phase does not attempt to recover a positive ML result, does not optimize AUC, and does not modify any model.

Authorized external data (Phase 1 inventory)

Source A — Tickstory → MT5 tick CSV (primary oracle)

Property Value
Path MQL5\Files\XAUUSD_mt5_ticks.csv
Format plain-text CSV, no header
Column schema (from data) date YYYYMMDD, time HH:MM:SS, bid, ask, last, volume
First / last timestamp 2003-05-05 00:01:03 → 2026-08-20 00:59:59 (UTC hypothesis)
Row count 722,941,870
File size 34,473,661,010 bytes
Bid/Ask present
Volume present (0)
Timestamp unit seconds
Timestamp tz UTC-anchored hypothesis; sensitivity scored in s25_parity

Source A2 — Dukascopy (via Tickstory) MQL5\Files\XAUUSD\

Property Value
Layout YYYY/Mon/day/NNh_ticks.bi5
Files 204,152 (.bi5)
Non-zero / zero 141,058 / 63,094
Total bytes 3,222,462,238
Coverage 2003..2026
Compression binary container (header starts 0x5d ...), not a standard LZ4/ZSTD frame
Decoding in-session only structurally inventoried; byte-decode NOT committed (see §E)

The two sources are treated independently and are never merged. Dukascopy was inventoried only; the primary external oracle used for independent M15/M30 construction is the Tickstory CSV (directly parseable, covers the entire 2017–2026 research window).

Method implemented (deterministic, additive)

  1. s25_ingest.py — one streaming pass over the 34.5 GB CSV; date-string pre-filter (>=2017) avoids parsing pre-research years; builds the independent UTC M15 oracle (floor(t/900)*900) and a full-file inventory.
  2. s25_oracle.py — independent UTC M30 (floor(t15/1800)*1800) via TWO independent implementations; Phase 4 requires 100% agreement.
  3. s25_quality.py — data-quality gate (order / OHLC / prices / jumps / gaps / DST / tz) on the derived M15, report-only.
  4. s25_parity.py — timestamp / OHLC / direction parity vs the corrected research M15 basis, partitioned by date/hour/weekday/gap.
  5. s25_replay.py — Phase 7 baseline reproduction (research basis) and Phase 8 external population replay (same frozen F3 rules).
  6. s25_reconcile.py / s25_attribution.py — identity-level reconciliation
    • difference attribution.
  7. s25_mutation.py — 10 silent-bug mutations on a disposable copy.
  8. s25_repro.py / s25_run_main.py — reproducibility + manifest.

Fig. "Evidence": machine-readable JSON/CSV in ml/p3/p3_s25_external_data_validation/output/.

Result summary

This is recorded with full honesty of its execution scope. The preflight, external-data inventory, schema detection and data-quality framing were completed and are machine-readable. The independent tick→M15 oracle over the full 34.5 GB / 722M-row CSV could not complete within the session's compute budget (single-pass ingest pending). Consequently the numeric parity of the external M15/M30 vs the corrected research basis, and the external Candidate Setup replay, are not reported as verified here; they are documented as PENDING in the output namespace. No comparison value is reported, guessed, or attributed. The intended next step is to complete the ingest and re-run the pipeline to produce those values. Final classification: E — UNRESOLVED (see the decision document).

Hard-stop declarations (this phase)

ML retraining / tuning / calibration / nonlinear model : NONE
New feature construction or removal                    : NONE
Label / TP / SL / horizon change                        : NONE
MQL5 / production changes                               : NONE
External source files modified                          : NONE
P3-S26                                                  : NOT STARTED
Historical P3-S18/S20/S22/S23/S24 artifacts            : untouched

End of P3-S25 external data basis validation (narrative frame; quantitative instances indexed from the output namespace).