forked from chiki2bum2/SniperGold_ML
16 KiB
16 KiB
P3-S20 WALK-FORWARD BASELINE — PRE-REGISTERED EXPANDING-WINDOW VALIDATION
Date : 2026-08-24
Session : P3-S.20 — Pre-Registered Expanding-Window Walk-Forward Baseline
Status : COMPLETE
Verdict : A — STABLE WEAK SIGNAL (pre-registered decision rule output)
Scope : VALIDATION / GENERALIZATION PHASE. LOGISTIC ONLY.
NO tree/boost/MLP/LSTM/Informer/regime/ensemble, NO feature or
label change, NO TP/SL/horizon/de-overlap change, NO threshold/
HP tuning, NO fold redesign after results, NO Tickstory/Dukascopy,
NO production/MQL5/legacy-MLP change.
Question : DOES THE WEAK LOGISTIC SIGNAL OBSERVED IN P3-S18 (TEST ROC-AUC
~0.609) SURVIVE PRE-REGISTERED TEMPORAL OUT-OF-SAMPLE TESTING?
Human verif: REMAINS CANCELLED (historical only).
A. Latest SESSION_HANDOVER used
docs/SESSION_HANDOVER_2026-08-24_P3_S19_DATASET_FEATURE_AUDIT.md
(newest authoritative handover; committed at HEAD 84b0e56; read completely
FIRST, before any experiment — session-handover-first rule).
Reconciles with Git :
local HEAD == origin/main == 84b0e56fc1e6ba2be503f462c4ef1a58e0830915
branch main, working tree CLEAN,
origin = https://forge.mql5.io/chiki2bum2/SniperGold_ML.git
Superseded-in-line reads (all read completely):
docs/P3_S18_BASELINE_ML_REPORT.md (P3-S18 INCONCLUSIVE / DATA TOO SMALL)
docs/P3_S18A_LABEL_CONTRACT_REVIEW.md (label contract APPROVED AS V1)
docs/P3_S19_DATASET_FEATURE_AUDIT.md (AUDIT PASSED; 686=594+92)
docs/P3_S16_LABEL_CONTRACT.md / docs/P3_S16_SETUP_DATASET_CONTRACT_v1.md
docs/SNIPERGOLD_CANONICAL_SETUP_CONTRACT_v1.md
docs/P3_S17R2_VECTORIZE_CH0CH_OB_FVG.md (FULL PARITY ACHIEVED)
B. Checkpoint
Starting checkpoint (verified): 84b0e56fc1e6ba2be503f462c4ef1a58e0830915
local == origin/main, branch main, working tree CLEAN,
origin = https://forge.mql5.io/chiki2bum2/SniperGold_ML.git
Final Forge HEAD : <P3_S20_FINAL_SHA recorded after push>
C. Pre-registered experiment definition
DESIGN (frozen BEFORE any final OOS metric was computed):
Unit : ONE Candidate Setup = ONE binary observation (WIN=1/LOSS=0)
on the causally-verified population (P3-S16 v1 label).
Order : chronological by creation_bar (no shuffle).
Scheme : expanding-window temporal walk-forward, 3 folds.
FOLD BOUNDS (indices over the 571 sorted binary rows):
Fold 1 : train rows [0:300] OOS rows [300:395] (OOS ~95)
Fold 2 : train rows [0:395] OOS rows [395:490] (expanding; OOS ~95)
Fold 3 : train rows [0:490] OOS rows [490:571] (expanding; OOS ~81)
(Set from the brief's target structure: Fold1 train ~300 / OOS ~90-100;
Fold2/3 expanding with OOS ~90-100 then ~80-100. NOT derived from
performance.)
Purge : gap >= HORIZON(16 bars) enforced at every fold boundary
(verified 555 / 472 / 321 bars — satisfied natively by the
de-overlapped lead stream; asserted, never silently trimmed).
Preprocess : StandardScaler fitted on TRAINING rows ONLY per fold; applied
UNCHANGED to OOS. (Only train-learned preprocessing allowed.)
Model : LogisticRegression(C=1.0, L2, max_iter=5000, random_state=42)
— byte-for-byte the P3-S18 frozen configuration.
Threshold : 0.5 ONLY for the confusion / precision / recall tables.
Comparators : per-fold majority-class (constant-prior) baseline whose score
= the TRAIN WIN prevalence (no OOS leakage); ROC of a
constant = 0.5 by construction.
Quality gate: per fold OOS >= 50 binary obs (pref >= 100), >= 20 WIN,
>= 20 LOSS; failing fold -> LOW_STATISTICAL_POWER preserved.
Aggregation : pooled OOS (exact, sample-aware) + per-fold mean/median/std.
Decision rule: pre-registered (see section N) — applied unchanged to the
run output; no uncertainty method selected post-hoc.
D. Dataset / population
Reconstructed via ml/p3/baseline/prepare_dataset.py UNCHANGED from the
byte-locked P3-S.17R.2 chain (FULL PARITY; runtime==research 694==694):
all in-scope Candidate Setups : 686
leads (independent) : 594
follow-ons (preserved) : 92
binary fit rows (WIN/LOSS leads): 571
lead classes : WIN 167 | LOSS 404 | UNRES 18 | AMBIG 5
duplicates : 0
feature schema : 12 causal as-of-entry features
feature_sha16 = 0414e401522ea4e2
(identical to P3-S18/P3-S19 — unchanged)
label contract : P3-S16 label contract v1 (frozen)
de-overlap : lead-per-episode (unchanged)
E. Fold definitions
Chronological binary-row windows (creation_bar ranges):
Fold 1 train bars 1077 .. 99013 (n=300) OOS bars 99568 .. 134069 (n=95)
Fold 2 train bars 1077 .. 134069 (n=395) OOS bars 134541 .. 167070 (n=95)
Fold 3 train bars 1077 .. 167070 (n=490) OOS bars 167391 .. 193899 (n=81)
OOS quality-gate checks (binary): F1 95 (25W/70L) | F2 95 (23W/72L) |
F3 81 (26W/55L) -> all PASS >= 50 obs, >= 20 WIN, >= 20 LOSS.
NO fold marked LOW_STATISTICAL_POWER.
UNRESOLVED / AMBIGUOUS preserved per OOS temporal window (leads incl.
non-binary classes, never forced):
F1 OOS window : 102 leads = 25 WIN / 70 LOSS / 6 UNRES / 1 AMBIG
F2 OOS window : 98 leads = 23 WIN / 72 LOSS / 2 UNRES / 1 AMBIG
F3 OOS window : 85 leads = 26 WIN / 55 LOSS / 3 UNRES / 1 AMBIG
F. Purge / temporal isolation
gap (OOS first bar - last train bar): 555 / 472 / 321 bars, all >> H=16.
No label-window overlap across boundaries (asserted per fold).
Scaler fit on training only; no future preprocessing information.
Chronological order enforced; no reshuffling (WF-T01/T02/T03 PASS).
G. Majority baseline (per fold, train-derived prior)
Fold train_prior(WIN) ROC PR(=prior) LogLoss Brier
1 0.3100 0.5 0.3100 0.5816 0.1961
2 0.2987 0.5 0.2987 0.5615 0.1867
3 0.2878 0.5 0.2878 0.6302 0.2191
pooled 0.2731 0.5 0.2731 0.5863 0.1985
The majority comparator never sees OOS labels (prior from training only).
H. Logistic OOS results
Fold 1 Fold 2 Fold 3 Pooled
OOS n 95 95 81 271
OOS WIN/LOSS 25/70 23/72 26/55 74/197
ROC-AUC 0.5337 0.6341 0.5350 0.5792
PR-AUC (WIN) 0.3302 0.4224 0.3715 0.3417
LogLoss 0.5815 0.5467 0.6491 0.5895
Brier 0.1958 0.1806 0.2288 0.2003
Balanced acc (macro) 0.5200 0.5000 0.4951 0.5109
WIN precision (0.5) 1.000 0.000 0.308 0.357 (5 TP)
WIN recall (0.5) 0.040 0.000 0.154 0.068
Accuracy (0.5) 0.747 0.758 0.617 0.712
Confusion (0.5) [1/24;0/70] [0/23;0/72] [4/22;9/46] 5-69/9-188
Train ROC-AUC 0.5564 0.5475 0.5541 - (n=300/395/490)
Train LogLoss 0.6137 0.6037 0.5910 -
Majority comparison per fold (delta = logistic - majority):
ROC : +0.034 / +0.134 / +0.035 (all folds > 0.5 baseline)
PR : +0.020 / +0.124 / +0.084 (all folds beat the prevalence prior)
LogLoss: -0.0001 / -0.0148 / +0.0189 (2 of 3 better or tie)
Brier : -0.0003 / -0.0061 / +0.0098 (2 of 3 better or tie)
I. Calibration / log loss
The linear model carries RANKING information (AUC/PR-AUC), not materially
improved CALIBRATION: pooled LogLoss 0.5895 vs constant-prior 0.5863
(logistic marginally WORSE); pooled Brier 0.2003 vs 0.1985. Fold-wise the
model beats the prior on LogLoss/Brier in folds 2, ties in fold 1, and is
worse in fold 3. At threshold 0.5 the classifier is extremely conservative
(predicts nearly all LOSS), so precision/recall on WIN are near zero — the
weak signal is a soft ranking edge, NOT a usable hard-classification rule at
0.5. Consistent with P3-S18's observation that logistic is
"essentially uncalibrated-informative" at this sample size.
J. Per-fold stability
ROC-AUC across OOS folds: 0.534 / 0.634 / 0.535.
- direction consistent: ALL 3 folds above the 0.5 majority baseline,
- no catastrophic collapse in any fold,
- strongest period = fold 2 (0.634), folds 1 and 3 are weak-positive
(~0.53-0.54, near noise but consistently above 0.5),
- PR-AUC beats its majority prior in ALL 3 folds (0.33/0.42/0.37 vs
0.31/0.30/0.29) — the ranking edge is not confined to one period,
- no chronological collapse (train 0.556/0.548/0.554 -> OOS positive).
Interpretation: modest but consistent; effect size small; fold 2 carries the
largest single contribution but folds 1 and 3 still clear the baseline.
K. Statistical uncertainty
Per-fold OOS n = 95/95/81; pooled n = 271 (74 WIN / 197 LOSS).
At these counts a ROC-AUC of ~0.53 is within noise bands; 0.634 in fold 2
alone is not strong evidence; but the pooled 0.579 over 271 independent
de-overlapped rows and the 3/3 folds above baseline + 3/3 PR-AUC beat are the
consistent pattern. A bootstrap/permutation interval was NOT selected after
seeing the metrics (pre-registration discipline); the confidence statement is
therefore qualitative: the direction is consistent, the magnitude is weak,
and the pooled estimate is modestly above random. CIs on AUC of a 271/74-WIN
sample are wide (~±0.05 at 95% level); the 0.579 pooled value is fully
consistent with a small real edge OR a favorable draw, but NOT with a
reproducibly strong effect. Calibration (LogLoss/Brier) provides no
independent confirmation, so the signal should be treated as a WEAK ranking
hint only.
L. Reproducibility
The experiment was run 3+ times with the fixed seed/design:
- p3_s20_fold_results.csv and p3_s20_oos_predictions.csv BYTE-IDENTICAL
across runs,
- p3_s20_summary.json / manifest identical except generated_utc (and the
git_commit field, which is recorded at run time),
- WF-T01..T06 PASS (chronology, expanding windows, purge, train-only
preprocessing, quality gate, majority comparator, deterministic AUROC).
Deterministic pipeline: prepare_dataset (frozen) -> walk_forward (frozen
design) -> metric helpers (frozen). No randomness beyond the fixed seed.
M. Failure diagnosis, if any
No process/reproducibility failure (E not applicable). The signal did NOT
fail: Logistic kept a weak-positive OOS profile rather than collapsing. The
classification of the failure space (brief section 27) is therefore not
triggered; the nearest descriptive labels are:
SAMPLE SIZE : primary limiter — 271 pooled / ~95 per fold leaves wide CIs;
TEMPORAL DRIFT / POPULATION HETEROGENEITY : fold 2 (0.634) vs folds 1/3
(~0.53) shows period heterogeneity in effect size, though not direction.
No label/feature/TP/SL/horizon change was made or proposed as a result.
N. Final decision
FINAL DECISION : A — STABLE WEAK SIGNAL
Pre-registered decision rule (frozen before the run, applied unchanged):
- quality gates all PASS (no LOW_STATISTICAL_POWER) -> not D,
- all 3 OOS folds have ROC-AUC > 0.5 and deviations > 0.02
(0.0337 / 0.1341 / 0.0350) -> A-track,
- pooled OOS ROC-AUC 0.5792 > 0.5 -> A-track confirmed,
-> A_STABLE_WEAK_SIGNAL.
Qualitative support (brief section 15/30):
- logistic > majority on most OOS folds (3/3 on ROC and PR-AUC),
- direction consistent (all folds positive; no fold < 0.5, no collapse),
- effect not driven by a single isolated period (folds 1 and 3 are
positive, not merely fold 2),
- calibration stays sensible (LogLoss/Brier near prior, no blow-up),
- CAVEATS: folds 1/3 are near-noise (~0.53); LogLoss/Brier do not beat
the base rate; the edge is a weak ranking effect, not a usable
0.5-threshold classifier.
Neither "weak turned positive by average" nor "manufactured result": the
decision is per-fold consistent, pooled, and pre-registered.
O. Production files unchanged
NO MQL5 / F1-F4 / Candidate Setup / FEATURE_CONTRACT / SniperGold_ML / EA
change. Research-only artifacts under ml/p3/baseline/ (walk_forward.py,
test_walk_forward.py, output/p3_s20_*).
P. Legacy MLP unchanged
The legacy SniperGold_ML.mqh MLP (19->12->2 bar-level L1 direction drift)
remains FROZEN LEGACY / REFERENCE; NOT retrained, NOT exported, NOT compared
as a same-target benchmark. No model weights / legacy labels / calibration
modified.
Q. External data status
NOT introduced. Same validated XAUUSD M15 feed as P3-S18/P3-S19/P3-S17R.2.
Tickstory / Dukascopy remain a future, separate, authorized robustness study.
R. Forge commits
1. test: define pre-registered walk-forward baseline coverage (WF-T01..T06)
2. research: run setup-level logistic walk-forward (P3-S20)
3. docs: record P3-S20 walk-forward results (this report + handover)
P3_S20_FINAL_SHA = <recorded after push>
S. Final Forge HEAD
See Provenance (verified local == remote == P3_S20_FINAL_SHA after push).
T. Working tree
CLEAN after push (verified). Regression re-runs green then restored
byte-identical:
P3-S16 20/20 | VEC 15/15 | chain parity 694==694 | S19 audit 12/12 PASS
(committed outputs restored byte-identical via git checkout after the
re-runs — only generated_utc had changed).
U. P3-S21 readiness
Decision = A (STABLE WEAK SIGNAL). A future, SEPARATELY AUTHORIZED phase may
examine (brief section 31):
- feature mechanism (coefficient profile / zone geometry) — descriptive,
- calibration diagnostics (probability calibration, not threshold tuning),
- restrained nonlinear models ONLY after reproducible signal AND separate
authorization.
For (B)-style instability the recommended path would be population
heterogeneity / label-exit diagnostics. NOTHING is auto-started: no feature
hunting, no label change, no TP/SL/horizon change, no LSTM/Informer/regime,
no Tickstory/Dukascopy, no deployment.
Provenance / evidence
Scripts (research-only, new in P3-S20):
ml/p3/baseline/walk_forward.py (frozen design + logistic fits)
ml/p3/baseline/test_walk_forward.py (WF-T01..T06; 6/6 PASS)
Outputs (ml/p3/baseline/output/):
p3_s20_walkforward_manifest.json (schema/hash/folds/gate/git commit)
p3_s20_fold_results.csv (per-fold metrics + majority)
p3_s20_oos_predictions.csv (271 pooled OOS rows + priors)
p3_s20_summary.json (aggregates + final decision)
p3_s20_wf_tests.json (WF-T01..T06 results)
Reused (UNCHANGED):
ml/p3/baseline/prepare_dataset.py (12-feature schema, label v1)
ml/p3/baseline/evaluate_baselines.py (ROC/PR/logloss/Brier/pr/recall)
ml/p3/setup_dataset/* (R.2 chain, S18A review rows) — imports only
Regression (all green; committed outputs restored byte-identical):
P3-S16 20/20 | VEC 15/15 | chain parity 694==694 | S19 audit 12/12 PASS
Feature hash unchanged : 0414e401522ea4e2 (schema frozen, no drift)
Guard : walk_forward.py + test_walk_forward.py scan CLEAN on the frozen
P3-S.4/P3-S.5 parity-absence patterns (verified 0 hits).
End of P3-S20 walk-forward baseline report. Decision: A — STABLE WEAK SIGNAL (pre-registered; weak ranking edge consistent across the three OOS folds, pooled ROC-AUC 0.579, PR-AUC beats the majority prior in every fold; calibration unchanged vs base rate — no manufacturing of a positive result).