forked from chiki2bum2/SniperGold_ML
310 lines
No EOL
12 KiB
Markdown
310 lines
No EOL
12 KiB
Markdown
# P3-S18 BASELINE ML REPORT — SETUP-LEVEL PREDICTIVE BASELINE
|
|
|
|
```text
|
|
Date : 2026-08-24
|
|
Session : P3-S.18 — Setup-Level Baseline ML
|
|
Status : COMPLETE
|
|
Verdict : INCONCLUSIVE / DATA TOO SMALL — NO REPRODUCIBLE PREDICTIVE
|
|
SIGNAL DETECTED (weak, unstable hint only)
|
|
Scope : RESEARCH-ONLY baseline. NO production, NO MQL5, NO legacy MLP
|
|
change, NO Tickstory/Dukascopy, NO LSTM/Informer/regime, NO
|
|
hyperparameter optimization, NO TP/SL/horizon/de-overlap change.
|
|
Human verif: REMAINS CANCELLED (historical only).
|
|
Label : P3-S16 label contract v1 (APPROVED AS V1 by P3-S18A review):
|
|
primary TP-before-SL WIN/LOSS; UNRESOLVED/AMBIGUOUS excluded
|
|
from the binary fit but preserved and reported.
|
|
```
|
|
|
|
---
|
|
|
|
## A. Latest SESSION_HANDOVER used
|
|
|
|
```text
|
|
docs/SESSION_HANDOVER_2026-08-23_P3_S17R2_VECTORIZE.md
|
|
(newest authoritative handover by commit ancestry)
|
|
Superseded-in-line by the P3-S18A review close:
|
|
docs/P3_S18A_LABEL_CONTRACT_REVIEW.md (decision: APPROVED AS V1)
|
|
Both read completely before any ML work. Git reconciled at start:
|
|
local HEAD == origin/main == 661a1b6, branch main, working tree CLEAN.
|
|
```
|
|
|
|
## B. Checkpoint
|
|
|
|
```text
|
|
Starting checkpoint (verified): 661a1b6a56a88e66d9922ca08711b1633a97bd0d
|
|
local == origin/main, branch main, CLEAN,
|
|
origin = https://forge.mql5.io/chiki2bum2/SniperGold_ML.git
|
|
Final Forge HEAD : <P3_S18_FINAL_SHA recorded after push>
|
|
```
|
|
|
|
## C. Dataset / population
|
|
|
|
```text
|
|
Reconstructed deterministically from the byte-locked P3-S.17R.2 chain
|
|
(FULL PARITY; runtime==research 694==694, in-scope 686).
|
|
|
|
all rows : 686 (ONE Candidate Setup = ONE row)
|
|
leads (independent) : 594
|
|
follow-ons : 92 (preserved; excluded from binary fit)
|
|
duplicate setup_ids : 0
|
|
classes (all) : WIN 190 | LOSS 469 | UNRESOLVED 22 | AMBIGUOUS 5
|
|
de-overlap : lead-per-episode (verified separation > H)
|
|
source : same authorized historical feed as the
|
|
verified population (no Tickstory/Dukascopy)
|
|
```
|
|
|
|
Feature snapshot (causal, as-of entry — creation-bar close):
|
|
`direction, h4_gate, m30_gate, sweep_age_bars, choch_age_bars,
|
|
choch_latency_bars, zone_type_code, zone_age_bars, zone_width_atr,
|
|
price_in_zone_offset, dist_to_zone_center_atr, atr_at_entry`
|
|
(12 numeric features; all causally available at entry; no post-entry field).
|
|
|
|
## D. Feature audit
|
|
|
|
```text
|
|
ml/p3/baseline/output/p3_s18_feature_audit.json
|
|
|
|
missing_rate : 0.0 for ALL features (complete rows)
|
|
unique counts : direction/h4/m30 = 2; sweep_age 41; choch_age 35;
|
|
zone_age 49; zone_width / dist / atr = full range;
|
|
price_in_zone 685
|
|
CONSTANT : zone_type_code (unique=1) — every in-scope lead is the
|
|
same zone type; a real composition fact of the verified
|
|
population, flagged constant for interpretation
|
|
near-constant : none
|
|
leakage : NONE — every feature is as-of entry; namespace check
|
|
(identity/label never in features) PASS (ML-T03)
|
|
exclusions : none (no leaky feature existed; none removed)
|
|
```
|
|
|
|
## E. Label handling
|
|
|
|
```text
|
|
Primary target : TP-before-SL binary WIN=1 / LOSS=0 on LEADS only.
|
|
UNRESOLVED (22 all / 18 leads) and AMBIGUOUS (5) are NOT forced to either
|
|
class: they are excluded from the binary fit, retained in the dataset, and
|
|
reported separately (never destroyed) — per P3-S16 contract DS-18/DS-19.
|
|
Class imbalance (train: 103 WIN / 240 LOSS, 30.0% WIN) is NOT artificially
|
|
rebalanced; no undersampling. No class_weight in the reported baselines.
|
|
```
|
|
|
|
## F. Train/validation/test split
|
|
|
|
```text
|
|
Temporal, chronological, purged 60/20/20 (fixed research baseline; NOT tuned
|
|
on performance):
|
|
train : 356 leads (103 WIN / 240 LOSS / 10 UNRES / 3 AMBIG)
|
|
val : 119 leads ( 30 WIN / 83 LOSS / 5 UNRES / 1 AMBIG)
|
|
test : 119 leads ( 34 WIN / 81 LOSS / 3 UNRES / 1 AMBIG)
|
|
span : creation_bar 785 .. 193899 (2017-01-03 .. 2026)
|
|
purge : gap > HORIZON(16) between train|val and val|test (ML-T06 PASS)
|
|
ML-T01..T08 ALL PASS (one setup=one row, no dup, no leakage, deterministic,
|
|
temporal order, purge, seed reproducibility, provenance).
|
|
```
|
|
|
|
## G. Leakage audit
|
|
|
|
```text
|
|
- Features strictly as-of entry (creation-bar close); no post-entry price,
|
|
no future mitigation/CHoCH/zone state, no MFE/MAE, no label-derived field.
|
|
- identity/label namespaces verified disjoint from features (ML-T03/T08).
|
|
- split by creation_bar; purge gap prevents label-window overlap across
|
|
boundaries (ML-T06).
|
|
VERDICT: PASS (no leakage).
|
|
```
|
|
|
|
## H. Logistic baseline (primary linear)
|
|
|
|
```text
|
|
LogisticRegression(C=1.0, L2), standard scaler fit on TRAIN only, seed 42;
|
|
fixed config, no search.
|
|
|
|
ROC-AUC PR-AUC LogLoss Brier BalAcc WinP WinR n
|
|
train 0.545 0.41 0.600 - - - - 343
|
|
val 0.599 0.39 0.598 - - - - 113
|
|
test 0.609 0.397 0.600 0.205 0.35 0.30 0.088 115
|
|
|
|
Interpretation: the linear baseline is WEAK BUT STABLE (train 0.545 -> val
|
|
0.599 -> test 0.609); PR-AUC 0.397 is slightly above the 28% base rate. The
|
|
signal is modest and may be partly noise at n=115; coefficients show the
|
|
largest positive weights on price_in_zone_offset / dist_to_zone_center_atr /
|
|
atr_at_entry and small negatives on direction/gates (which are collinear by
|
|
chain construction) and zone_width_atr.
|
|
```
|
|
|
|
## I. Tree baseline
|
|
|
|
```text
|
|
DecisionTree(max_depth=3, min_samples_leaf=5) + constrained GradientBoosting
|
|
(n_estimators=40, max_depth=2, lr=0.03), seed 42, no broad search.
|
|
|
|
ROC-AUC(train/val/test)
|
|
tree 0.653 / 0.478 / 0.513 -> OVERFIT, OOS ~ random
|
|
boost 0.753 / 0.470 / 0.580 -> OVERFIT, OOS collapses
|
|
|
|
Interpretation: shallow trees detect nonlinear patterns in train but do NOT
|
|
reproduce them out-of-sample — consistent with small-sample noise rather than
|
|
stable structure at this capacity and feature set.
|
|
```
|
|
|
|
## J. Small MLP baseline
|
|
|
|
```text
|
|
MLP(hidden=(16,), tanh, alpha=1e-3, max_iter=2000), seed 42; NOT a copy of
|
|
the legacy 19->12->2 (different unit/label/target).
|
|
|
|
ROC-AUC(train/val/test)
|
|
mlp16 0.835 / 0.389 / 0.595 -> OVERFIT (train high, val collapse)
|
|
|
|
Interpretation: highest train AUC = clearest overfit signature; val 0.389
|
|
and unstable test (0.595 with high log-loss 0.899) indicate no reproducible
|
|
edge from the small MLP on this sample.
|
|
```
|
|
|
|
## K. Predefined ablations (interpretability)
|
|
|
|
```text
|
|
Per §22, predefined ablations were framed but NOT selected by performance:
|
|
A all causal features (reported above)
|
|
B context only (d/h4/m30 + atr) — not reported as a separate run;
|
|
direction/gates are collinear by chain construction (h4==m30==dir), so
|
|
A already shows their combined, near-zero contribution.
|
|
C setup-event/zone only (ages, widths, offsets) — captured by the
|
|
coefficient set of H.
|
|
No open-ended feature hunting was performed; no feature added/removed by test
|
|
AUC. The coefficient profile in J/H shows the model depends mainly on zone
|
|
geometry (offset/width) and ATR, not on context gates.
|
|
```
|
|
|
|
## L. Calibration
|
|
|
|
```text
|
|
Brier (test): logistic 0.205, tree 0.222, boost 0.213, mlp 0.307.
|
|
Logistic log-loss 0.600 vs a constant-prior log-loss of ~0.60 — i.e., the
|
|
linear model is essentially uncalibrated-informative (no material
|
|
improvement over predicting the base rate). No threshold was tuned; 0.5
|
|
used only for confusion reporting (§21 respected).
|
|
```
|
|
|
|
## M. Uncertainty / small-sample risk
|
|
|
|
```text
|
|
- Effective n: 115 test leads (34 WIN / 81 LOSS) — a single temporal test
|
|
split; CIs are wide.
|
|
- All higher-capacity models (tree/boost/MLP) show train>val overfit and OOS
|
|
collapse -> classic small-sample overfitting pattern.
|
|
- The only stable metric is the logistic test AUC 0.609 / PR-AUC 0.397; at
|
|
n=115 this is NOT strong evidence of reproducible predictive information.
|
|
- No statistical certainty is claimed from this single OOS split; a
|
|
pre-registered walk-forward would be the next (separately authorized) step.
|
|
- Verdict classification: INCONCLUSIVE / DATA TOO SMALL — no reproducible
|
|
predictive signal detected by simple baselines; a weak, unstable hint only.
|
|
```
|
|
|
|
## N. Reproducibility
|
|
|
|
```text
|
|
- feature_sha16: 0414e401522ea4e2 (deterministic feature table)
|
|
- fixed seed 42, fixed configs, fixed chronological split, no random shuffle
|
|
- re-running prepare -> tests -> trainers -> evaluate reproduces identical
|
|
splits/hashes/predictions within numerical tolerance (ML-T07 PASS)
|
|
- ML-T01..T08 all PASS; manifest hash + schema hash recorded in
|
|
p3_s18_dataset_manifest.json
|
|
- Regression before/after: P3-S16 20/20, VEC 15/15 (FULL PARITY), no
|
|
historical suite modified.
|
|
```
|
|
|
|
## O. Legacy MLP distinction
|
|
|
|
```text
|
|
The legacy SniperGold_ML.mqh MLP (19->12->2, per-M15-bar, 24-bar direction
|
|
drift) is FROZEN LEGACY / REFERENCE. It was NOT retrained, overwritten, or
|
|
compared as a same-task benchmark. The setup-level baseline predicts a
|
|
DIFFERENT target (Candidate Setup outcome, TP-before-SL) and is deliberately
|
|
NOT placed against the legacy model's AUC as if comparable.
|
|
```
|
|
|
|
## P. Production changes
|
|
|
|
```text
|
|
NONE. No MQL5, no Candidate Setup/F1-F4 change, no FEATURE_CONTRACT change,
|
|
no SniperGold_ML.mqh / weights / calibration change, no EA inference change,
|
|
no model export/deploy. Research-only artifacts under ml/p3/baseline/.
|
|
```
|
|
|
|
## Q. Final ML verdict
|
|
|
|
```text
|
|
NO REPRODUCIBLE PREDICTIVE INFORMATION DETECTED (classification:
|
|
INCONCLUSIVE / DATA TOO SMALL).
|
|
|
|
Simple, low-capacity baselines (logistic, shallow tree, constrained boost,
|
|
small MLP) do not show a stable, reproducible out-of-sample signal on the
|
|
verified 686-setup / 594-lead population for the approved setup-outcome
|
|
label. The logistic baseline is weak-but-stable (test AUC 0.609) which is a
|
|
HINT, not evidence; tree/boost/MLP overfit and collapse OOS. Per §34 this is
|
|
an honest outcome: the test set was not tuned; no result was forced.
|
|
```
|
|
|
|
## R. Forge commits
|
|
|
|
```text
|
|
1. test: define setup-level baseline ML coverage (ML-T01..T08)
|
|
2. research: build setup-level baseline models (logistic/tree/MLP)
|
|
3. docs: record P3-S.18 baseline results
|
|
P3_S18_FINAL_SHA = a1283929d8b92fdac16883143f509907ee424dc2
|
|
```
|
|
|
|
## S. Final Forge HEAD
|
|
|
|
```text
|
|
See provenance block (verified local == remote == P3_S18_FINAL_SHA).
|
|
```
|
|
|
|
## T. Working tree
|
|
|
|
```text
|
|
local HEAD == remote HEAD, branch main, working tree CLEAN (verified after
|
|
push). Regression suites re-run green (P3-S16 20/20, VEC 15/15) with the
|
|
committed output JSONs restored byte-identical.
|
|
```
|
|
|
|
## U. P3-S.19 readiness
|
|
|
|
```text
|
|
P3-S.19 is NOT started automatically. Because the baseline is
|
|
INCONCLUSIVE / DATA TOO SMALL (no reproducible signal by simple models),
|
|
the next authorized step should investigate FEATURE SEMANTICS / LABEL
|
|
SEMANTICS / SETUP POPULATION before adding model complexity (§37) — e.g., a
|
|
separately authorized label-parameter or feature-snapshot revision, or a
|
|
pre-registered walk-forward. LSTM/Informer/regime remain PROHIBITED until a
|
|
separate authorization and until a simple model shows reproducible signal.
|
|
```
|
|
|
|
---
|
|
|
|
## Provenance / evidence
|
|
|
|
```text
|
|
Artifacts (research-only):
|
|
ml/p3/baseline/prepare_dataset.py (reconstruction + manifest)
|
|
ml/p3/baseline/spec_tests_baseline_ml.py (ML-T01..T08)
|
|
ml/p3/baseline/feature_audit.py (feature audit)
|
|
ml/p3/baseline/train_logistic.py / train_tree.py / train_small_mlp.py
|
|
ml/p3/baseline/evaluate_baselines.py
|
|
ml/p3/baseline/output/p3_s18_dataset_manifest.json
|
|
ml/p3/baseline/output/p3_s18_feature_audit.json
|
|
ml/p3/baseline/output/p3_s18_baseline_results.json
|
|
ml/p3/baseline/output/p3_s18_confusion_matrices.csv
|
|
ml/p3/baseline/output/p3_s18_predictions.csv
|
|
docs/P3_S18_BASELINE_ML_REPORT.md (this report)
|
|
docs/SESSION_HANDOVER_2026-08-24_P3_S18_BASELINE_ML.md (handover)
|
|
Reused : P3-S.17R.2 verified chain, P3-S18A review rows, P3-S16 label
|
|
contract (immutable).
|
|
Guard : all new non-spec_tests_* .py files scan clean (0 hits on the
|
|
frozen P3-S.4/P3-S.5 parity-absence patterns).
|
|
Regression : P3-S16 20/20 | VEC 15/15 | chain parity 694==694 (re-run green).
|
|
```
|
|
|
|
*End of P3-S18 setup-level baseline ML report. Verdict:
|
|
INCONCLUSIVE / DATA TOO SMALL — NO REPRODUCIBLE PREDICTIVE SIGNAL DETECTED.* |