# P3-S21.2 — CALIBRATION DIAGNOSTICS — FINAL TECHNICAL REPORT ```text Date : 2026-08-25 Session : P3-S21.2 — Calibration Diagnostics Status : COMPLETE (research-only; NO production change) Decision : B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE Authoritative handover used : docs/SESSION_HANDOVER_2026-08-25_P3_S21_1_FEATURE_CONTRIBUTION_AUDIT.md Starting SHA: 45b79d524b9e1cc2760487c052413d6af6103084 Scope : probability-calibration diagnostics on the FROZEN P3-S20 logistic. ``` --- ## 1. Primary question > Can the existing weak Logistic ranking signal be calibrated out-of-sample in a > reproducible way, without changing the frozen feature, label, population, or > temporal contracts? **Short answer: NO reproducible calibration improvement was found.** The weak ranking signal (pooled OOS ROC-AUC 0.579) is reproduced exactly, but neither a sigmoid (Platt) calibration nor an isotonic calibration improves out-of-sample log-loss / Brier vs the uncalibrated logistic, and the constant (train-prevalence) prior remains the best probability output. The phase is classified **B**. --- ## 2. Frozen baseline — exact reproduction The uncalibrated baseline is the P3-S20 walk-forward logistic, recomputed from the frozen recipe and validated against the committed artifacts: ```text Model : LogisticRegression(C=1.0, L2/lbfgs, max_iter=5000, random_state=42, fit_intercept=True, class_weight=None) Scaler : StandardScaler(z=(x-mean)/std) fit on TRAINING rows only Features : 12 frozen causal features (feature_sha16 = 0414e401522ea4e2) Population : 571 binary WIN/LOSS leads (167 WIN / 404 LOSS); 18 UNRESOLVED + 5 AMBIGUOUS preserved, never forced (as before) Label contract : P3-S16 v1 (frozen), H=16, WIN=1 / LOSS=0 Folds : F1[300:395] F2[395:490] F3[490:571], expanding window, purge gaps 555/472/321 bars (all >> H=16), no shuffle ``` ```text REPRODUCTION GATE PASSED: 271/271 OOS rows, max_abs_prob_diff = 0.00e+00 vs committed ml/p3/baseline/output/p3_s20_oos_predictions.csv pooled OOS ROC-AUC recomputed = 0.5791603786527645 (== committed) ``` --- ## 3. Calibration design (pre-registered, temporally causal) For each fold F: | step | detail | |---|---| | base fit | frozen logistic + scaler on fold-F TRAINING rows only | | calibrator fit | on the **same TRAINING rows only** (all strictly before F's OOS window) — no future/OOS leakage | | sigmoid (Platt) | penalty-free logistic: `logit(p_cal) = A*logit(p_raw) + B`, fit on train `(score, y)` | | isotonic | `IsotonicRegression(y_min=0, y_max=1, out_of_bounds="clip")`, fit on train `(score, y)` | | application | calibrator applied to the fold's OOS raw scores only | | evaluation | OOS per fold and pooled; per-version log-loss, Brier, ROC/PR-AUC, calibration curve, calibration intercept/slope, bootstrap Brier delta | Why no inner split: the frozen walk-forward gives no earlier hold-out to devote to calibration without altering the frozen base fit (which is forbidden); calibrating on the full pre-OOS training window is therefore the only design consistent with the frozen contracts. Isotonic statistical-power caveat (declared before reading results): per-fold OOS evaluation (n=95/95/81) is small for a non-parametric calibrator, so isotonic OOS reliability is flagged `LOW STATISTICAL POWER` and is not treated as primary evidence. --- ## 4. Results ### 4.1 Pooled OOS (n=271; 74 WIN / 197 LOSS) | version | log-loss | Brier | ROC-AUC | PR-AUC | |---|---|---|---|---| | constant prior (fold-train prevalence, no leakage) | 0.589090 | 0.199666 | — | — | | **uncalibrated logistic (raw)** | **0.589505** | **0.200347** | **0.579160** | **0.341710** | | sigmoid (Platt) | 0.589735 | 0.200454 | 0.578817 | 0.341469 | | isotonic | 1.010779 | 0.218080 | 0.556695 | 0.325568 | - Sigmoid (Platt) changes almost nothing: Δlog-loss = +0.0002 (worse), ΔBrier = +0.0001 (worse). The fitted sigmoid is nearly the identity (A≈1.01-1.02, B≈0.01-0.02), which is expected because a logistic is already calibrated on its own training data; the OOS miscalibration therefore cannot be repaired by a train-fit recalibration. - Isotonic **degrades** OOS calibration materially (pooled Brier +0.018 vs raw; fold-3 log-loss collapses to 1.77). The non-parametric fit on a small, imbalanced training set does not transfer. This is the predicted LOW STATISTICAL POWER outcome; it is reported, not forced into a result. ### 4.2 Per-fold OOS | fold | n (WIN/LOSS) | version | log-loss | Brier | |---|---|---|---|---| | F1 | 95 (25/70) | raw / sigmoid / isotonic | 0.58152 / 0.58167 / 0.83640 | 0.19576 / 0.19581 / 0.19334 | | F2 | 95 (23/72) | raw / sigmoid / isotonic | 0.54668 / 0.54650 / 0.53949 | 0.18063 / 0.18056 / 0.17745 | | F3 | 81 (26/55) | raw / sigmoid / isotonic | 0.64910 / 0.64990 / 1.76805 | 0.22885 / 0.22924 / 0.29476 | - Sigmoid improved fold 2 by a negligible amount (ΔBrier ≈ -0.00007) and worsened folds 1 and 3 — no fold-consistent calibration improvement. - Direction of change is **not consistent across folds** for any method (pre-registered A-gate requires all 3 folds to improve on both log-loss and Brier; this fails for sigmoid and isotonic). ### 4.3 Reliability / calibration curve (pooled OOS) - Raw and sigmoid curves are essentially identical: observed WIN rates stay ≈0.15-0.37 across the predicted range, and even the top decile (mean predicted ≈0.51) observes only ≈0.39 — clear overconfidence at the top of the distribution that recalibration-from-train does not fix. - Isotonic curve is unstable (bins collapse at ≈0.29 with a discontinuous jump to ≈0.79 in the top bin) — not a usable calibration. ### 4.4 Calibration intercept & slope (pooled OOS, logit space) | version | intercept | slope | |---|---|---| | raw | -0.5086 | 0.6623 | | sigmoid | -0.5163 | 0.6530 | | isotonic | -0.9735 | 0.0162 | A well-calibrated output would have intercept ≈ 0 and slope ≈ 1. The raw logistic has slope ≈ 0.66 on OOS (probabilities too extreme relative to realized frequencies); recalibration from training data leaves this essentially unchanged (sigmoid slope 0.653). ### 4.5 Bootstrap uncertainty: pooled Brier delta (raw − calibrated) | method | mean delta | SE (1000 draws) | |---|---|---| | sigmoid | −0.00010 | 0.00008 | | isotonic | −0.01743 | 0.00733 | Negative mean = calibrated is *worse* than raw. The sigmoid difference is within ±2 SE of zero (no improvement), isotonic is materially worse. No evidence of calibration benefit. --- ## 5. Statistical interpretation - The ranking edge is reproduced exactly (ROC-AUC 0.579 > prior; PR-AUC 0.342 vs prior 0.273) but this is a **ranking** property, unaffected by monotone recalibration, and does not imply usable probability outputs. - Probability calibration: sigmoid(Platt) provides **no material or temporally consistent improvement**; the difference from raw is far below the 0.005 pooled materiality floor and is not fold-consistent. Isotonic is degraded OOS and is classified **LOW STATISTICAL POWER** for reliable calibration inference on this sample. - The constant-prior baseline has the LOWEST pooled log-loss (0.58909) and Brier (0.19967) of all variants — i.e., the frozen signal adds ranking value, but its probability output is not superior to a flat prior, and calibration does not change that. - Sample limitations: OOS evaluation is n=95/95/81 per fold (271 pooled, only 74 WIN). Small per-fold samples cannot resolve calibration differences below a few percent; a material improvement cannot be positively excluded outright, but none is observed, and per §9 we do not manufacture significance. --- ## 6. Decision gate (pre-registered) ```text A — calibration improves reproducibly ........ NO (no all-fold, material, pooled improvement) B — ranking signal exists, calibration does YES (pooled OOS ROC 0.579; not improve sigmoid no better than raw; prior best probability output) C — inconclusive / sample too small .......... not the primary class (ranking + calibration null are clear, but any calibration *magnitude* is small-sample-limited) D — period / regime dependence ............... NO (fold-2 strength is the known noise pattern, not a calibration design issue) E — no reproducible ranking or calibration ... NO (ranking reproduces exactly) ``` ## 7. Primary classification ```text B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE ``` The weak logistic ranking signal exists and is reproducible; probability calibration provides no robust out-of-sample improvement, and the constant prior remains the best probability output. Result is reported honestly with sample-size caveats; nothing here is a production-ready calibrated probability. --- ## 8. What this phase does NOT do / authorize - NO production deployment, NO MQL5 change, NO F1-F4 / Candidate Setup / FEATURE_CONTRACT change, NO label / TP / SL / horizon change. - NO feature ablation / removal; NO nonlinear model escalation (tree/boost/ MLP/LSTM/Informer/regime); NO external data (Tickstory/Dukascopy); NO threshold, entry/exit, profit-factor, win-rate, or sizing optimization. - No calibration method was selected after inspecting pooled OOS results; the sigmoid/isotonic comparison was pre-registered before any final metric was computed. --- ## 9. Artifacts ```text ml/p3/baseline/p3_s21_2_calibration.py research module (this phase) ml/p3/baseline/test_p3_s21_2_calibration.py deterministic spec tests (8/8 PASS) ml/p3/baseline/output/p3_s21_2_oos_predictions.csv per-row raw/sigmoid/isotonic probs ml/p3/baseline/output/p3_s21_2_fold_metrics.csv per-fold metrics by version ml/p3/baseline/output/p3_s21_2_reliability.csv pooled calibration curves ml/p3/baseline/output/p3_s21_2_calibrators.json fitted calibrator parameters ml/p3/baseline/output/p3_s21_2_summary.json machine-readable summary ml/p3/baseline/output/p3_s21_2_tests.json test results (8/8 PASS) docs/P3_S21_2_CALIBRATION_REPORT.md this report docs/P3_S21_2_CALIBRATION_PRODUCTION_READINESS.md production-readiness assessment docs/SESSION_HANDOVER_2026-08-25_P3_S21_2_CALIBRATION_DIAGNOSTICS.md handover ``` Historical P3-S20 artifacts were used read-only and are byte-identical (verified: `git diff` empty for all tracked files). --- *End P3-S21.2 technical report. Verdict: B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE. The weak ranking edge reproduces exactly; no train-fit recalibration (sigmoid or isotonic) improves OOS probability calibration, and the constant prior remains the best probability output.*