11 KiB
P3-S21.2 — CALIBRATION DIAGNOSTICS — FINAL TECHNICAL REPORT
Date : 2026-08-25
Session : P3-S21.2 — Calibration Diagnostics
Status : COMPLETE (research-only; NO production change)
Decision : B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE
Authoritative handover used : docs/SESSION_HANDOVER_2026-08-25_P3_S21_1_FEATURE_CONTRIBUTION_AUDIT.md
Starting SHA: 45b79d524b9e1cc2760487c052413d6af6103084
Scope : probability-calibration diagnostics on the FROZEN P3-S20 logistic.
1. Primary question
Can the existing weak Logistic ranking signal be calibrated out-of-sample in a reproducible way, without changing the frozen feature, label, population, or temporal contracts?
Short answer: NO reproducible calibration improvement was found. The weak ranking signal (pooled OOS ROC-AUC 0.579) is reproduced exactly, but neither a sigmoid (Platt) calibration nor an isotonic calibration improves out-of-sample log-loss / Brier vs the uncalibrated logistic, and the constant (train-prevalence) prior remains the best probability output. The phase is classified B.
2. Frozen baseline — exact reproduction
The uncalibrated baseline is the P3-S20 walk-forward logistic, recomputed from the frozen recipe and validated against the committed artifacts:
Model : LogisticRegression(C=1.0, L2/lbfgs, max_iter=5000,
random_state=42, fit_intercept=True, class_weight=None)
Scaler : StandardScaler(z=(x-mean)/std) fit on TRAINING rows only
Features : 12 frozen causal features (feature_sha16 = 0414e401522ea4e2)
Population : 571 binary WIN/LOSS leads (167 WIN / 404 LOSS); 18 UNRESOLVED
+ 5 AMBIGUOUS preserved, never forced (as before)
Label contract : P3-S16 v1 (frozen), H=16, WIN=1 / LOSS=0
Folds : F1[300:395] F2[395:490] F3[490:571], expanding window,
purge gaps 555/472/321 bars (all >> H=16), no shuffle
REPRODUCTION GATE PASSED:
271/271 OOS rows, max_abs_prob_diff = 0.00e+00 vs committed
ml/p3/baseline/output/p3_s20_oos_predictions.csv
pooled OOS ROC-AUC recomputed = 0.5791603786527645 (== committed)
3. Calibration design (pre-registered, temporally causal)
For each fold F:
| step | detail |
|---|---|
| base fit | frozen logistic + scaler on fold-F TRAINING rows only |
| calibrator fit | on the same TRAINING rows only (all strictly before F's OOS window) — no future/OOS leakage |
| sigmoid (Platt) | penalty-free logistic: logit(p_cal) = A*logit(p_raw) + B, fit on train (score, y) |
| isotonic | IsotonicRegression(y_min=0, y_max=1, out_of_bounds="clip"), fit on train (score, y) |
| application | calibrator applied to the fold's OOS raw scores only |
| evaluation | OOS per fold and pooled; per-version log-loss, Brier, ROC/PR-AUC, calibration curve, calibration intercept/slope, bootstrap Brier delta |
Why no inner split: the frozen walk-forward gives no earlier hold-out to devote to calibration without altering the frozen base fit (which is forbidden); calibrating on the full pre-OOS training window is therefore the only design consistent with the frozen contracts.
Isotonic statistical-power caveat (declared before reading results): per-fold
OOS evaluation (n=95/95/81) is small for a non-parametric calibrator, so
isotonic OOS reliability is flagged LOW STATISTICAL POWER and is not treated
as primary evidence.
4. Results
4.1 Pooled OOS (n=271; 74 WIN / 197 LOSS)
| version | log-loss | Brier | ROC-AUC | PR-AUC |
|---|---|---|---|---|
| constant prior (fold-train prevalence, no leakage) | 0.589090 | 0.199666 | — | — |
| uncalibrated logistic (raw) | 0.589505 | 0.200347 | 0.579160 | 0.341710 |
| sigmoid (Platt) | 0.589735 | 0.200454 | 0.578817 | 0.341469 |
| isotonic | 1.010779 | 0.218080 | 0.556695 | 0.325568 |
- Sigmoid (Platt) changes almost nothing: Δlog-loss = +0.0002 (worse), ΔBrier = +0.0001 (worse). The fitted sigmoid is nearly the identity (A≈1.01-1.02, B≈0.01-0.02), which is expected because a logistic is already calibrated on its own training data; the OOS miscalibration therefore cannot be repaired by a train-fit recalibration.
- Isotonic degrades OOS calibration materially (pooled Brier +0.018 vs raw; fold-3 log-loss collapses to 1.77). The non-parametric fit on a small, imbalanced training set does not transfer. This is the predicted LOW STATISTICAL POWER outcome; it is reported, not forced into a result.
4.2 Per-fold OOS
| fold | n (WIN/LOSS) | version | log-loss | Brier |
|---|---|---|---|---|
| F1 | 95 (25/70) | raw / sigmoid / isotonic | 0.58152 / 0.58167 / 0.83640 | 0.19576 / 0.19581 / 0.19334 |
| F2 | 95 (23/72) | raw / sigmoid / isotonic | 0.54668 / 0.54650 / 0.53949 | 0.18063 / 0.18056 / 0.17745 |
| F3 | 81 (26/55) | raw / sigmoid / isotonic | 0.64910 / 0.64990 / 1.76805 | 0.22885 / 0.22924 / 0.29476 |
- Sigmoid improved fold 2 by a negligible amount (ΔBrier ≈ -0.00007) and worsened folds 1 and 3 — no fold-consistent calibration improvement.
- Direction of change is not consistent across folds for any method (pre-registered A-gate requires all 3 folds to improve on both log-loss and Brier; this fails for sigmoid and isotonic).
4.3 Reliability / calibration curve (pooled OOS)
- Raw and sigmoid curves are essentially identical: observed WIN rates stay ≈0.15-0.37 across the predicted range, and even the top decile (mean predicted ≈0.51) observes only ≈0.39 — clear overconfidence at the top of the distribution that recalibration-from-train does not fix.
- Isotonic curve is unstable (bins collapse at ≈0.29 with a discontinuous jump to ≈0.79 in the top bin) — not a usable calibration.
4.4 Calibration intercept & slope (pooled OOS, logit space)
| version | intercept | slope |
|---|---|---|
| raw | -0.5086 | 0.6623 |
| sigmoid | -0.5163 | 0.6530 |
| isotonic | -0.9735 | 0.0162 |
A well-calibrated output would have intercept ≈ 0 and slope ≈ 1. The raw logistic has slope ≈ 0.66 on OOS (probabilities too extreme relative to realized frequencies); recalibration from training data leaves this essentially unchanged (sigmoid slope 0.653).
4.5 Bootstrap uncertainty: pooled Brier delta (raw − calibrated)
| method | mean delta | SE (1000 draws) |
|---|---|---|
| sigmoid | −0.00010 | 0.00008 |
| isotonic | −0.01743 | 0.00733 |
Negative mean = calibrated is worse than raw. The sigmoid difference is within ±2 SE of zero (no improvement), isotonic is materially worse. No evidence of calibration benefit.
5. Statistical interpretation
- The ranking edge is reproduced exactly (ROC-AUC 0.579 > prior; PR-AUC 0.342 vs prior 0.273) but this is a ranking property, unaffected by monotone recalibration, and does not imply usable probability outputs.
- Probability calibration: sigmoid(Platt) provides no material or temporally consistent improvement; the difference from raw is far below the 0.005 pooled materiality floor and is not fold-consistent. Isotonic is degraded OOS and is classified LOW STATISTICAL POWER for reliable calibration inference on this sample.
- The constant-prior baseline has the LOWEST pooled log-loss (0.58909) and Brier (0.19967) of all variants — i.e., the frozen signal adds ranking value, but its probability output is not superior to a flat prior, and calibration does not change that.
- Sample limitations: OOS evaluation is n=95/95/81 per fold (271 pooled, only 74 WIN). Small per-fold samples cannot resolve calibration differences below a few percent; a material improvement cannot be positively excluded outright, but none is observed, and per §9 we do not manufacture significance.
6. Decision gate (pre-registered)
A — calibration improves reproducibly ........ NO (no all-fold, material,
pooled improvement)
B — ranking signal exists, calibration does YES (pooled OOS ROC 0.579;
not improve sigmoid no better than raw;
prior best probability output)
C — inconclusive / sample too small .......... not the primary class
(ranking + calibration
null are clear, but any
calibration *magnitude*
is small-sample-limited)
D — period / regime dependence ............... NO (fold-2 strength is the
known noise pattern, not
a calibration design issue)
E — no reproducible ranking or calibration ... NO (ranking reproduces exactly)
7. Primary classification
B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE
The weak logistic ranking signal exists and is reproducible; probability calibration provides no robust out-of-sample improvement, and the constant prior remains the best probability output. Result is reported honestly with sample-size caveats; nothing here is a production-ready calibrated probability.
8. What this phase does NOT do / authorize
- NO production deployment, NO MQL5 change, NO F1-F4 / Candidate Setup / FEATURE_CONTRACT change, NO label / TP / SL / horizon change.
- NO feature ablation / removal; NO nonlinear model escalation (tree/boost/ MLP/LSTM/Informer/regime); NO external data (Tickstory/Dukascopy); NO threshold, entry/exit, profit-factor, win-rate, or sizing optimization.
- No calibration method was selected after inspecting pooled OOS results; the sigmoid/isotonic comparison was pre-registered before any final metric was computed.
9. Artifacts
ml/p3/baseline/p3_s21_2_calibration.py research module (this phase)
ml/p3/baseline/test_p3_s21_2_calibration.py deterministic spec tests (8/8 PASS)
ml/p3/baseline/output/p3_s21_2_oos_predictions.csv per-row raw/sigmoid/isotonic probs
ml/p3/baseline/output/p3_s21_2_fold_metrics.csv per-fold metrics by version
ml/p3/baseline/output/p3_s21_2_reliability.csv pooled calibration curves
ml/p3/baseline/output/p3_s21_2_calibrators.json fitted calibrator parameters
ml/p3/baseline/output/p3_s21_2_summary.json machine-readable summary
ml/p3/baseline/output/p3_s21_2_tests.json test results (8/8 PASS)
docs/P3_S21_2_CALIBRATION_REPORT.md this report
docs/P3_S21_2_CALIBRATION_PRODUCTION_READINESS.md production-readiness assessment
docs/SESSION_HANDOVER_2026-08-25_P3_S21_2_CALIBRATION_DIAGNOSTICS.md handover
Historical P3-S20 artifacts were used read-only and are byte-identical
(verified: git diff empty for all tracked files).
End P3-S21.2 technical report. Verdict: B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE. The weak ranking edge reproduces exactly; no train-fit recalibration (sigmoid or isotonic) improves OOS probability calibration, and the constant prior remains the best probability output.