SniperGold_ML/docs/P3_S21_2_CALIBRATION_REPORT.md

# P3-S21.2 — CALIBRATION DIAGNOSTICS — FINAL TECHNICAL REPORT

```text
Date        : 2026-08-25
Session     : P3-S21.2 — Calibration Diagnostics
Status      : COMPLETE (research-only; NO production change)
Decision    : B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE
Authoritative handover used : docs/SESSION_HANDOVER_2026-08-25_P3_S21_1_FEATURE_CONTRIBUTION_AUDIT.md
Starting SHA: 45b79d524b9e1cc2760487c052413d6af6103084
Scope       : probability-calibration diagnostics on the FROZEN P3-S20 logistic.
```

---

## 1. Primary question

> Can the existing weak Logistic ranking signal be calibrated out-of-sample in a
> reproducible way, without changing the frozen feature, label, population, or
> temporal contracts?

**Short answer: NO reproducible calibration improvement was found.**
The weak ranking signal (pooled OOS ROC-AUC 0.579) is reproduced exactly, but
neither a sigmoid (Platt) calibration nor an isotonic calibration improves
out-of-sample log-loss / Brier vs the uncalibrated logistic, and the constant
(train-prevalence) prior remains the best probability output. The phase is
classified **B**.

---

## 2. Frozen baseline — exact reproduction

The uncalibrated baseline is the P3-S20 walk-forward logistic, recomputed from
the frozen recipe and validated against the committed artifacts:

```text
Model          : LogisticRegression(C=1.0, L2/lbfgs, max_iter=5000,
                 random_state=42, fit_intercept=True, class_weight=None)
Scaler         : StandardScaler(z=(x-mean)/std) fit on TRAINING rows only
Features       : 12 frozen causal features (feature_sha16 = 0414e401522ea4e2)
Population     : 571 binary WIN/LOSS leads (167 WIN / 404 LOSS); 18 UNRESOLVED
                 + 5 AMBIGUOUS preserved, never forced (as before)
Label contract : P3-S16 v1 (frozen), H=16, WIN=1 / LOSS=0
Folds          : F1[300:395] F2[395:490] F3[490:571], expanding window,
                 purge gaps 555/472/321 bars (all >> H=16), no shuffle
```

```text
REPRODUCTION GATE PASSED:
  271/271 OOS rows, max_abs_prob_diff = 0.00e+00 vs committed
  ml/p3/baseline/output/p3_s20_oos_predictions.csv
  pooled OOS ROC-AUC recomputed = 0.5791603786527645 (== committed)
```

---

## 3. Calibration design (pre-registered, temporally causal)

For each fold F:

| step | detail |
|---|---|
| base fit | frozen logistic + scaler on fold-F TRAINING rows only |
| calibrator fit | on the **same TRAINING rows only** (all strictly before F's OOS window) — no future/OOS leakage |
| sigmoid (Platt) | penalty-free logistic: `logit(p_cal) = A*logit(p_raw) + B`, fit on train `(score, y)` |
| isotonic | `IsotonicRegression(y_min=0, y_max=1, out_of_bounds="clip")`, fit on train `(score, y)` |
| application | calibrator applied to the fold's OOS raw scores only |
| evaluation | OOS per fold and pooled; per-version log-loss, Brier, ROC/PR-AUC, calibration curve, calibration intercept/slope, bootstrap Brier delta |

Why no inner split: the frozen walk-forward gives no earlier hold-out to devote
to calibration without altering the frozen base fit (which is forbidden);
calibrating on the full pre-OOS training window is therefore the only design
consistent with the frozen contracts.

Isotonic statistical-power caveat (declared before reading results): per-fold
OOS evaluation (n=95/95/81) is small for a non-parametric calibrator, so
isotonic OOS reliability is flagged `LOW STATISTICAL POWER` and is not treated
as primary evidence.

---

## 4. Results

### 4.1 Pooled OOS (n=271; 74 WIN / 197 LOSS)

| version | log-loss | Brier | ROC-AUC | PR-AUC |
|---|---|---|---|---|
| constant prior (fold-train prevalence, no leakage) | 0.589090 | 0.199666 | — | — |
| **uncalibrated logistic (raw)** | **0.589505** | **0.200347** | **0.579160** | **0.341710** |
| sigmoid (Platt) | 0.589735 | 0.200454 | 0.578817 | 0.341469 |
| isotonic | 1.010779 | 0.218080 | 0.556695 | 0.325568 |

- Sigmoid (Platt) changes almost nothing: Δlog-loss = +0.0002 (worse), ΔBrier = +0.0001 (worse). The fitted sigmoid is nearly the identity (A≈1.01-1.02, B≈0.01-0.02), which is expected because a logistic is already calibrated on its own training data; the OOS miscalibration therefore cannot be repaired by a train-fit recalibration.
- Isotonic **degrades** OOS calibration materially (pooled Brier +0.018 vs raw; fold-3 log-loss collapses to 1.77). The non-parametric fit on a small, imbalanced training set does not transfer. This is the predicted LOW STATISTICAL POWER outcome; it is reported, not forced into a result.

### 4.2 Per-fold OOS

| fold | n (WIN/LOSS) | version | log-loss | Brier |
|---|---|---|---|---|
| F1 | 95 (25/70) | raw / sigmoid / isotonic | 0.58152 / 0.58167 / 0.83640 | 0.19576 / 0.19581 / 0.19334 |
| F2 | 95 (23/72) | raw / sigmoid / isotonic | 0.54668 / 0.54650 / 0.53949 | 0.18063 / 0.18056 / 0.17745 |
| F3 | 81 (26/55) | raw / sigmoid / isotonic | 0.64910 / 0.64990 / 1.76805 | 0.22885 / 0.22924 / 0.29476 |

- Sigmoid improved fold 2 by a negligible amount (ΔBrier ≈ -0.00007) and worsened folds 1 and 3 — no fold-consistent calibration improvement.
- Direction of change is **not consistent across folds** for any method (pre-registered A-gate requires all 3 folds to improve on both log-loss and Brier; this fails for sigmoid and isotonic).

### 4.3 Reliability / calibration curve (pooled OOS)

- Raw and sigmoid curves are essentially identical: observed WIN rates stay
  ≈0.15-0.37 across the predicted range, and even the top decile
  (mean predicted ≈0.51) observes only ≈0.39 — clear overconfidence at the
  top of the distribution that recalibration-from-train does not fix.
- Isotonic curve is unstable (bins collapse at ≈0.29 with a discontinuous jump
  to ≈0.79 in the top bin) — not a usable calibration.

### 4.4 Calibration intercept & slope (pooled OOS, logit space)

| version | intercept | slope |
|---|---|---|
| raw | -0.5086 | 0.6623 |
| sigmoid | -0.5163 | 0.6530 |
| isotonic | -0.9735 | 0.0162 |

A well-calibrated output would have intercept ≈ 0 and slope ≈ 1. The raw
logistic has slope ≈ 0.66 on OOS (probabilities too extreme relative to
realized frequencies); recalibration from training data leaves this essentially
unchanged (sigmoid slope 0.653).

### 4.5 Bootstrap uncertainty: pooled Brier delta (raw − calibrated)

| method | mean delta | SE (1000 draws) |
|---|---|---|
| sigmoid | −0.00010 | 0.00008 |
| isotonic | −0.01743 | 0.00733 |

Negative mean = calibrated is *worse* than raw. The sigmoid difference is
within ±2 SE of zero (no improvement), isotonic is materially worse. No
evidence of calibration benefit.

---

## 5. Statistical interpretation

- The ranking edge is reproduced exactly (ROC-AUC 0.579 > prior; PR-AUC 0.342
  vs prior 0.273) but this is a **ranking** property, unaffected by monotone
  recalibration, and does not imply usable probability outputs.
- Probability calibration: sigmoid(Platt) provides **no material or
  temporally consistent improvement**; the difference from raw is far below
  the 0.005 pooled materiality floor and is not fold-consistent. Isotonic is
  degraded OOS and is classified **LOW STATISTICAL POWER** for reliable
  calibration inference on this sample.
- The constant-prior baseline has the LOWEST pooled log-loss (0.58909) and
  Brier (0.19967) of all variants — i.e., the frozen signal adds ranking
  value, but its probability output is not superior to a flat prior, and
  calibration does not change that.
- Sample limitations: OOS evaluation is n=95/95/81 per fold (271 pooled, only
  74 WIN). Small per-fold samples cannot resolve calibration differences
  below a few percent; a material improvement cannot be positively excluded
  outright, but none is observed, and per §9 we do not manufacture
  significance.

---

## 6. Decision gate (pre-registered)

```text
A — calibration improves reproducibly ........ NO  (no all-fold, material,
                                                     pooled improvement)
B — ranking signal exists, calibration does  YES (pooled OOS ROC 0.579;
    not improve                                  sigmoid no better than raw;
                                                 prior best probability output)
C — inconclusive / sample too small .......... not the primary class
                                                     (ranking + calibration
                                                      null are clear, but any
                                                      calibration *magnitude*
                                                      is small-sample-limited)
D — period / regime dependence ............... NO  (fold-2 strength is the
                                                     known noise pattern, not
                                                     a calibration design issue)
E — no reproducible ranking or calibration ... NO  (ranking reproduces exactly)
```

## 7. Primary classification

```text
B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE
```

The weak logistic ranking signal exists and is reproducible; probability
calibration provides no robust out-of-sample improvement, and the constant
prior remains the best probability output. Result is reported honestly with
sample-size caveats; nothing here is a production-ready calibrated probability.

---

## 8. What this phase does NOT do / authorize

- NO production deployment, NO MQL5 change, NO F1-F4 / Candidate Setup /
  FEATURE_CONTRACT change, NO label / TP / SL / horizon change.
- NO feature ablation / removal; NO nonlinear model escalation (tree/boost/
  MLP/LSTM/Informer/regime); NO external data (Tickstory/Dukascopy); NO
  threshold, entry/exit, profit-factor, win-rate, or sizing optimization.
- No calibration method was selected after inspecting pooled OOS results; the
  sigmoid/isotonic comparison was pre-registered before any final metric was
  computed.

---

## 9. Artifacts

```text
ml/p3/baseline/p3_s21_2_calibration.py          research module (this phase)
ml/p3/baseline/test_p3_s21_2_calibration.py     deterministic spec tests (8/8 PASS)
ml/p3/baseline/output/p3_s21_2_oos_predictions.csv   per-row raw/sigmoid/isotonic probs
ml/p3/baseline/output/p3_s21_2_fold_metrics.csv      per-fold metrics by version
ml/p3/baseline/output/p3_s21_2_reliability.csv       pooled calibration curves
ml/p3/baseline/output/p3_s21_2_calibrators.json      fitted calibrator parameters
ml/p3/baseline/output/p3_s21_2_summary.json          machine-readable summary
ml/p3/baseline/output/p3_s21_2_tests.json            test results (8/8 PASS)
docs/P3_S21_2_CALIBRATION_REPORT.md                  this report
docs/P3_S21_2_CALIBRATION_PRODUCTION_READINESS.md    production-readiness assessment
docs/SESSION_HANDOVER_2026-08-25_P3_S21_2_CALIBRATION_DIAGNOSTICS.md  handover
```

Historical P3-S20 artifacts were used read-only and are byte-identical
(verified: `git diff` empty for all tracked files).

---

*End P3-S21.2 technical report. Verdict: B — RANKING SIGNAL EXISTS, BUT
CALIBRATION DOES NOT IMPROVE. The weak ranking edge reproduces exactly; no
train-fit recalibration (sigmoid or isotonic) improves OOS probability
calibration, and the constant prior remains the best probability output.*