forked from chiki2bum2/SniperGold_ML
232 lines
No EOL
11 KiB
Markdown
232 lines
No EOL
11 KiB
Markdown
# P3-S21.2 — CALIBRATION DIAGNOSTICS — FINAL TECHNICAL REPORT
|
|
|
|
```text
|
|
Date : 2026-08-25
|
|
Session : P3-S21.2 — Calibration Diagnostics
|
|
Status : COMPLETE (research-only; NO production change)
|
|
Decision : B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE
|
|
Authoritative handover used : docs/SESSION_HANDOVER_2026-08-25_P3_S21_1_FEATURE_CONTRIBUTION_AUDIT.md
|
|
Starting SHA: 45b79d524b9e1cc2760487c052413d6af6103084
|
|
Scope : probability-calibration diagnostics on the FROZEN P3-S20 logistic.
|
|
```
|
|
|
|
---
|
|
|
|
## 1. Primary question
|
|
|
|
> Can the existing weak Logistic ranking signal be calibrated out-of-sample in a
|
|
> reproducible way, without changing the frozen feature, label, population, or
|
|
> temporal contracts?
|
|
|
|
**Short answer: NO reproducible calibration improvement was found.**
|
|
The weak ranking signal (pooled OOS ROC-AUC 0.579) is reproduced exactly, but
|
|
neither a sigmoid (Platt) calibration nor an isotonic calibration improves
|
|
out-of-sample log-loss / Brier vs the uncalibrated logistic, and the constant
|
|
(train-prevalence) prior remains the best probability output. The phase is
|
|
classified **B**.
|
|
|
|
---
|
|
|
|
## 2. Frozen baseline — exact reproduction
|
|
|
|
The uncalibrated baseline is the P3-S20 walk-forward logistic, recomputed from
|
|
the frozen recipe and validated against the committed artifacts:
|
|
|
|
```text
|
|
Model : LogisticRegression(C=1.0, L2/lbfgs, max_iter=5000,
|
|
random_state=42, fit_intercept=True, class_weight=None)
|
|
Scaler : StandardScaler(z=(x-mean)/std) fit on TRAINING rows only
|
|
Features : 12 frozen causal features (feature_sha16 = 0414e401522ea4e2)
|
|
Population : 571 binary WIN/LOSS leads (167 WIN / 404 LOSS); 18 UNRESOLVED
|
|
+ 5 AMBIGUOUS preserved, never forced (as before)
|
|
Label contract : P3-S16 v1 (frozen), H=16, WIN=1 / LOSS=0
|
|
Folds : F1[300:395] F2[395:490] F3[490:571], expanding window,
|
|
purge gaps 555/472/321 bars (all >> H=16), no shuffle
|
|
```
|
|
|
|
```text
|
|
REPRODUCTION GATE PASSED:
|
|
271/271 OOS rows, max_abs_prob_diff = 0.00e+00 vs committed
|
|
ml/p3/baseline/output/p3_s20_oos_predictions.csv
|
|
pooled OOS ROC-AUC recomputed = 0.5791603786527645 (== committed)
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Calibration design (pre-registered, temporally causal)
|
|
|
|
For each fold F:
|
|
|
|
| step | detail |
|
|
|---|---|
|
|
| base fit | frozen logistic + scaler on fold-F TRAINING rows only |
|
|
| calibrator fit | on the **same TRAINING rows only** (all strictly before F's OOS window) — no future/OOS leakage |
|
|
| sigmoid (Platt) | penalty-free logistic: `logit(p_cal) = A*logit(p_raw) + B`, fit on train `(score, y)` |
|
|
| isotonic | `IsotonicRegression(y_min=0, y_max=1, out_of_bounds="clip")`, fit on train `(score, y)` |
|
|
| application | calibrator applied to the fold's OOS raw scores only |
|
|
| evaluation | OOS per fold and pooled; per-version log-loss, Brier, ROC/PR-AUC, calibration curve, calibration intercept/slope, bootstrap Brier delta |
|
|
|
|
Why no inner split: the frozen walk-forward gives no earlier hold-out to devote
|
|
to calibration without altering the frozen base fit (which is forbidden);
|
|
calibrating on the full pre-OOS training window is therefore the only design
|
|
consistent with the frozen contracts.
|
|
|
|
Isotonic statistical-power caveat (declared before reading results): per-fold
|
|
OOS evaluation (n=95/95/81) is small for a non-parametric calibrator, so
|
|
isotonic OOS reliability is flagged `LOW STATISTICAL POWER` and is not treated
|
|
as primary evidence.
|
|
|
|
---
|
|
|
|
## 4. Results
|
|
|
|
### 4.1 Pooled OOS (n=271; 74 WIN / 197 LOSS)
|
|
|
|
| version | log-loss | Brier | ROC-AUC | PR-AUC |
|
|
|---|---|---|---|---|
|
|
| constant prior (fold-train prevalence, no leakage) | 0.589090 | 0.199666 | — | — |
|
|
| **uncalibrated logistic (raw)** | **0.589505** | **0.200347** | **0.579160** | **0.341710** |
|
|
| sigmoid (Platt) | 0.589735 | 0.200454 | 0.578817 | 0.341469 |
|
|
| isotonic | 1.010779 | 0.218080 | 0.556695 | 0.325568 |
|
|
|
|
- Sigmoid (Platt) changes almost nothing: Δlog-loss = +0.0002 (worse), ΔBrier = +0.0001 (worse). The fitted sigmoid is nearly the identity (A≈1.01-1.02, B≈0.01-0.02), which is expected because a logistic is already calibrated on its own training data; the OOS miscalibration therefore cannot be repaired by a train-fit recalibration.
|
|
- Isotonic **degrades** OOS calibration materially (pooled Brier +0.018 vs raw; fold-3 log-loss collapses to 1.77). The non-parametric fit on a small, imbalanced training set does not transfer. This is the predicted LOW STATISTICAL POWER outcome; it is reported, not forced into a result.
|
|
|
|
### 4.2 Per-fold OOS
|
|
|
|
| fold | n (WIN/LOSS) | version | log-loss | Brier |
|
|
|---|---|---|---|---|
|
|
| F1 | 95 (25/70) | raw / sigmoid / isotonic | 0.58152 / 0.58167 / 0.83640 | 0.19576 / 0.19581 / 0.19334 |
|
|
| F2 | 95 (23/72) | raw / sigmoid / isotonic | 0.54668 / 0.54650 / 0.53949 | 0.18063 / 0.18056 / 0.17745 |
|
|
| F3 | 81 (26/55) | raw / sigmoid / isotonic | 0.64910 / 0.64990 / 1.76805 | 0.22885 / 0.22924 / 0.29476 |
|
|
|
|
- Sigmoid improved fold 2 by a negligible amount (ΔBrier ≈ -0.00007) and worsened folds 1 and 3 — no fold-consistent calibration improvement.
|
|
- Direction of change is **not consistent across folds** for any method (pre-registered A-gate requires all 3 folds to improve on both log-loss and Brier; this fails for sigmoid and isotonic).
|
|
|
|
### 4.3 Reliability / calibration curve (pooled OOS)
|
|
|
|
- Raw and sigmoid curves are essentially identical: observed WIN rates stay
|
|
≈0.15-0.37 across the predicted range, and even the top decile
|
|
(mean predicted ≈0.51) observes only ≈0.39 — clear overconfidence at the
|
|
top of the distribution that recalibration-from-train does not fix.
|
|
- Isotonic curve is unstable (bins collapse at ≈0.29 with a discontinuous jump
|
|
to ≈0.79 in the top bin) — not a usable calibration.
|
|
|
|
### 4.4 Calibration intercept & slope (pooled OOS, logit space)
|
|
|
|
| version | intercept | slope |
|
|
|---|---|---|
|
|
| raw | -0.5086 | 0.6623 |
|
|
| sigmoid | -0.5163 | 0.6530 |
|
|
| isotonic | -0.9735 | 0.0162 |
|
|
|
|
A well-calibrated output would have intercept ≈ 0 and slope ≈ 1. The raw
|
|
logistic has slope ≈ 0.66 on OOS (probabilities too extreme relative to
|
|
realized frequencies); recalibration from training data leaves this essentially
|
|
unchanged (sigmoid slope 0.653).
|
|
|
|
### 4.5 Bootstrap uncertainty: pooled Brier delta (raw − calibrated)
|
|
|
|
| method | mean delta | SE (1000 draws) |
|
|
|---|---|---|
|
|
| sigmoid | −0.00010 | 0.00008 |
|
|
| isotonic | −0.01743 | 0.00733 |
|
|
|
|
Negative mean = calibrated is *worse* than raw. The sigmoid difference is
|
|
within ±2 SE of zero (no improvement), isotonic is materially worse. No
|
|
evidence of calibration benefit.
|
|
|
|
---
|
|
|
|
## 5. Statistical interpretation
|
|
|
|
- The ranking edge is reproduced exactly (ROC-AUC 0.579 > prior; PR-AUC 0.342
|
|
vs prior 0.273) but this is a **ranking** property, unaffected by monotone
|
|
recalibration, and does not imply usable probability outputs.
|
|
- Probability calibration: sigmoid(Platt) provides **no material or
|
|
temporally consistent improvement**; the difference from raw is far below
|
|
the 0.005 pooled materiality floor and is not fold-consistent. Isotonic is
|
|
degraded OOS and is classified **LOW STATISTICAL POWER** for reliable
|
|
calibration inference on this sample.
|
|
- The constant-prior baseline has the LOWEST pooled log-loss (0.58909) and
|
|
Brier (0.19967) of all variants — i.e., the frozen signal adds ranking
|
|
value, but its probability output is not superior to a flat prior, and
|
|
calibration does not change that.
|
|
- Sample limitations: OOS evaluation is n=95/95/81 per fold (271 pooled, only
|
|
74 WIN). Small per-fold samples cannot resolve calibration differences
|
|
below a few percent; a material improvement cannot be positively excluded
|
|
outright, but none is observed, and per §9 we do not manufacture
|
|
significance.
|
|
|
|
---
|
|
|
|
## 6. Decision gate (pre-registered)
|
|
|
|
```text
|
|
A — calibration improves reproducibly ........ NO (no all-fold, material,
|
|
pooled improvement)
|
|
B — ranking signal exists, calibration does YES (pooled OOS ROC 0.579;
|
|
not improve sigmoid no better than raw;
|
|
prior best probability output)
|
|
C — inconclusive / sample too small .......... not the primary class
|
|
(ranking + calibration
|
|
null are clear, but any
|
|
calibration *magnitude*
|
|
is small-sample-limited)
|
|
D — period / regime dependence ............... NO (fold-2 strength is the
|
|
known noise pattern, not
|
|
a calibration design issue)
|
|
E — no reproducible ranking or calibration ... NO (ranking reproduces exactly)
|
|
```
|
|
|
|
## 7. Primary classification
|
|
|
|
```text
|
|
B — RANKING SIGNAL EXISTS, BUT CALIBRATION DOES NOT IMPROVE
|
|
```
|
|
|
|
The weak logistic ranking signal exists and is reproducible; probability
|
|
calibration provides no robust out-of-sample improvement, and the constant
|
|
prior remains the best probability output. Result is reported honestly with
|
|
sample-size caveats; nothing here is a production-ready calibrated probability.
|
|
|
|
---
|
|
|
|
## 8. What this phase does NOT do / authorize
|
|
|
|
- NO production deployment, NO MQL5 change, NO F1-F4 / Candidate Setup /
|
|
FEATURE_CONTRACT change, NO label / TP / SL / horizon change.
|
|
- NO feature ablation / removal; NO nonlinear model escalation (tree/boost/
|
|
MLP/LSTM/Informer/regime); NO external data (Tickstory/Dukascopy); NO
|
|
threshold, entry/exit, profit-factor, win-rate, or sizing optimization.
|
|
- No calibration method was selected after inspecting pooled OOS results; the
|
|
sigmoid/isotonic comparison was pre-registered before any final metric was
|
|
computed.
|
|
|
|
---
|
|
|
|
## 9. Artifacts
|
|
|
|
```text
|
|
ml/p3/baseline/p3_s21_2_calibration.py research module (this phase)
|
|
ml/p3/baseline/test_p3_s21_2_calibration.py deterministic spec tests (8/8 PASS)
|
|
ml/p3/baseline/output/p3_s21_2_oos_predictions.csv per-row raw/sigmoid/isotonic probs
|
|
ml/p3/baseline/output/p3_s21_2_fold_metrics.csv per-fold metrics by version
|
|
ml/p3/baseline/output/p3_s21_2_reliability.csv pooled calibration curves
|
|
ml/p3/baseline/output/p3_s21_2_calibrators.json fitted calibrator parameters
|
|
ml/p3/baseline/output/p3_s21_2_summary.json machine-readable summary
|
|
ml/p3/baseline/output/p3_s21_2_tests.json test results (8/8 PASS)
|
|
docs/P3_S21_2_CALIBRATION_REPORT.md this report
|
|
docs/P3_S21_2_CALIBRATION_PRODUCTION_READINESS.md production-readiness assessment
|
|
docs/SESSION_HANDOVER_2026-08-25_P3_S21_2_CALIBRATION_DIAGNOSTICS.md handover
|
|
```
|
|
|
|
Historical P3-S20 artifacts were used read-only and are byte-identical
|
|
(verified: `git diff` empty for all tracked files).
|
|
|
|
---
|
|
|
|
*End P3-S21.2 technical report. Verdict: B — RANKING SIGNAL EXISTS, BUT
|
|
CALIBRATION DOES NOT IMPROVE. The weak ranking edge reproduces exactly; no
|
|
train-fit recalibration (sigmoid or isotonic) improves OOS probability
|
|
calibration, and the constant prior remains the best probability output.* |