SniperGold_ML/docs/P3_S23_SILENT_BUG_VERIFICATION.md

# P3-S23 — SILENT-BUG VERIFICATION

```text
Date       : 2026-08-26
Session    : P3-S23 — corrected-population pre-registered ML baseline
Policy     : docs/CODE_VERIFICATION_AND_SILENT_BUG_POLICY_v1.md
Result     : NO SILENT BUG FOUND — 8/8 mutation tests detected, independent
             metric oracle matches (~1e-16), two-run byte reproducibility.
```

## 1. Scope

Applies the code-verification / silent-bug policy to the new P3-S23
pre-registered experiment. Mutations operate on disposable in-memory copies
only; the invariant/real pipeline is never broken. Detection uses independent
invariants/oracles, not "the training code re-prints its own number".

## 2. Mutation battery (8/8)

| # | Mutation | Detection signal | Detected |
|---|----------|------------------|----------|
| 1 | future feature injection (later rows leak into earlier rows) | ΔROC-AUC 0.3790 | YES |
| 2 | global scaler (fit on train+OOS) | ΔROC-AUC 0.0050 | YES |
| 3 | temporal shuffle (order broken) | order not preserved | YES |
| 4 | OOS injected into training | ΔROC-AUC 0.0245 | YES |
| 5 | row shift (feature/label misalign) | ΔROC-AUC 0.1087 | YES |
| 6 | train/OOS boundary violation (purge < H) | boundary flag | YES |
| 7 | label alignment shift (labels rolled) | ΔROC-AUC 0.1477 | YES |
| 8 | prediction/row alignment mismatch | 79 rows changed | YES |

detection_count : 8/8 ; all_detected : YES

## 3. Independent (from-scratch) metric oracle

The pooled evaluation metrics were recomputed by a disposable oracle that
implements ROC-AUC (rank/tie Mann-Whitney-U), PR-AUC (average-precision
step-integral matching average_precision_score), LogLoss, and Brier manually
(no sklearn.metrics, no shared helper). Cross-check on pooled OOS (n=279):

| Metric | committed         | oracle            | diff  |
|--------|-------------------|-------------------|-------|
| ROC-AUC| 0.52352729885     | 0.52352729885     | 1.1e-16 |
| PR-AUC | 0.3301550655      | 0.3301550655      | 5.6e-17 |
| LogLoss| 0.6473685391      | 0.6473685391      | 0      |
| Brier  | 0.2259499408      | 0.2259499408      | 0      |

Verification note: an initial oracle used a naive trapezoid PR-AUC which
differed from the committed average-precision convention by ~0.011. This was a
DEFINITIONAL convention difference (discovered by the oracle), not a data /
pipeline defect; the oracle was realigned to the standard average-precision
convention, after which it matched to ~5e-16. Documented as process, not
silently skipped.

## 4. Reproducibility

Two independent runs produced byte-identical fold results and OOS predictions
(SHA-256 match on p3_s23_fold_results.csv and p3_s23_oos_predictions.csv);
metadata (generated_utc / git_commit) differ as permitted.

## 5. Result

```text
Dataset accounting / fold topology / purge / train-only scaler  : PASS
Mutation tests 8/8                                              : PASS
Independent metric oracle match                                 : PASS (~1e-6)
Two-run byte reproducibility                                    : PASS
NO SILENT BUG FOUND
```

Evidence: `ml/p3/p3_s23_corrected_population_ml/output/p3_s23_silent_bug_verification.json`,
`p3_s23_reproducibility.json`.

*End of P3-S23 silent-bug verification.*