SniperGold_ML/docs/CODE_VERIFICATION_AND_SILENT_BUG_POLICY_v1.md

# CODE VERIFICATION AND SILENT BUG POLICY — v1

```text
Date       : 2026-08-25
Version    : v1
Session    : P3-S21.R — RETROSPECTIVE SILENT-BUG VERIFICATION
Status     : ADOPTED (apply to historical-results retrospective verification)
Scope      : ESTABLISHES the verification standard for auditing historical
             results. It does NOT authorize changing any historical result.
Follow-on  : This policy is applied retrospectively (P3-S20, P3-S21.2) and
             forward to any future verification/experimental phase.
```

---

## 0. Purpose

A "test PASS" or "reproducible output" is NOT sufficient evidence that a
result is correct. A defect can be present, deterministic, and reproducible,
yet invisible to the verification regime that produced it. This policy
formalizes the distinction between routine reproduction and independent
verification, and defines the discipline for conducting a retrospective
silent-bug audit of historical research results.

This policy DOES NOT authorize modifying, deleting, or "fixing" any
historical report, contract, model, label, feature, or output. A defect,
once found, is documented, classified, and kept auditable. Corrective action
happens only in a separate, explicitly authorized phase.

---

## 1. Contract-first verification

Any result must be checked against a stable, frozen contract — not against a
"reasonable value". Contracts include: label contract (P3-S16 v1), dataset
contract, feature schema (12 causal features, feature_sha16), fold definition,
preprocessing rule (train-only scaler), model configuration, and the
pre-registered decision rule. Verification asks does the artifact satisfy the
contract, not does it look plausible.

## 2. Independent oracle / reference implementation

Metrics and derived results must be recomputed by code that does not share
the implementation under test and does not call the same helper in a way that
merely echoes its own output. The oracle may be:
- a from-scratch reference implementation of the metric (e.g., manual
  trapezoid ROC-AUC, manual log-loss / Brier),
- an alternative library path,
- or a mathematically independent check.
If the only available method is the historical helper itself, that is
"historical reproduction", NOT independent verification, and must be labeled
as such.

## 3. Boundary-condition testing

Verify behavior at boundaries: purge gap exactly == H (16), gap H-1 (must
fail), gap 0, empty partitions, single-class OOS, first/last fold edges,
index boundaries [0:300]/[300:395]. Tests on a boundary must exercise the
branch that decides PASS/FAIL, not just the nominal path.

## 4. Temporal isolation testing

Verify that no future observation is available to any earlier training step:
- chronological ordering of rows,
- expanding (not shuffling) windows,
- purge gap > H at every boundary,
- no future feature value enters training,
- no folded/metric or calibration fit feedback into earlier decision.

## 5. Leakage testing

Verify that out-of-sample data influence nither:
- feature transformation (scaler fit),
- model fitting,
- hyperparameter / threshold selection,
- calibration fitting,
- comparator (majority prior).
Each of these must keep the OOS/inference partition information-free.

## 6. Metamorphic testing

Apply input transformations that preserve the expected invariant and check the
invariant holds. For example: permuting non-temporal feature order should not
change a per-row prediction if rows are re-assigned consistently; applying a
strictly monotone transform to a ranking score should preserve ROC-AUC;
duplicating rows must increase counts exactly and must not change per-fold
structural checks.

## 7. Mutation testing

Temporarily inject a single deliberate fault into a DISPOSABLE copy of the
pipeline (never into the real artifact) and confirm the verification suite
DETECTS it. Representative mutations:
- purge boundary shifted by -1 bar,
- a future observation included in an earlier training set,
- scaler fit globally instead of per-train,
- temporal order shuffled,
- OOS leaked into training,
- feature rows shifted by one bar (row misalignment).
If a mutation passes the suite undetected, the corresponding verification
coverage is classed INSUFFICIENT. Intentionally-broken code is never committed
to the real project and is never used to produce a reported metric.

## 8. Independent reimplementation where practical

Where a result materially drives a conclusion (e.g., pooled OOS ROC-AUC),
reconstruct the result from a fresh, minimal, audited implementation and
compare to the committed number within an explicit tolerance. This is
stronger than re-running the historical script.

## 9. Human-auditable fixtures

Verification must rest on small, inspectable fixtures (synthetic datasets
with hand-computed expected values / known defects) in addition to the real
population. A fixture that a human can hand-check with pencil and paper is
treated higher than a fixture only interpretable by the same code.

## 10. Reproducibility

Every verification artifact is itself reproducible: fixed seed, byte-stable
outputs where deterministic, and a recorded run footprint. A verification
that can't be rerun to the same numbers is not evidence.

## 11. Distinction of evidence grades

The following grades are distinct and must be reported explicitly:
- **test PASS**: an existing test suite reports green. It is the weakest,
  a statement about the suite, not about the truth.
- **implementation reproduction**: running the historical pipeline again
  yields the same outputs. Establishes determinism, not correctness.
- **independent verification**: an independent oracle, boundary, leakage,
  or mutation test confirms a specific property. This is the claim used for
  the classifications in section 13.
- **validated evidence**: multiple independent methods converge on the same
  conclusion. The strongest grade.

## 12. Retrospective verification requirements for historical results

Historical phases are graded not on the historical regime that produced them,
but on the current policy in this document. A historical result may be
reclassified (its evidence status downgraded) without changing any of its
bytes.

## 13. Evidence classification (one per audited phase)

- VERIFIED — independent verification under this policy establishes
  correctness.
- VERIFIED WITH LIMITATIONS — no defect found, but some verification
  dimensions remain statistically/technically limited.
- VERIFICATION INCOMPLETE — evidence insufficient to establish correctness.
- POTENTIAL SILENT BUG — plausible defect identified but not proven.
- CONFIRMED SILENT BUG — defect reproduced and shown to affect results.
- INVALIDATED — confirmed defect materially changes the phase conclusion.

The word "PASS" alone is never used as the final classification.

## 14. Rules for downgrading evidence without deleting historical results

Downgrading is a change in the *evidentiary status*, not in the recorded
value. Historical outputs remain byte-identical and auditable. A grade may
be changed in the governing report/handover, with the basis of the change
any exposed defect. The original number printed is never edited.

## 15. Rules prohibiting silent modification of historical artifacts

- No historical report, CSV, JSON, model config, label, feature, frozen
  contract, or MQL5 file may be edited or re-written during a verification
  pass.
- Running a historical script that writes to its own committed outputs is
  permitted only if the writes are byte-identical; otherwise the outputs
  must be restored (e.g., git checkout) and any residual diff documented.
- Mutation/alternate implementations live only in disposable copies, never in
  tracked files that could reach the real project artifacts.

---

*End of policy v1. It establishes the standard for the P3-S21.R retrospective
audit. It authorizes no change to any historical result.*