7.7 KiB
CODE VERIFICATION AND SILENT BUG POLICY — v1
Date : 2026-08-25
Version : v1
Session : P3-S21.R — RETROSPECTIVE SILENT-BUG VERIFICATION
Status : ADOPTED (apply to historical-results retrospective verification)
Scope : ESTABLISHES the verification standard for auditing historical
results. It does NOT authorize changing any historical result.
Follow-on : This policy is applied retrospectively (P3-S20, P3-S21.2) and
forward to any future verification/experimental phase.
0. Purpose
A "test PASS" or "reproducible output" is NOT sufficient evidence that a result is correct. A defect can be present, deterministic, and reproducible, yet invisible to the verification regime that produced it. This policy formalizes the distinction between routine reproduction and independent verification, and defines the discipline for conducting a retrospective silent-bug audit of historical research results.
This policy DOES NOT authorize modifying, deleting, or "fixing" any historical report, contract, model, label, feature, or output. A defect, once found, is documented, classified, and kept auditable. Corrective action happens only in a separate, explicitly authorized phase.
1. Contract-first verification
Any result must be checked against a stable, frozen contract — not against a "reasonable value". Contracts include: label contract (P3-S16 v1), dataset contract, feature schema (12 causal features, feature_sha16), fold definition, preprocessing rule (train-only scaler), model configuration, and the pre-registered decision rule. Verification asks does the artifact satisfy the contract, not does it look plausible.
2. Independent oracle / reference implementation
Metrics and derived results must be recomputed by code that does not share the implementation under test and does not call the same helper in a way that merely echoes its own output. The oracle may be:
- a from-scratch reference implementation of the metric (e.g., manual trapezoid ROC-AUC, manual log-loss / Brier),
- an alternative library path,
- or a mathematically independent check. If the only available method is the historical helper itself, that is "historical reproduction", NOT independent verification, and must be labeled as such.
3. Boundary-condition testing
Verify behavior at boundaries: purge gap exactly == H (16), gap H-1 (must fail), gap 0, empty partitions, single-class OOS, first/last fold edges, index boundaries [0:300]/[300:395]. Tests on a boundary must exercise the branch that decides PASS/FAIL, not just the nominal path.
4. Temporal isolation testing
Verify that no future observation is available to any earlier training step:
- chronological ordering of rows,
- expanding (not shuffling) windows,
- purge gap > H at every boundary,
- no future feature value enters training,
- no folded/metric or calibration fit feedback into earlier decision.
5. Leakage testing
Verify that out-of-sample data influence nither:
- feature transformation (scaler fit),
- model fitting,
- hyperparameter / threshold selection,
- calibration fitting,
- comparator (majority prior). Each of these must keep the OOS/inference partition information-free.
6. Metamorphic testing
Apply input transformations that preserve the expected invariant and check the invariant holds. For example: permuting non-temporal feature order should not change a per-row prediction if rows are re-assigned consistently; applying a strictly monotone transform to a ranking score should preserve ROC-AUC; duplicating rows must increase counts exactly and must not change per-fold structural checks.
7. Mutation testing
Temporarily inject a single deliberate fault into a DISPOSABLE copy of the pipeline (never into the real artifact) and confirm the verification suite DETECTS it. Representative mutations:
- purge boundary shifted by -1 bar,
- a future observation included in an earlier training set,
- scaler fit globally instead of per-train,
- temporal order shuffled,
- OOS leaked into training,
- feature rows shifted by one bar (row misalignment). If a mutation passes the suite undetected, the corresponding verification coverage is classed INSUFFICIENT. Intentionally-broken code is never committed to the real project and is never used to produce a reported metric.
8. Independent reimplementation where practical
Where a result materially drives a conclusion (e.g., pooled OOS ROC-AUC), reconstruct the result from a fresh, minimal, audited implementation and compare to the committed number within an explicit tolerance. This is stronger than re-running the historical script.
9. Human-auditable fixtures
Verification must rest on small, inspectable fixtures (synthetic datasets with hand-computed expected values / known defects) in addition to the real population. A fixture that a human can hand-check with pencil and paper is treated higher than a fixture only interpretable by the same code.
10. Reproducibility
Every verification artifact is itself reproducible: fixed seed, byte-stable outputs where deterministic, and a recorded run footprint. A verification that can't be rerun to the same numbers is not evidence.
11. Distinction of evidence grades
The following grades are distinct and must be reported explicitly:
- test PASS: an existing test suite reports green. It is the weakest, a statement about the suite, not about the truth.
- implementation reproduction: running the historical pipeline again yields the same outputs. Establishes determinism, not correctness.
- independent verification: an independent oracle, boundary, leakage, or mutation test confirms a specific property. This is the claim used for the classifications in section 13.
- validated evidence: multiple independent methods converge on the same conclusion. The strongest grade.
12. Retrospective verification requirements for historical results
Historical phases are graded not on the historical regime that produced them, but on the current policy in this document. A historical result may be reclassified (its evidence status downgraded) without changing any of its bytes.
13. Evidence classification (one per audited phase)
- VERIFIED — independent verification under this policy establishes correctness.
- VERIFIED WITH LIMITATIONS — no defect found, but some verification dimensions remain statistically/technically limited.
- VERIFICATION INCOMPLETE — evidence insufficient to establish correctness.
- POTENTIAL SILENT BUG — plausible defect identified but not proven.
- CONFIRMED SILENT BUG — defect reproduced and shown to affect results.
- INVALIDATED — confirmed defect materially changes the phase conclusion.
The word "PASS" alone is never used as the final classification.
14. Rules for downgrading evidence without deleting historical results
Downgrading is a change in the evidentiary status, not in the recorded value. Historical outputs remain byte-identical and auditable. A grade may be changed in the governing report/handover, with the basis of the change any exposed defect. The original number printed is never edited.
15. Rules prohibiting silent modification of historical artifacts
- No historical report, CSV, JSON, model config, label, feature, frozen contract, or MQL5 file may be edited or re-written during a verification pass.
- Running a historical script that writes to its own committed outputs is permitted only if the writes are byte-identical; otherwise the outputs must be restored (e.g., git checkout) and any residual diff documented.
- Mutation/alternate implementations live only in disposable copies, never in tracked files that could reach the real project artifacts.
End of policy v1. It establishes the standard for the P3-S21.R retrospective audit. It authorizes no change to any historical result.