SniperGold_ML/docs/P3_S25_1C_FINAL_DATASET_CERTIFICATION_PLAN.md

# P3-S25.1C — FINAL DATASET CERTIFICATION PLAN (for the future 34.5 GB dataset)

```text
Date       : 2026-08-28
Phase      : P3-S25.1C
Status     : DESIGN ONLY - NOT EXECUTED
Scope      : Defines how the eventual 34.5 GB external tick dataset will be
             certified. Execution is a FUTURE authorized phase.
```

## Purpose

Define an independent certification architecture for the eventual full 34.5 GB
ingestion, on top of the chunk-size invariance and dynamic-chunk verification
established in this phase. Nothing here is executed during P3-S25.1C.

## 1. Structural integrity

- **source SHA-256**: full-file digest of `XAUUSD_mt5_ticks.csv`
  (recorded `5252ce8f...cba1`; recomputed once at certification).
- **chunk hashes**: per-chunk `raw_chunk_sha256` and `parsed_chunk_sha256`
  recorded in each chunk manifest.
- **checkpoint chain**: cumulative `rows_processed`, `m15_rows_emitted`,
  `m30_rows_emitted`, `byte_offset`, `next_byte_start`, `last_timestamp`, and
  `checkpoint_sha256` after every committed chunk.
- **byte continuity**: `byte_end(chunk N) == byte_start(chunk N+1)`
  (line-safe extension may slightly exceed target; boundaries must join).
- **row accounting**: cumulative rows sum must equal per-chunk row sums.
- **state continuity**: carry-in of each chunk equals carry-out of the prior
  chunk; first_ts(chunk N+1) >= last_ts(chunk N) (equal timestamps legal for
  same-second groups).

## 2. Numerical integrity

- **independent M15 oracle**: re-derive M15 OHLC from raw ticks using a
  from-scratch bucket aggregator (not the production one) on representative
  spans; require byte-identical OHLC and counts.
- **independent M30 oracle**: likewise for M30 from M15 sub-buckets.
- **representative boundary replay**: replay a sample of chunk boundaries and
  confirm single-stream == chunked (the invariance method of P3-S25.1C).
- **cross-chunk checks**: verify carry hand-off across pairs of adjacent
  chunks on a sample.
- **chunk-size invariance**: re-run the 12-vs-24-vs-48-vs-64 invariance test
  at certification-time on a freshly selected representative range.

## 3. Semantic integrity (ONLY after 1 and 2)

- **F1 / F2 / F3 / F4** population counts.
- **Candidate Setup** detection counts.
- **population** reconciliation against the established external-data basis.
- **label consistency** under the P3-S16 label contract.

These semantic steps are gated: they are NOT run in P3-S25.1C.

## 4. Periodic audit policy (future, not yet executed)

```text
every chunk      : local structural validation (hash, row count, continuity,
                   malformed==0)
every 10 chunks  : cross-boundary audit (carry hand-off across 1 boundary)
every 25-50 chunks: independent replay sample (fresh oracle on a sub-range)
final            : independent source-based certification (sections 1-3)
```

## 5. Chunk-dependency principle

> If an upstream chunk is later found invalid, every downstream chunk whose
> state depends on that chunk is UNTRUSTED until replayed from the corrected
> upstream checkpoint.

The system must support:

```text
invalidate chunk N
        -> replay N
        -> rebuild downstream state
```

Downstream hashes are NOT independent evidence if they depend on invalid
upstream carry. Certification must therefore keep the full carry-chain and
checkpoint lineage to enable targeted invalidation + replay.

## 6. Mutation-test coverage (already established here)

The dynamic-chunk controller verifier detects all 10 required mutations
(10/10): wrong chunk size, skipped chunk, duplicated chunk, byte offset
shift, source hash mismatch, carry mutation, chunk-order mutation, output-row
deletion, output-row duplication, semantic output mutation.