P3-S25.1C — FINAL DATASET CERTIFICATION PLAN (for the future 34.5 GB dataset)

Date       : 2026-08-28
Phase      : P3-S25.1C
Status     : DESIGN ONLY - NOT EXECUTED
Scope      : Defines how the eventual 34.5 GB external tick dataset will be
             certified. Execution is a FUTURE authorized phase.

Purpose

Define an independent certification architecture for the eventual full 34.5 GB ingestion, on top of the chunk-size invariance and dynamic-chunk verification established in this phase. Nothing here is executed during P3-S25.1C.

1. Structural integrity

source SHA-256: full-file digest of XAUUSD_mt5_ticks.csv (recorded 5252ce8f...cba1; recomputed once at certification).
chunk hashes: per-chunk raw_chunk_sha256 and parsed_chunk_sha256 recorded in each chunk manifest.
checkpoint chain: cumulative rows_processed, m15_rows_emitted, m30_rows_emitted, byte_offset, next_byte_start, last_timestamp, and checkpoint_sha256 after every committed chunk.
byte continuity: byte_end(chunk N) == byte_start(chunk N+1) (line-safe extension may slightly exceed target; boundaries must join).
row accounting: cumulative rows sum must equal per-chunk row sums.
state continuity: carry-in of each chunk equals carry-out of the prior chunk; first_ts(chunk N+1) >= last_ts(chunk N) (equal timestamps legal for same-second groups).

2. Numerical integrity

independent M15 oracle: re-derive M15 OHLC from raw ticks using a from-scratch bucket aggregator (not the production one) on representative spans; require byte-identical OHLC and counts.
independent M30 oracle: likewise for M30 from M15 sub-buckets.
representative boundary replay: replay a sample of chunk boundaries and confirm single-stream == chunked (the invariance method of P3-S25.1C).
cross-chunk checks: verify carry hand-off across pairs of adjacent chunks on a sample.
chunk-size invariance: re-run the 12-vs-24-vs-48-vs-64 invariance test at certification-time on a freshly selected representative range.

3. Semantic integrity (ONLY after 1 and 2)

F1 / F2 / F3 / F4 population counts.
Candidate Setup detection counts.
population reconciliation against the established external-data basis.
label consistency under the P3-S16 label contract.

These semantic steps are gated: they are NOT run in P3-S25.1C.

4. Periodic audit policy (future, not yet executed)

every chunk      : local structural validation (hash, row count, continuity,
                   malformed==0)
every 10 chunks  : cross-boundary audit (carry hand-off across 1 boundary)
every 25-50 chunks: independent replay sample (fresh oracle on a sub-range)
final            : independent source-based certification (sections 1-3)

5. Chunk-dependency principle

If an upstream chunk is later found invalid, every downstream chunk whose state depends on that chunk is UNTRUSTED until replayed from the corrected upstream checkpoint.

The system must support:

invalidate chunk N
        -> replay N
        -> rebuild downstream state

Downstream hashes are NOT independent evidence if they depend on invalid upstream carry. Certification must therefore keep the full carry-chain and checkpoint lineage to enable targeted invalidation + replay.

6. Mutation-test coverage (already established here)

The dynamic-chunk controller verifier detects all 10 required mutations (10/10): wrong chunk size, skipped chunk, duplicated chunk, byte offset shift, source hash mismatch, carry mutation, chunk-order mutation, output-row deletion, output-row duplication, semantic output mutation.

3.6 KiB Raw Permalink Blame History