# P3_DATA_ENGINE_V1_SPEC — SNIPERGOLD DATA ENGINE v1 ## Data Engine Architecture & Clean Reprocessing Specification | Field | Value | |---|---| | Document ID | P3_DATA_ENGINE_V1_SPEC | | Version | 1.1.0 (amended by P3-DATA-ENGINE-004; Appendix C authoritative for the amended clauses) | | Status | DESIGN FROZEN + CONTROLLED AMENDMENT APPLIED (P3-DE-004); implemented & qualified | | Session type | Amendment: source-grammar correction + implementation re-qualification | | Authorized execution | Qualification only (NO real-data ingestion; NO G-14) | | Classification | Internal research infrastructure | --- ## 0. Executive Summary This document defines SNIPERGOLD DATA ENGINE v1 ("the Engine"): a standalone, headless Python system that performs a clean, deterministic, resumable full reprocessing of the authoritative XAUUSD Tickstory tick source (`XAUUSD_mt5_ticks.csv`, current known size 34,473,661,010 bytes). The Engine is designed so that the entire source processing lifecycle can complete with ChatGPT closed. The AI Assistant role is limited to research design, governance, audit, and interpretation. Python is the deterministic data processor. This session produced this specification only. No real-data ingestion was performed, no checkpoint was modified, and no pipeline code was changed. --- ## 1. Purpose Produce an implementation-ready architecture for a data engine that: 1. certifies the identity of the 34.5 GB raw tick source before any processing; 2. performs a clean full reprocessing of that source into a canonical, columnar, versioned, resumable dataset; 3. builds deterministic canonical time bars (M1, M5, M15, M30, H1); 4. builds independently reproducible research datasets from canonical data without ever re-reading the raw source; 5. runs headlessly, survives interruption (including Windows restarts), and resumes without AI assistance; 6. exposes full auditability: every layer has a deterministic identity and every artifact is independently verifiable. ## 2. Scope The Engine covers: - source certification (size, mtime, full SHA-256, format facts); - deterministic line-safe chunking of the raw source; - canonical tick parsing, classification, serialization, hashing; - parallel canonical tick conversion with deterministic merge semantics; - canonical bar aggregation (M1, M5, M15, M30, H1); - persistent canonical storage (Parquet by default; see Human Decision Gate G-1); - standalone checkpoint / resume with atomic fail-closed transitions; - headless CLI (`sniper-data`), config, manifests, evidence and hashing; - independent verification tooling, golden test corpus, failure model, resource management, completion protocol, clean reprocessing protocol, legacy comparison protocol; - research dataset builder driven strictly by research configuration. ## 3. Non-Goals The Engine MUST NOT: - require an AI Assistant session (or any human) to remain active; - make trading decisions, optimize a trading strategy, or select models; - chase a "successful" timeframe or perform forecasting; - read the legacy checkpoint as its own execution state (legacy checkpoint 759 is preserved and untouched; the new Engine starts its own lifecycle); - reuse the legacy parser/aggregator code as its implementation; - silently downgrade certification requirements; - auto-recover from material failures by reasoning; - use CSV as a default intermediate storage format; - write to, delete, or modify any legacy pipeline directories or evidence. ## 4. Architecture ### 4.1 Logical layers | # | Layer | Responsibility | Immutable contract | |---|---|---|---| | 0 | RAW SOURCE | immutable Tickstory CSV | never modified by the Engine | | 1 | SOURCE CERTIFICATION | identity + format facts | certificate JSON | | 2 | CANONICAL TICK REPRESENTATION | raw row → canonical record | canonical schema + hash | | 3 | CHUNKING | line-safe byte ranges | chunk map | | 4 | PROCESSING MODEL | parallel deterministic conversion | chunk output files | | 5 | AGGREGATION | tick → M1/M5/M15/M30/H1 bars | bar files + carries | | 6 | PERSISTENT CANONICAL DATASET | Parquet layer | dataset manifest | | 7 | CHECKPOINT / RESUME | atomic state | checkpoint + commit journal | | 8 | HEADLESS CLI | operator interface | exit codes | | 9 | RESEARCH DATASET BUILDER | config-driven derived data | research dataset + manifest | | 10 | DATASET MANIFEST | reproducibility record | manifest JSON | | 11 | EVIDENCE & HASHING | identity at every layer | evidence manifest | | 12 | INDEPENDENT VERIFICATION | non-producer verification | verification report | | 13 | GOLDEN TEST CORPUS | adversarial micro-inputs | expected outputs | | 14 | FAILURE MODEL | three-class fail-closed behavior | state transitions | | 15 | RESOURCE MANAGEMENT | bounded local execution | telemetry | | 16 | RUN COMPLETION PROTOCOL | machine-verifiable completion | RUN_COMPLETE.json | | 17 | CLEAN REPROCESSING PROTOCOL | the authorized future procedure | step gates | ### 4.2 Design invariants (binding) 1. **Determinism**: canonical output is a pure function of (source bytes, engine version, parser version, algorithm version, config snapshot). Worker count, scheduling, result arrival order, disk layout and wall-clock time NEVER affect output. 2. **Fail-closed**: every transition is atomic; on doubt the system stops in a documented state. There is no silent fallback and no autonomous recovery-by-reasoning. 3. **Resumable**: any chunk is either fully committed (file + journal record + digest) or not committed at all. A hard power-loss leaves at most an orphan `.tmp` file that resume deletes and re-dispatches. 4. **Source immutability**: the Engine opens the source read-only (`rb`), never writes it, and re-checks size+mtime at every resume. 5. **Integer-only canonical math**: the canonical path (parse → serialization → aggregation) uses integer arithmetic only. No floating point anywhere in canonical ingestion or bar aggregation. 6. **Append-only outputs**: committed chunk/bar files are never rewritten in place. Resume appends or skips; it never mutates committed data. 7. **Independent verification**: the verifier is a separate implementation (different parsing and aggregation code), not a re-run of producer code. ### 4.3 Execution topology ``` sniper-data CLI (parent/operator process) ├── chunkmap scanner (reads source bytes for newline boundaries) ├── certification (size/mtime/SHA-256 + format sniff) ├── dispatcher (assigns chunk indices to workers) │ ├── worker process 1..N (parse chunk -> chunk parquet + digests) │ └── ... (spawn context; no shared state) ├── commit journal (append-only hash-chained commit records) ├── checkpoint writer (atomic tmp+rename) ├── sequential aggregator (canonical ticks -> bars, carry-aware) ├── research builder (canonical bars -> research datasets) ├── verifier (independent implementation) └── completion/finalization (evidence + RUN_COMPLETE.json) ``` No component talks to a network. No component requires an interactive session. ## 5. Data Flow ``` RAW SOURCE (34.5 GB CSV, read-only) │ [certify] size, mtime, FULL SHA-256, format facts ▼ source_certificate.json (Layer 1) │ [chunkmap] newline-scan only ▼ chunk_map.json (Layer 3) │ [parse+canonicalize, parallel] ▼ ticks/chunk_%06d.parquet + malformed/malformed_%06d.jsonl (Layer 2/4/6) │ [ordered merge; sequential aggregator, carry-aware] ▼ bars//wl_%03d.parquet (Layer 5) │ [dataset manifest + content digests] ▼ dataset manifest (canonical) (Layer 6/10) │ [research config] ▼ datasets//… (Layer 9) │ [evidence + hash] ▼ evidence.json + RUN_COMPLETE.json (Layer 11/16) ``` Every layer records its inputs (`source_id`, versions) so each stage can be recomputed and compared. Research datasets are built from the canonical bar layer ONLY — never re-reading the 34 GB raw source. ## 6. Component Design ### 6.1 Module map (implementation target) ``` engine/ cli.py # argument parsing, command dispatch, exit codes config.py # config load + strict schema validation versions.py # canonical version constants (see §26) certify.py # Layer 1 chunkmap.py # Layer 3 (newline scanning; pure function of bytes) parse.py # Layer 2 producer parser (optimized, integer-only) canonical.py # canonical record serialize/hash (Layer 2) worker.py # Layer 4 worker entry point (spawn-safe) dispatcher.py # Layer 4 dispatch + retry + telemetry journal.py # commit journal (append-only, hash-chained) checkpoint.py # Layer 7 atomic checkpoint read/write aggregate.py # Layer 5 sequential deterministic aggregator storage.py # Layer 6 parquet writers, naming, schema metadata dataset_builder.py # Layer 9 research dataset builder manifest.py # Layer 10 manifest build/verify evidence.py # Layer 11 hashing + evidence manifest run_complete.py # Layer 16 completion protocol verify/ # Layer 12 INDEPENDENT verifier (separate impl) __init__.py vparse.py # separate CSV/line parser written from spec vaggregate.py # separate aggregator written from spec vinvariants.py # invariant checks vcompare.py # differential comparison tests/golden/ # Layer 13 corpus + expected outputs cases/ expected/ tests/mutations/ # mutation test driver + cases ``` ### 6.2 Responsibilities matrix | Component | Reads | Writes | Fails closed by | |---|---|---|---| | certifier | source (rb) | `certification/source_certificate.json` | never writing source; recording status | | chunkmapper | source (rb) | `chunkmap/chunkmap-.json` | deterministic scan; no parsing | | worker | source range, chunk map | `staging/chunk_%06d.parquet.tmp` → `ticks/chunk_%06d.parquet`; `malformed/...` | one chunk per process; tmp+rename; digest computed before rename | | dispatcher | chunk map, journal | journal entries, `progress.json` | bounded retries then FAILED | | aggregator | ticks parquet (ordered) | `bars//wl_%03d.parquet`; carries in checkpoint | append-only; carry continuity check on resume | | verifier | source ranges, artifacts | `verification/...` | never reusing producer parse/aggregate code | | finalizer | everything | `evidence.json`, `RUN_COMPLETE.json` | only from COMPLETED + verified state | ### 6.3 Configuration Config is a strict-schema JSON file passed at `init`. Fields relevant to data identity: `source_path`, `source_tz_offset_minutes`, `has_volume`, `timeframes`. Fields relevant to operation: `workers_requested`, `chunk_bytes_nominal`, `workload_bytes_nominal`, `checkpoint_every_workload`, `retry_limit`, `memory_limits`, `disk_floor_bytes`, `timeouts`. Every config field that affects output is captured in the dataset manifest (`config_snapshot` + `config_sha256`). Operational-only fields are excluded from `config_sha256`. ## 7. Source Certification (Layer 1) ### 7.1 Purpose Before any processing, the Engine certifies that the configured source file is the authorized, unmodified Tickstory source. ### 7.2 Certificate schema `certification/source_certificate.json`: ``` schema_version "CERT_V1" certification_timestamp ISO-8601 UTC certification_status FULLY_VERIFIED | VERIFIED_WITH_LIMITATION | FAILED source_path absolute path as configured source_size_bytes integer, stat size source_mtime_ns integer, stat mtime nanoseconds (Windows st_mtime_ns) sha256_full hex (only present when the full file was hashed) sha256_mode "full" | "not_computed" limitation_reason null | enum(see 7.4) source_format "csv_tickstory_mt5" | "unrecognized" | "mismatch" grammar_id "tickstory_mt5_v1" | "dotted_v1" | null (P3-DE-004: pins the source-adapter preset; init cross-check §8.1) line_ending "CRLF" | "LF" | "CR" | "MIXED" (with ratio record) encoding detected (BOM-aware) ; "utf8" | "ascii" | other timestamp_format_detected "compact_date_time" | "dotted_ms" | "dotted_s" | "unrecognized" delimiter "," (only comma supported in v1) has_header bool (detection rule §8.3) column_layout {"fields":N,"names":[...]} observed; 3/4 => [datetime,bid,ask(,vol)]; 6 => [date,time,bid,ask,last,volume] parser_version (constants §26) engine_version (constants §26) ``` ### 7.3 Certification procedure 1. Verify path exists and is a regular file. 2. Record `size` and `mtime_ns`. 3. Stream the file once, `rb`, in fixed 8 MiB buffers, computing SHA-256 over ALL bytes, and simultaneously sniffing: first 1 MiB (line terminator classification), first 1000 parseable line-shaped rows (timestamp/delimiter/ header detection), BOM presence. 4. On success: write certificate with status `FULLY_VERIFIED` and the full hash. 5. If the configurable time budget (`certify_timeout_sec`, default 3600; config option) elapses before the hash completes, the process stops cleanly and records `VERIFIED_WITH_LIMITATION` with `sha256_mode="not_computed"` and `limitation_reason="timeout"`. It MUST NOT claim a full hash. 6. Write is atomic (`*.tmp` + fsync + rename). Certificate path is derived from source identity fields; re-certification is an allowed overwrite, always recorded with a new timestamp. ### 7.4 Certification statuses | Status | Meaning | Consequences | |---|---|---| | FULLY_VERIFIED | full file hashed; format facts consistent | canonical pipeline eligible (subject to Gate G-3) | | VERIFIED_WITH_LIMITATION | full SHA-256 not completed within budget; only size/mtime/format facts guaranteed | pipeline MUST NOT claim full certification; starting a run requires explicit human authorization (Gate G-3) | | FAILED | size/mtime changed mid-read, unreadable, or format facts self-contradictory | RESUME_BLOCKED; no processing | A size or mtime change detected between the pre-stat and post-stat is an automatic FAILED with `limitation_reason="source_changed_during_certify"`. ### 7.5 Operational requirement FULL SOURCE SHA-256 VERIFIED is the required precondition of the canonical pipeline. Any decision to proceed with `VERIFIED_WITH_LIMITATION` is a HUMAN DECISION GATE (G-3). The Engine never silently downgrades this requirement. ## 8. Canonical Tick Representation (Layer 2) ### 8.1 CSV interpretation (deterministic rules) - **Encoding**: UTF-8 (a leading UTF-8 BOM is stripped). Any undecodable byte sequence marks the row `MALFORMED_ENCODING`. ASCII is a strict subset and is processed identically. - **Delimiter**: comma only in v1. Detection result is recorded in the certificate; a mismatch between certificate and config blocks init. - **Line endings**: the row scanner splits on any of `\r\n`, `\n`, `\r` (deterministic; independent of the recorded classification). The recorded `line_ending` is diagnostic. - **Blank lines**: skipped, counted in `malformed.blank_line` (expected class). - **Header**: after BOM strip, if the first non-blank row has a first field matching `^(datetime|date|time)([^0-9]|$)` (case-insensitive), that row is the header and is skipped with `has_header=true` recorded in the certificate. The header row boundary is deterministic. - **Grammar presets (P3-DATA-ENGINE-004 source adapter)**: exactly two named, fully-specified, versioned grammar presets exist: `G_TICKSTORY_MT5` (six-column `YYYYMMDD,HH:MM:SS,bid,ask,last,volume`; the authoritative production grammar, derived from the P3-DATA-ENGINE-003 pilot evidence) and `G_DOTTED_V1` (dotted 3/4-column corpus grammar, retained for the synthetic G01–G17 fixtures). The certificate pins `grammar_id` from the observed facts; the operator config `source_grammar` MUST equal the certified `grammar_id` or `init` is BLOCKED (certificate/config mismatch, spec 8.1). There is no silent per-row format guessing. - **Timestamp**: accepted forms (recorded at certification): - `YYYYMMDD` + `HH:MM:SS` (compact date + time fields; G_TICKSTORY_MT5) — PRIMARY; - `YYYY.MM.DD HH:MM:SS.mmm` (dotted, 3-digit milliseconds; G_DOTTED_V1) — corpus preset; - `YYYY.MM.DD HH:MM:SS` (no ms) — corpus preset, ms = 0. Any other shape → `MALFORMED_TIMESTAMP_PARSE`. - **Timezone**: source text is a wall-clock in the trader's server time. Canonical `ts_ms` = epoch milliseconds UTC computed as: `ts_ms = julian_day_ms(wall_clock) - source_tz_offset_minutes * 60000`. `source_tz_offset_minutes` is an explicit config value recorded in the manifest; DST is NOT modeled in v1 (see Open Decisions, risk R-TZ). The parse→canonical mapping is therefore a pure, documented function of the source text, so any future re-interpretation is a re-derivation with a new dataset version, never an in-place mutation. - **Timestamp range**: valid only if `2000-01-01T00:00:00Z <= ts_ms <= certify_time+7d`; outside → `MALFORMED_TIMESTAMP_RANGE`. ### 8.2 Price, bid/ask, volume semantics - Canonical prices are **scaled integers**: `PRICE_SCALE = 1_000_000` (micro-USD units). `bid_u = int(decimal_text * PRICE_SCALE)` computed by exact decimal-string parsing (no floats). - `ask_u` likewise. `spread_u = ask_u - bid_u`. - Invalid relationships: - `bid_u <= 0` or `ask_u <= 0` → `MALFORMED_PRICE_NONPOSITIVE`; - `spread_u < 0` → `MALFORMED_BID_ASK_RELATION`; - fractional digits beyond the scale → `MALFORMED_PRICE_PRECISION`. - **Volume (amended P3-DATA-ENGINE-004)**: the grammar presets fix the volume semantics — for `G_TICKSTORY_MT5` the source HAS an explicit mandatory volume column so `has_volume=true` is enforced by config validation; `vol = 1` substitution (`has_volume=false`) is FORBIDDEN for this source. Volume is a non-negative integer token; `0` is VALID (observed in the real source: many canonical rows carry volume 0); negative or non-integer values are `MALFORMED_VOLUME`. The dotted preset retains `has_volume=false` (default) → `vol = 1` tick-volume semantics. Any violation of the certificate's `column_layout` vs config → init block. - Duplicate detection: identical (`ts_ms,bid_u,ask_u,vol`) records within one chunk are flagged `MALFORMED_DUPLICATE` (bounded in-memory set per chunk). - Monotonicity: within a chunk, `ts_ms` must be non-decreasing; otherwise the offending row is `MALFORMED_NON_MONOTONIC`. Across a chunk boundary the first `ts_ms` of chunk N must be >= the last `ts_ms` of chunk N-1 (carried in checkpoint); a violation is counted as `MALFORMED_NON_MONOTONIC_BOUNDARY` and, if > threshold, becomes material (§20). ### 8.3 Malformed-record taxonomy (complete) | Class | Triggers | |---|---| | MALFORMED_EMPTY_LINE | blank row | | MALFORMED_ENCODING | undecodable bytes | | MALFORMED_FIELD_COUNT | field count ≠ grammar-expected (G_DOTTED_V1: 3/4; G_TICKSTORY_MT5: exactly 6) | | MALFORMED_TIMESTAMP_PARSE | timestamp text unrecognized | | MALFORMED_TIMESTAMP_RANGE | ts outside validity window | | MALFORMED_PRICE_PARSE | bid/ask non-numeric | | MALFORMED_PRICE_PRECISION | decimals exceed PRICE_SCALE precision | | MALFORMED_PRICE_NONPOSITIVE | bid<=0 or ask<=0 | | MALFORMED_BID_ASK_RELATION | ask < bid | | MALFORMED_VOLUME | volume invalid (when has_volume) | | MALFORMED_DUPLICATE | exact duplicate within chunk | | MALFORMED_NON_MONOTONIC | ts regress within chunk | | MALFORMED_NON_MONOTONIC_BOUNDARY | ts regression across chunk boundary | Malformed rows are NEVER fabricated, skipped silently, or injected into the canonical stream. Each is appended to `malformed/malformed_%06d.jsonl` (`chunk_index`, `byte_start`, `global_line_no`, `class`, `raw_sha256`, `reason`). Counters per class accumulate in the checkpoint and in `malformed_summary.json` at completion. If malformed rows exceed `malformed_rate_limit` (default 0.001 of parsed rows in a workload) OR the absolute cap (default 10,000 in a workload), the run enters FAILED (§20). ### 8.4 Canonical record (CTS_V1) | Column | Type | Semantics | |---|---|---| | ts_ms | int64 | epoch ms UTC | | bid_u | int64 | micro-USD × 1e6 | | ask_u | int64 | micro-USD × 1e6 | | spread_u | int64 | ask_u − bid_u (≥ 0) | | vol | int64 | source volume or 1 | | src_line | int64 | global 1-based raw line number (audit anchor) | | rec_ord | int64 | 0-based record ordinal within the chunk | All values are integers. No floats, no nulls, no strings in the canonical tick table. ### 8.5 Canonical serialization & hashing The canonical serialization of a record r is the single line: ``` ts_ms|bid_u|ask_u|vol ``` (no spaces, `\n` terminated). The **chunk content digest** is: ``` chunk_content_id = SHA-256( concat(serialize(r_0), serialize(r_1), …, serialize(r_{n-1})) ) ``` in `rec_ord` order. The chunk content id is the reproducibility identity of the chunk (Parquet physical bytes are not part of the reproducibility contract, because physical writer metadata may differ run-to-run; the logical serialization is fixed by this spec). The chunk physical SHA-256 is also recorded for integrity evidence. ### 8.6 LAST-field treatment and source-preservation layer (P3-DE-004) The G_TICKSTORY_MT5 source carries a fourth price column `last` (last traded/ last quote price) that does NOT map into CTS_V1 (bid/ask/spread/vol). Decision chain, based on preservation, reproducibility, future research usefulness and schema integrity: 1. **Validated** at parse time with the same decimal micro-price grammar as bid/ask (`PRICE_SCALE`; non-positive or over-precision → the corresponding MALFORMED_PRICE_* class). Bid/ask relationship validation remains mandatory; `last` is deliberately NOT cross-validated against bid/ask (a last price may legally equal bid, ask or any value; imposing an invariant would be an unsupported assumption). 2. **Deliberately EXCLUDED from CTS_V1**: the canonical tick schema (and its serialization/content-id contract) is UNCHANGED — P3-DE-004 does not re-open G-4/G-5. Promoting `last` into the canonical schema would be a new CTS version and a new dataset version (human gate G-5). 3. **Preserved in a source-preservation layer**: `source_preservation/chunk_%06d.jsonl`, one line per canonical row in rec_ord order, `src_line|last_u` (e.g. `12|340345000`). The canonical preservation serialization is the single line `src_line|last_u\n`; the preservation chunk content id is SHA-256 over the concatenation (micro-units, integer math only). The sidecar content id and physical SHA-256 are recorded in the evidence manifest and independently re-verified (spec 18). 4. **Evidence basis**: the immutable raw source (certified FULL SHA-256) remains the ultimate preservation record; the sidecar makes `last` recoverable from canonical-run artifacts without re-reading the 34 GB source, and keeps future research (e.g., last-price labels) derivable under the frozen canonical identity. Dotted-grammar runs produce no sidecars. ## 9. Chunking (Layer 3) ### 9.1 Evaluation of legacy values - Legacy `atomic chunk = 24 MiB`, `workload = 512 MiB nominal`. Evaluation: - 24 MiB ⇒ ~1,370 chunks for the 34.5 GB source; each chunk holds ~0.6–0.9 M tick lines (~20–40 MB RSS per worker) — excellent checkpoint granularity, bounded worker memory, fine-grained resume. - 512 MiB nominal workloads ⇒ ~67 workloads; matches legacy differential comparison granularity (chunk 759/760 boundary lies on the same grid). - Both values are RETAINED as nominal defaults for v1. Actual boundaries are line-safe (below). These remain a human decision gate (G-7/G-8) because they affect the differential comparison surface. ### 9.2 Chunk map (deterministic) A single newline-scan pass over the source (no parsing) produces `chunkmap/chunkmap-.json`: ``` {"source_id", "chunk_bytes_nominal":25165824, "workload_bytes_nominal":536870912, "chunks":[ {"index":i, "byte_start":b, "byte_end":e, "line_count":n, "workload_index":w} , …], "total_bytes", "total_chunks", "total_workloads", "chunkmap_sha256"} ``` Rules: - `byte_start` is 0-based from file start. Chunk 0 starts at byte 0. - A chunk's nominal end = `byte_start + chunk_bytes_nominal`; the actual `byte_end` = first line-terminator position at or after the nominal end, INCLUSIVE of the terminator bytes. Consequence: `byte_end` is always a line boundary; no chunk ever splits a CSV record. - Chunk N+1 `byte_start` = chunk N `byte_end`. Boundaries are contiguous and cover `[0, total_bytes)` exactly. - If the source does not end with a newline, the final line is terminated by EOF; the last chunk ends at `total_bytes`. - A file smaller than one chunk yields exactly one chunk. - Workload grouping: accumulate whole chunks whose cumulative `byte_end − byte_start` first reaches or exceeds `workload_bytes_nominal`; the final workload contains the remaining chunks (partial final workload) and is recorded as `partial=true`. No padding. - `chunkmap_sha256` = SHA-256 of the deterministic JSON serialization (sorted keys, fixed field order). The chunk map is cached per `source_id`; on resume it is recomputed cheaply only if the source identity changed (which blocks). ### 9.3 Byte cursor semantics - Checkpoint records `next_chunk` and `next_byte_start` (= byte_start of the next uncommitted chunk). They always agree with the chunk map; the chunk map is the sole authority for boundaries. ## 10. Processing Model (Layer 4) ### 10.1 Model selection - **Canonical tick conversion**: parallel multiprocessing (spawn context, Windows-safe), one chunk per worker task. - **Aggregation**: sequential deterministic pass over ordered canonical ticks (see §11). Parallel aggregation is explicitly NOT in v1; it is a documented future optimization that must preserve identical content ids and pass the reordered-execution determinism test before adoption. - Rationale: conversion is CPU/parse-bound (parallel-friendly, pure per chunk); aggregation is stateful and order-dependent (sequential is the safest deterministic design for v1). ### 10.2 Worker & dispatch model - `workers_requested` (config, default 24 — Gate G-2); - `workers_created` = min(requested, `os.cpu_count()` or 8, memory estimates) — recorded, never assumed; - `workers_with_non_empty_partitions` = workers that actually received ≥ 1 chunk before the queue drained; - `cpu_utilization_pct` = measured (parent-side polling), reported but never tuned automatically. - Dispatch: parent keeps a deterministic ordered work list (chunk indices 0..N−1) and a simple job queue (process pool). A worker pulls exactly one chunk at a time, processes it, writes `staging/chunk_%06d.parquet.tmp`, computes digests, fsyncs, renames to `ticks/chunk_%06d.parquet`, returns (chunk_index, stats). The parent records the commit in the journal (hash-chained). Completion order NEVER affects outputs: each chunk file is independently named and content-identified; the canonical reading layer iterates chunk indices in ascending order. - No shared memory between workers; no queues carrying tick data; workers only communicate via files. ### 10.3 Deterministic merge "Merge" is a sorted concatenation by `chunk_index` (and by `rec_ord` within a chunk). The aggregator reads chunk files in ascending index order. Because chunk files are immutable once committed, merged output is identical for every arrival order. ### 10.4 Worker failure handling (bounded) | Event | Action (exactly one predefined recovery) | |---|---| | worker exits non-zero before rename | redispatch chunk (attempt counter); after `retry_limit=2` → FAILED | | worker killed (crash/OOM) | redispatch (attempt counter as above); stale `.tmp` deleted by dispatcher before redispatch | | chunk file exists but journal has no record | journal is truth; file re-verified; if digest matches → journal repaired to record commit (recorded `repaired=commit`); else re-dispatched | | journal has record but file missing/corrupt | FAILED (material; no silent re-issue) | | worker OOM repeatedly | attempt counter applies; then FAILED with `reason=worker_oom` | Commit is defined as: chunk parquet renamed and fsynced AND journal line appended and fsynced. Only both count. ## 11. Aggregation (Layer 5) ### 11.1 Timeframes and justification Included: **M1, M5, M15, M30, H1**. Justification: - M1 is the finest bar technically justified by tick input and is the base unit for all coarser frames; - M5..H1 form the standard ladder required by the research plans for horizons of 15/30/60 min and by the legacy differential protocol (M15/M30 are legacy comparison anchors); - no other timeframe is added; "interesting" frames are explicitly out of scope until a research decision demands one (then a new dataset version). Periods (ms): M1=60_000, M5=300_000, M15=900_000, M30=1_800_000, H1=3_600_000. ### 11.2 Boundary definition - Bar boundaries are UTC clock boundaries: `period_id = floor(ts_ms / period_ms)`; `start_ms = period_id * period_ms`; `end_ms = start_ms + period_ms`. - A tick belongs to the bar of its `period_id`. - Ticks are consumed in canonical order `(ts_ms ascending, then src_line ascending)` — stable and deterministic. - Within identical `ts_ms`, order is by `src_line` (raw file order). ### 11.3 OHLC / volume / spread (integer-exact) - The representative bar price is the mid: `mid_u2 = bid_u + ask_u` (units: 2×PRICE_SCALE = 2,000,000 per USD). This keeps all OHLC math in integers with no truncation: - `open_u2` = first mid_u2 in the bar; - `high_u2` = max mid_u2; `low_u2` = min mid_u2; `close_u2` = last mid_u2; - `PRICE_SCALE_BAR = 2_000_000` is recorded in the bar schema metadata. - Volume: `vol_sum` = Σ tick vol; `tick_count` = number of ticks. - Spread: `spread_min_u`, `spread_max_u`, `spread_sum_u`, `spread_avg_u = spread_sum_u // tick_count` (floor division; defined). - Audit anchors: `first_src_line`, `last_src_line`. ### 11.4 Carry state For each timeframe the aggregator keeps exactly one in-progress bar (the partial bar) as the carry: ``` TF carry = {period_id, start_ms, open_u2, high_u2, low_u2, close_u2, tick_count, vol_sum, spread_min_u, spread_max_u, spread_sum_u, last_ts_ms, last_src_line} ``` Carries are checkpointed with every commit (see §13). On resume the aggregator verifies that its first input `ts_ms` is ≥ the carried `last_ts_ms`; otherwise FAILED (carry mismatch = material). ### 11.5 Finalization rules - A bar is finalized when the first tick of a later period_id arrives. It is written to `bars//wl_%03d.parquet` with `is_final=true`. - At EOF, all carried partial bars are flushed with `is_final=false` and the dataset manifest records `final_bar_partial=true` for each timeframe. - Bar uniqueness key: `(TF, period_id)`. Append-only: a period_id is written at most once per run. On resume, the aggregator validates that the last written record for each TF equals the carried period_id (continuity check); mismatch → FAILED. - Bar global ordinal `bar_idx` (0-based per timeframe, contiguous across parts) makes reading order-independent. ### 11.6 Bar schema (CBS_V1) | Column | Type | Semantics | |---|---|---| | bar_idx | int64 | global ordinal per timeframe | | period_id | int64 | start_ms / period_ms | | start_ms / end_ms | int64 | UTC boundaries | | open_u2, high_u2, low_u2, close_u2 | int64 | units = 2 × PRICE_SCALE | | tick_count, vol_sum | int64 | counts | | spread_min_u, spread_max_u, spread_sum_u, spread_avg_u | int64 | spread in PRICE_SCALE units | | is_final | bool | false only for EOF partial bars | | first_src_line, last_src_line | int64 | audit anchors | ## 12. Storage Format & Persistent Canonical Dataset (Layer 6) ### 12.1 Format decision Default: **Apache Parquet**, compression **zstd** level 3, `write_page` layout with pinned writer version (see §26 determinism table). Rationale: columnar, typed (no float in canonical tabsh), compressed, stream-appendable by immutable files, universally readable, and row-group scannable. CSV is explicitly rejected as the default intermediate format (no types, slow, no random access, ambiguous escaping). PyArrow version is pinned and recorded in every manifest. ### 12.2 Layout ``` engine_output/ certification/source_certificate.json chunkmap/chunkmap-.json ticks/chunk_%06d.parquet # final per-chunk columnar files staging/… # .tmp intermediates (prunable after verification) bars/M1/wl_%03d.parquet , bars/M5/… , bars/M15/… , bars/M30/… , bars/H1/… malformed/malformed_%06d.jsonl state/checkpoint.json, state/commits.jsonl, state/lock, state/control.json logs/*.jsonl progress.json evidence/evidence.json RUN_COMPLETE.json datasets//… verification/… ``` ### 12.3 Naming rules - `chunk_%06d` (0-padded index), `wl_%03d` (workload index), `part` derived deterministically from workload index. - Every parquet file embeds schema key-value metadata: `schema_version`, `engine_version`, `parser_version`, `algorithm_version`, `source_id`, `chunk_index` (ticks) / `timeframe` (bars), `price_scale`, `content_id` is NOT embedded (it is computed over logical rows; see §12.4). ### 12.4 Dataset identity (content ids, not physical bytes) - `content_id(ticks chunk i)` = `chunk_content_id` from §8.5. - `content_id(bars wl w, TF)` = SHA-256 over canonical bar serialization: each row as `bar_idx|period_id|open_u2|high_u2|low_u2|close_u2|tick_count|vol_sum|spread_min_u|spread_max_u|spread_sum_u|is_final|first_src_line|last_src_line\n` concatenated in ascending `bar_idx`. - The **canonical dataset hash** = SHA-256 over the canonical concatenation (fixed field order) of `content_id(ticks chunk 0) ‖ content_id(chunk 1) ‖ … ‖ content_id(bars files…)` in sorted path order. Ordering is defined by the manifest file list. - Physical file SHA-256s are recorded alongside content ids in `evidence.json`; content ids are the reproducibility identity; physical hashes are integrity evidence. This distinction is explicit and non-negotiable (Parquet physical bytes are writer-dependent). ### 12.5 Schema metadata & versioning Manifest record per dataset: `{"dataset_version": "DS_V1.0.0", schema_version, engine_version, parser_version, algorithm_version, source_id, config snapshot}`. Dataset version bumps are HUMAN DECISION GATES (G-5). ## 13. Checkpoint / Resume (Layer 7) ### 13.1 Files - `state/checkpoint.json` — atomic (tmp+fsync+rename+dir fsync). - `state/commits.jsonl` — append-only commit journal, each line hash-chained to the previous: `{"seq":n,"chunk":i,"content_id":"…","file_sha256":"…","prev":"","ts_utc":"…"}` - `state/lock` — PID, run_id, heartbeat. Stale detection §21. ### 13.2 Checkpoint schema (CP_V1) ``` schema_version, run_id, status, source_identity {path,size,mtime_ns,sha256_full,cert_status,cert_path}, versions {engine, parser, algorithm, dataset, manifest schema}, chunkmap_sha256, journal_tail {seq, hash}, last_completed_chunk, next_chunk, next_byte_start, cumulative {rows_parsed, rows_canonical, bytes_read, malformed:{class:count}, chunks_committed, bars_per_timeframe:{TF:count}}, carries {TF: carry object (§11.4)}, update_ts_utc, status_reason (nullable), checkpoint_hash (SHA-256 over the canonical JSON serialization of all fields EXCEPT checkpoint_hash — non-self-referential) ``` ### 13.3 Status machine ``` INITIALIZED → RUNNING → CHUNK_COMMITTED* → PAUSED → RUNNING → … → COMPLETED │ │ └──────────── FAILED ──────────────────────┘ INITIALIZED / RUNNING / PAUSED / FAILED ──(resume guard)──▶ RESUME_BLOCKED ``` - `INITIALIZED` — after `init`, before first dispatch. - `RUNNING` — while workers are dispatched. - `CHUNK_COMMITTED` — recorded transition on every journal append (the durable state of progress; the file is rewritten with this status after each workload, see 13.4). - `PAUSED` — graceful stop (signal/control file), clean checkpoint, exit 0. - `FAILED` — material condition; processing stopped; resume is permitted only after operator resolution and explicit re-run (see §20). - `COMPLETED` — final after verification; final checkpoint written. - `RESUME_BLOCKED` — a resume attempt refused entry; no state mutation. Transitions are atomic (single rename) and fail-closed: any exception during a checkpoint write leaves the previous valid checkpoint untouched. ### 13.4 Checkpoint frequency Checkpoint + journal fsync after **every workload** (~67 writes for the full run) and always before graceful pause/stop and after any worker failure that causes re-dispatch. Chunk commit (journal append) is the fine-grained durable record; the checkpoint file is the coarse summary. On resume, journal replay (from the recorded tail) determines actual committed chunks — including chunks committed after the last checkpoint write. ### 13.5 Resume procedure (deterministic) 1. Load checkpoint; verify `checkpoint_hash`. 2. Verify source identity: recompute size+mtime; require certificate present and matching (`sha256_full` if FULLY_VERIFIED). Any mismatch → **RESUME_BLOCKED** (never auto-recovery). 3. Verify versions (engine/parser/algorithm/dataset/schema) match exactly → mismatch = RESUME_BLOCKED. 4. Verify `chunkmap_sha256` matches the cached map → mismatch = RESUME_BLOCKED. 5. Replay journal from `journal_tail.seq`; validate chain; derive `next_chunk`; verify each committed chunk file exists and its content id matches the journal → any anomaly = FAILED (material), not auto-fixed. 6. Restore carries from checkpoint; verify aggregator continuity (§11.5). 7. Resume dispatch from `next_chunk`. ### 13.6 Source identity mismatch `source_identity` mismatch → `RESUME_BLOCKED` with `status_reason` (`source_size_changed` | `source_mtime_changed` | `certificate_missing` | `certificate_mismatch` | `sha256_mismatch`). The operator must re-certify and decide (human gate) whether to start a new run. ## 14. Headless CLI (Layer 8) ### 14.1 Commands (semantics, not implemented this session) ``` sniper-data --version sniper-data certify --source [--timeout-sec N] [--output ] sniper-data init --config [--run-id ] sniper-data ingest [--limit-chunks N] [--pause-after N] sniper-data resume [--limit-chunks N] sniper-data pause | stop sniper-data status [--json] sniper-data verify --mode golden | range | dataset | run-complete | legacy [--start-byte N --end-byte M] [--samples N] sniper-data manifest [--dataset ] [--print] sniper-data build-bars [--timeframes M1,M5,M15,M30,H1] sniper-data build-dataset --config sniper-data test [--quick] sniper-data audit-dataset --dataset sniper-data diff-legacy --legacy-evidence --report ``` ### 14.2 Semantics and safety properties | Command | Meaning | Safety property | |---|---|---| | certify | compute certificate (read-only on source) | never opens source writable; atomic cert write | | init | create a new run (new run_id) from a certified source; refuses if a live run/lock exists; archives prior run on explicit `--new-run` | refuses to touch legacy evidence; validates cert + config | | ingest | start processing from INITIALIZED (or resume-equivalent allowed states) | idempotent dispatch; journal-before-checkpoint durability | | resume | continue from PAUSED/FAILED-after-resolution; RESUME_BLOCKED otherwise | verifies every guard in §13.5; changes nothing on refusal | | pause / stop | graceful stop at next chunk/workload boundary; checkpoint; exit | never kills a worker mid-chunk without completing its rename | | status | print checkpoint, progress, lock, journal tail; exit code indicates completion | read-only | | verify | run the independent verifier | refuses to run pipeline stages; only reads source/artifacts | | manifest | build/print the dataset manifest | refuses to list missing/corrupt files | | build-dataset | research dataset builder (§15) | reads canonical layers only; never the raw source | | diff-legacy | differential comparison vs legacy evidence | read-only on legacy evidence | | test | golden corpus runner | tiny synthetic inputs only | ### 14.3 Exit codes ``` 0 success / COMPLETED 1 operational error (I/O, config invalid) 2 usage error 3 not completed (status: RUNNING/PAUSED/INITIALIZED) 4 BLOCKED (RESUME_BLOCKED / lock held / certification missing) 5 verification or audit failed ``` The CLI is designed so start / pause / resume / verify / status / completion detection require no human conversation. ## 15. Research Dataset Builder (Layer 9) ### 15.1 Separation of concerns ``` DATA ENGINE → CANONICAL DATA → RESEARCH DATASET → FORECASTING EXPERIMENT → MODEL → TRADING DECISION ``` The Engine provides canonical ticks+bars and a fixed library of deterministic feature/label primitives. It holds NO forecasting hypothesis. Examples such as "M15 + 15m direction", "M15 + 30m return", "M30 + 60m return" are research configurations, never ingestion rules. ### 15.2 Research config schema (strict; hash → config_sha256) ``` { "dataset_id": "xau_m15_15m_dir_v1", "source_dataset_version": "DS_V1.0.0", "timeframe": "M15", "prediction_horizon_min": 15, "target": {"type": "direction|cls_3|return_reg", "horizon": "15m", "from": "close"}, "features": [{"name":..., "primitive": "return|log_return|volatility|range|spread_stats|rolling_*", "window": k, "args": {...}}], "date_range": ["2015-01-01","2020-12-31"], "split": {"policy": "time_ordered", "train":0.7,"val":0.15,"test":0.15, "oos_barrier": true, "barrier_gap_min": 15}, "cost_model": {"type":"fixed_spread","spread_units":…,"commission_per_lot":…,"lot_size":…}, "label_rules": {"min_abs_move_units":…,"neutral_zone":…}} } ``` Rules: - Inputs: the canonical bar layer of `source_dataset_version` ONLY. - Feature/label primitives are deterministic pure functions of bars (integer or pinned-float operations, fixed evaluation order). - Leakage rule: the label for row t uses bars `(t, t+h]` only; rows lacking `h` future bars are excluded and counted in the manifest (`rows_excluded_no_future`). - Split: strictly time-ordered; `oos_barrier` demands a gap of ≥ horizon between the last training bar and the first test bar (in bars), recorded in the manifest. - All boundaries are (row_id, bar_idx) based, never shuffled at row level. - Float math is allowed HERE (research layer), but the pipeline is deterministic: pinned library versions, fixed operation order, no hash randomization, no set-iteration in serialization paths. ### 15.3 Outputs ``` datasets// config.json, config_sha256, features.parquet, labels.parquet, dataset.parquet (row_id, ts, features, label, split), split.parquet (row_id → split), manifest.json ``` ## 16. Dataset Manifest (Layer 10) `manifest.json` schema (MS_V1) — applies to the canonical dataset and to every research dataset: ``` schema_version, dataset_id, dataset_version, engine_version, parser_version, algorithm_version, source_identity {certificate reference}, timeframe (research only), prediction_horizon_min, target_definition, features (list), date_range, split_definition, oos_barrier, cost_model, row_counts {total, per_split, excluded_no_future, malformed…}, files [ {relpath, bytes, sha256_physical, content_id} ], # sorted by relpath config_snapshot, config_sha256, creation_ts_utc, manifest_hash, # H(canonical JSON of manifest WITHOUT manifest_hash and dataset_hash) dataset_hash # H(canonical concat of (relpath|content_id) list) — excludes manifest_hash ``` Non-self-referential hashing: `manifest_hash` explicitly excludes both hash fields; `dataset_hash` is computed over file content ids only (plus version identity fields), never over the manifest that contains it. No hash cycle. ## 17. Evidence & Hashing Strategy (Layer 11) ### 17.1 Evidence hierarchy ``` SOURCE ↓ certification (FULLY_VERIFIED gate) CANONICAL TICKS content_id per chunk; dataset hash CANONICAL BARS content_id per workload/TF file DERIVED FEATURES content_id per dataset file LABELS content_id per dataset file RESEARCH DATASET dataset_hash per manifest ``` ### 17.2 Identity definitions - **run_id**: `RUN--`. Assigned at `init`. - **file hash**: SHA-256 of physical bytes (`sha256_physical`). - **content_id**: layer-specific canonical logical digest (§8.5, §12.4). - **aggregate/layer hash**: SHA-256 over the canonical concatenation of the layer's content ids in manifest-sorted order. - **manifest hash** / **dataset hash**: see §16 (no self-reference). - **checkpoint hash**: SHA-256 of the checkpoint payload excluding the hash field itself (§13.2). - **evidence manifest** (`evidence/evidence.json`): lists every produced file with both hashes, plus `evidence_manifest_hash` computed over itself minus the hash field. `RUN_COMPLETE.json` records the `evidence_manifest_hash`; no cycle (RUN_COMPLETE.json is not hashed by evidence.json). ### 17.3 Rules - All hash inputs use fixed canonical serialization (sorted keys at each JSON level, `\n` line endings, no trailing whitespace). - No hash is ever computed over a blob that contains that same hash. - `PYTHONHASHSEED` is irrelevant by construction (no set/dict iteration in any serialization path); the Engine additionally pins `PYTHONHASHSEED=0` as a belt-and-braces control. ## 18. Independent Verification (Layer 12) ### 18.1 Principle The verifier (`engine/verify/`) is a SEPARATE implementation written from this specification's tables — its own line splitter, timestamp parser, price parser, and aggregator. It does NOT import or call producer parsing, aggregation, or serialization code. Producer and verifier may share only the version constant tables. ### 18.2 Verification modes | Mode | What it does | |---|---| | golden | runs the golden corpus through BOTH implementations; byte-equal expected outputs | | range | re-parses selected source byte ranges with the verifier parser; compares canonical rows, ts, bid/ask against chunk parquet logical rows | | dataset | recomputes every content_id and dataset_hash from artifacts (independent of producer); checks manifest consistency | | run-complete | validates RUN_COMPLETE.json plus evidence manifest against artifacts | | legacy | differential comparison vs legacy evidence (§24) | ### 18.3 Verification techniques - **Golden vectors**: hand-computed expected outputs for tiny inputs. - **Deterministic invariants** (at least): OHLC validity (low ≤ open,close ≤ high; counts ≥ 1), ts_ms non-decreasing across merged stream, Σ per-chunk tick counts == total tick count, bar continuity (close of bar k == first tick of bar k+1 order boundary, carry continuity), content_id stability across producer/verifier, append-only uniqueness of (TF, period_id). - **Byte-range verification**: independent parse+aggregate over chosen ranges, compared to chunk artifacts. - **Hash verification**: full recomputation from files. - **Differential comparison** vs legacy (§24). - **Mutation tests**: flip a byte / duplicate a row / drop a row / alter a timestamp in a golden corpus COPY (never the source) → engine must change digests or classify per taxonomy. - **Boundary tests**: chunk boundary exactly at a terminator; boundary inside a line; CRLF/LF/CR/mixed inputs. - **Resume tests**: kill+resume cycles must reproduce identical content ids. - **Reordered-worker tests**: same input dispatched in different orders → identical content ids. - **Duplicated/missing data tests**: injected duplicates/missing rows must be detected by invariants (counters/hash changes). ### 18.4 Verdicts `verification/report_.json` with per-check `PASS|FAIL|SKIP(reason)|DIFF(classification)` and an overall `VERIFIER_ACCEPTED | REJECTED`. A FAILED invariant or a failed independent content-id comparison is material (§20). ## 19. Golden Test Corpus (Layer 13) Small, permanent, adversarial; each input < 100 KiB; full run < 10 s. | Case | Input characteristics | Expected effect | |---|---|---| | G01 CRLF | terminator `\r\n` everywhere | parse ok; counts exact | | G02 LF | `\n` only | parse ok; identical canonical ids to G01 | | G03 header present | first row header | skipped; has_header=true | | G04 header absent | data only | all rows data | | G05 boundary at exact byte | chunk cut lands exactly on terminator | one line per chunk | | G06 boundary inside line | chunk cut lands mid-line | chunk map extends boundary to line end; no split record | | G07 malformed timestamp | `2023.01.01 12:00:00` variants / garbage | MALFORMED_TIMESTAMP_PARSE count | | G08 malformed price | `abc` / signed / extra decimals | MALFORMED_PRICE_* classes | | G09 invalid bid/ask | ask < bid | MALFORMED_BID_ASK_RELATION | | G10 duplicated row | exact duplicate in chunk | MALFORMED_DUPLICATE + digest sensitivity | | G11 dropped row | verifier-independent count mismatch | invariant FAIL (verification-time) | | G12 non-monotonic | regressed ts | MALFORMED_NON_MONOTONIC | | G13 worker reorder | shuffled dispatch of same input | identical content ids | | G14 missing input carry | resume without carry state | RESUME_BLOCKED / continuity FAIL | | G15 altered state | corrupt checkpoint hash / journal chain | RESUME_BLOCKED / FAILED | | G16 repeated resume | pause+resume+resume | identical content ids (determinism proof) | | G17 already-transformed data | canonical-shaped input fed as raw | format mismatch → certification fails / blocked | Source-grammar golden cases SG01–SG15 (P3-DATA-ENGINE-004) extend the corpus with dedicated synthetic fixtures for the authoritative six-column G_TICKSTORY_MT5 grammar: SG01 grammar acceptance, SG02 CRLF handling, SG03 valid six-column row, SG04 wrong field count, SG05 invalid date, SG06 invalid time, SG07 invalid bid, SG08 invalid ask, SG09 invalid last, SG10 invalid volume, SG11 bid>ask, SG12 zero/negative volume, SG13 timestamp conversion (regression anchor `20030505,00:01:03` → `1_052_092_863_000` ms), SG14 decimal precision, SG15 header absence/presence semantics. Like G01–G17, every SG case requires producer == independent verifier == expected values. Each case has an `expected/` JSON (counters, classes, content ids) that both the producer and the verifier must reproduce. ## 20. Failure Model (Layer 14) Three classes. The Engine's explicit action set is: continue (with counter), apply ONE predefined recovery, or STOP. | # | Condition | Class | Action | Result state | |---|---|---|---|---| | F01 | malformed row (any taxon) | EXPECTED | classify + counter + sidecar | continue | | F02 | malformed rate > 0.001 or >10,000 in a workload | MATERIAL | STOP; record reason | FAILED | | F03 | blank lines / BOM / header | EXPECTED | deterministic skip | continue | | F04 | worker non-zero exit | BOUNDED | ONE recovery: redispatch, ≤2 attempts | continue/FAILED after limit | | F05 | worker OOM | BOUNDED | ONE recovery: redispatch ≤2 | FAILED(`worker_oom`) after limit | | F06 | chunk file present, no journal | BOUNDED | ONE recovery: verify digest, repair journal OR redispatch | continue | | F07 | journal says committed, file missing/digest mismatch | MATERIAL | STOP | FAILED | | F08 | checkpoint hash mismatch | MATERIAL | STOP; no auto-recovery | RESUME_BLOCKED | | F09 | source size/mtime/cert mismatch | MATERIAL | STOP | RESUME_BLOCKED | | F10 | version mismatch (engine/parser/algorithm/dataset/schema) | MATERIAL | STOP | RESUME_BLOCKED | | F11 | carry continuity / bar append mismatch | MATERIAL | STOP | FAILED | | F12 | disk free below floor at runtime | BOUNDED | ONE recovery: graceful pause | PAUSED(reason=disk_space); resume after human frees space | | F13 | disk free below floor at init | MATERIAL | STOP | init blocked | | F14 | lock held by live process | MATERIAL | STOP (no kill) | exit 4 (locked) | | F15 | unknown exception in canonical writer/aggregator | MATERIAL | STOP | FAILED | | F16 | verifier verdict REJECTED | MATERIAL | STOP before COMPLETED | FAILED(verification) | | F17 | malformed threshold inside chunk (pre-workload) | EXPECTED | carry counters | continue (workload check at end) | Rules: - No autonomous recovery-by-reasoning. No silent fallback. No broad self-repair. - Every STOP writes a checkpoint (atomic), a `status_reason`, and a `logs/failure_.jsonl` entry with the exact condition code. - `FAILED` runs resume only after an operator (human) resolves the root cause and re-invokes `resume` or starts a new run; the Engine never decides to bypass F07–F16. ## 21. Resource Management (Layer 15) | Resource | Requirement | |---|---| | Max memory, parent | ≤ 512 MB RSS (dispatcher/coordinator only) | | Max memory, worker | ≤ 1 GiB RSS per worker (chunk-bounded; one chunk in memory) | | Total memory target | requested workers × per-worker bound; never exceeds measured machine memory / 2 | | Worker isolation | separate processes (`spawn`); worker crash cannot corrupt parent | | Logging | JSONL, one operator log + per-worker logs; rotation at 100 MB, 5 files; logs never influence outputs | | Progress | `progress.json`: run_id, status, chunks done/total, bytes, workloads, workers{requested, created, with_data, busy, cpu_util_pct}, started/updated ts, eta_sec | | Checkpoint frequency | journal per chunk commit; checkpoint file per workload; always at pause/fail | | Graceful shutdown | SIGINT/SIGTERM or `state/control.json` → finish current chunk, join workers (≤300 s), checkpoint PAUSED, exit 0 | | OS restart recovery | any interruption leaves only orphan `.tmp`; resume (§13.5) restores exactly | | Disk-space checks | preflight: free ≥ max(2×estimated_canonical_size, 20 GiB); per-workload: free ≥ 4 GiB else PAUSED(disk_space) | | Output-space checks | staging cleaned after successful verification; `--prune-staging` explicit only | | Process cleanup | atexit + signal handlers remove lock; children joined; no zombie policy beyond retry | | Stale-process detection | lock heartbeat every 60 s; stale if >300 s old; resume with a stale lock → exit 4 and `status` reports `lock_stale`; releasing requires explicit `--force-release-lock` (human gate G-11) — default: refuse | The Engine never relies on AI session lifetime; all of the above executes under the OS scheduler. ## 22. Run Completion Protocol (Layer 16) `RUN_COMPLETE.json` is produced ONLY after: all chunks committed, aggregator flushed, evidence manifest computed, and the independent verifier (`verify --mode dataset` and `--mode run-complete`) returned ACCEPTED. Schema (RC_V1): ``` status: "COMPLETED", run_id, source_identity {ref to certificate}, versions {engine, parser, algorithm, dataset, manifest schema}, rows {parsed, canonical, per-chunk-min/max}, malformed {per class, total}, outputs {ticks_files, bar_files_per_TF, dataset_files}, checkpoint {last_completed_chunk, next_chunk(=total), status}, hashes {canonical_dataset_hash, manifest_hash, evidence_manifest_hash, per-layer aggregate hashes}, completion_ts_utc, verifier_report (path + verdict) ``` Machine-verifiability: `sniper-data verify --mode run-complete` re-checks every hash and file from the disk; exit code 0 ⇔ genuine completion. Operator flow: run the Engine headlessly; hours later, independently of any AI session, run `sniper-data status` (→ completion detection), then hand the result package (RUN_COMPLETE.json + evidence + manifests) to ChatGPT for audit. ## 23. Clean Reprocessing Protocol (Layer 17) Exact future procedure (NOT executed in this session; each step has an authorization requirement): | # | Step | Entry condition | Authority | |---|---|---|---| | 1 | Freeze legacy pipeline | approved; legacy remains reference-only, evidence preserved | G-12 | | 2 | Certify source | step 1; certificate FULLY_VERIFIED preferred | G-3 | | 3 | Initialize NEW engine | certificate accepted; new run_id (not checkpoint 759) | G-4 | | 4 | Run golden tests | `sniper-data test` green (producer + verifier) | — | | 5 | Run small real-data pilot | e.g., first 2 workloads or a configurable 512 MiB window | G-13 (pilot scope) | | 6 | Independently verify pilot | verifier PASS on pilot ranges | — | | 7 | START full 34 GB processing | step 6 accepted | G-14 (full reprocessing) | | 8 | Allow Python to run independently | no AI required; resume as needed | — | | 9 | Resume if interrupted | §13.5; no AI required | — | | 10 | Complete full dataset | status COMPLETED + RUN_COMPLETE.json | — | | 11 | Independently certify final dataset | verifier ACCEPTED on full dataset | — | | 12 | Freeze canonical research dataset | mark frozen in manifest; publish hashes | G-15 (freeze) | The legacy pipeline is not extended during or after this procedure. Legacy checkpoint 759 and all workload evidence remain untouched. ## 24. Legacy Comparison Protocol ### 24.1 Role of the legacy pipeline The P3-S25 pipeline is a REFERENCE, not an unquestionable oracle. Legacy cursor (chunks 738–759 / workload 45 completed, next chunk 760 at byte 18,442,355,762) is preserved and read-only. The new Engine does not inherit it. ### 24.2 Differential verification scope ``` NEW ENGINE ── selected byte ranges ──▶ canonical ticks (ts, bid, ask) ── selected ticks ─────────▶ row-level samples ── timestamps ─────────────▶ M15/M30 bar boundaries ── M15/M30 bars ───────────▶ OHLC/tick counts ── cumulative counters ────▶ parsed/canonical/malformed totals ── hashes ─────────────────▶ layer digests ▼ LEGACY EVIDENCE ``` Selection set (deterministic): first 64 KiB; the 24 MiB range preceding legacy chunk 759/760 boundary; and 8 evenly spaced sample ranges across the file (config-driven; each ≤ 24 MiB). ### 24.3 Discrepancy classification (mandatory taxonomy) | Class | Meaning | Auto-action | |---|---|---| | EXPECTED_SEMANTIC | documented spec difference (e.g., mid-based OHLC, UTC alignment, carry policy) | record, no alarm | | IMPLEMENTATION_DIFFERENCE | same semantics, different internals | record; verify both | | LEGACY_DEFECT | legacy evidence contradicts the source within known legacy scope | record; escalate (G-16) | | NEW_ENGINE_DEFECT | new Engine deviates from its own spec | FAILED; fix before proceeding | | UNRESOLVED | cannot classify from available evidence | record; escalate (G-16); never auto-assume | Output: `verification/legacy_diff_.json` with per-item range/tick/bar/counter/hash evidence and a classification. The Legacy Comparison stage never modifies legacy files and never chooses an answer on the operator's behalf; UNRESOLVED entries are G-16 material. ## 25. Security & Integrity Controls 1. Source is opened `rb` only; the Engine never holds a writable handle to the raw source and fails if `stat` changes during certification (`F09`). 2. All outputs live under `engine_output/`; path validation (resolved absolute paths under the configured write root) rejects `..`, symlink/junction escape, and absolute paths outside the root. 3. No network access, no credentials, no account data, no secrets, no eval of config (config validated against JSON schema). 4. Checkpoint and journal are hash-chained (`checkpoint_hash`, per-line `prev`); optional HMAC layer (key file outside repo) is a NON-BLOCKING future hardening (G-17). 5. Every produced artifact carries `sha256_physical` + `content_id` (§17); evidence manifest enumerates all files with both. 6. Legacy evidence directories are excluded from every access path the Engine may write; a runtime guard refuses any write whose target path is not under the Engine's own output root. 7. Logs are sanitized (no raw bid/ask values, no account identifiers). 8. Malformed sidecar rows contain hashes (`raw_sha256`), not raw text. ## 26. Versioning | Constant | Value (v1) | Bump rule | |---|---|---| | ENGINE_VERSION | 1.1.0 | any Engine behavior change (semver) | | PARSER_VERSION | PARSER_V1.1 | any change to CSV/timestamp/price parsing rules (P3-DE-004: extended with G_TICKSTORY_MT5; unambiguous vs the failed PARSER_V1) | | GRAMMAR REGISTRY | G_TICKSTORY_MT5 / G_DOTTED_V1 | named presets, certificate-pinned (P3-DE-004; §8.1) | | ALGORITHM_VERSION | ALG_V1 | any change to canonical serialization, chunking rule, or aggregation semantics (unchanged by P3-DE-004) | | SCHEMA_VERSION (ticks) | CTS_V1 | schema change | | SCHEMA_VERSION (bars) | CBS_V1 | bar schema change | | DATASET_VERSION | DS_V1.0.0 | human decision gate only (G-5) | | MANIFEST_SCHEMA_VERSION | MS_V1 | manifest schema change | | CHECKPOINT_SCHEMA_VERSION | CP_V1 | checkpoint schema change | | RUN_COMPLETE_SCHEMA_VERSION | RC_V1 | completion schema change | Determinism pins (recorded in every manifest): Python ≥ 3.11, pyarrow exact version, pandas (research layer) exact version, `PYTHONHASHSEED=0`, Parquet writer version string. Any pin change bumps ENGINE_VERSION and requires re-running the golden corpus and determinism tests before further runs. Version mismatch on resume = RESUME_BLOCKED (F10). No automatic data migration; schema migrations are new dataset versions (G-5). ## 27. Acceptance Criteria The Engine is accepted for full reprocessing only when ALL hold: 1. Golden corpus G01–G17 passes under both producer and verifier. 1a. Source-grammar corpus SG01–SG15 (G_TICKSTORY_MT5 six-column grammar) passes under both producer and verifier with the regression timestamp anchor (`20030505,00:01:03` → `1_052_092_863_000` ms at tz offset 0), the compact-grammar end-to-end pipeline run, and the certificate/grammar fail-closed cross-check all green. 1b. CLI headless E2E includes the run-less-directory regression: `status` and `resume` on an uninitialized output directory MUST both exit 4 (BLOCKED) without a traceback. 2. Independent verifier accepts the pilot (step 6). 3. Resume invariance: content ids of a run interrupted 3× and resumed are identical to an uninterrupted run of the same range. 4. `verify --mode dataset` recomputes all content ids and the dataset hash from disk independently and matches. 5. `verify --mode run-complete` returns exit 0 only on a genuine COMPLETED. 6. Differential report vs legacy is produced with every item classified. 7. Source certificate: FULL_SOURCE_SHA256_VERIFIED achieved or the run did not start without G-3 authorization. 8. Malformed rates within limits; malformed sidecar complete. 9. Research dataset regeneration: two runs of `build-dataset` with the same config produce identical dataset hashes without touching the raw source. 10. Engine completes a full 34.5 GB pass headlessly (AI closed), including at least one forced resume. 11. Legacy evidence directories byte-identical before/after full run. 12. Source file size and mtime unchanged by the full run. ## 28. Risks & Mitigations | # | Risk | Mitigation | |---|---|---| | R-TZ | Server-time/timezone ambiguity (DST) shifts bar boundaries | canonical ts is a pure function of text; re-derivation with new offset = new dataset version; manifest records offset; G-6 | | R-DISK | 34.5 GB source + canonical outputs exceed free space | preflight floor; per-workload floor; PAUSED(disk_space) | | R-HW | Disk/CPU failure mid-run | atomic commits; journal truth; resume | | R-AV | Antivirus/OneDrive locking files | rename+fsync pattern; documented deployment requirement; retry on transient EACCES (bounded) | | R-DETS | Library/dependency drift breaks determinism | pinned versions recorded; golden corpus gate; upgrade bumps ENGINE_VERSION | | R-MEM | Parent or worker memory blowup | chunk-bounded workers; memory telemetry; retry/FAILED policy | | R-DUP | Duplicate/overlapping data in source (Tickstory known cases) | taxonomy counters; verifier invariants; research layer documents dedup policy per dataset config | | R-LEG | Legacy evidence drift/incomplete | legacy dirs read-only; differential classification taxonomy | | R-GAP | Weekend/illiquid gaps produce zero-tick bars | bars exist only when ticks exist (no empty-bar inflation); gap analysis is research-layer | | R-CERT | Full SHA-256 too slow on target hardware | certification budget; VERIFIED_WITH_LIMITATION is explicit and gated (G-3) | | R-SRC | Source replaced/moved by another process | size/mtime re-checks; cert binding; RESUME_BLOCKED | | R-SPEC | Spec ambiguity discovered at implementation | every ambiguity must be filed as an Open Decision, not silently assumed | ## 29. Open Decisions Requiring Human Authorization Every item below is a HUMAN DECISION GATE. The Engine must NOT proceed past the gate without an explicit owner decision recorded in `docs/CURRENT_DECISION_GATE.md` (or successor register). | # | Decision | Recommended default | Consequence if not authorized | |---|---|---|---| | G-1 | Final storage format (Parquet + zstd-3 + pinned pyarrow) | accept default | init blocked | | G-2 | Default worker count (24; created = min(24, cpu_count, mem-constrained)) | accept default | ingest uses 1 worker until decided | | G-3 | Source certification acceptance: require FULLY_VERIFIED; allow VERIFIED_WITH_LIMITATION only with explicit authorization | FULLY_VERIFIED only | no run under limitation | | G-4 | Canonical schema CTS_V1/CBS_V1 (columns, PRICE_SCALE=1e6, mid-based OHLC with PRICE_SCALE_BAR=2e6, vol semantics) | accept default | canonicalization blocked | | G-5 | Dataset version identity & bump policy (DS_X.Y.Z, human-only bumps) | accept default | freeze/dataset publish blocked | | G-6 | Source timezone offset (default config `source_tz_offset_minutes`; DST not modeled in v1) | record + document | canonicalization blocked | | G-7 | Atomic chunk size (24 MiB nominal, line-safe) | accept default | chunkmap frozen with legacy grid | | G-8 | Workload size (512 MiB nominal) | accept default | workload grouping frozen | | G-9 | Aggregation semantics (UTC clock boundaries, mid OHLC, spread min/max/avg floor, carry policy) | accept default | bar layer blocked | | G-10 | Bar partial policy at EOF (flush as is_final=false) & no empty-bar inflation | accept default | bar layer publish blocked | | G-11 | Stale-lock release policy (default: refuse; `--force-release-lock` explicit) | refuse by default | blocked while stale lock present | | G-12 | Freeze legacy pipeline as reference-only | after G-1..G-10 | reprocessing not authorized | | G-13 | Pilot scope (default first 512 MiB window) | accept default | pilot not run | | G-14 | FULL 34 GB REPROCESSING authorization | after pilot verification | step 7 never starts | | G-15 | Freeze of canonical research dataset + publish hashes | after independent certification | step 12 not executed | | G-16 | Legacy-vs-new discrepancy resolution (per UNRESOLVED item) | adjudicate per item | run stays paused | | G-17 | Optional checkpoint HMAC hardening | non-blocking future | no impact on v1 | Additionally: any research target definition (Layer 9 configs such as M15+15m direction) is owner-authorized research configuration and is out of engine scope until G-14 completes. No gate is decided by the Engine. Approved governance clarifications (P3-DATA-ENGINE-001 GOV-CLOSE, human decision baseline 2026-09-07): - G-14 remains NOT AUTHORIZED even after design freeze. - G-6 approval does not define the actual UTC offset value; `source_tz_offset_minutes` must be established explicitly at source initialization. - G-12 is a legacy freeze/reference decision and does not mean the legacy checkpoint becomes a new-engine checkpoint. --- ## Appendix A — Final Review Gate (design self-check) | # | Question | Answer | |---|---|---| | 1 | Works with ChatGPT completely closed? | YES — headless CLI, no AI dependency anywhere in the pipeline (§4.3, §14) | | 2 | Windows restart interrupts without corrupting state? | YES — atomic tmp+rename, journal truth, orphan-tmp cleanup (§13) | | 3 | Resume without AI intervention? | YES — §13.5 procedure is deterministic `sniper-data resume` | | 4 | Source identity immutable? | YES — read-only handles, cert binding, F09 (§7, §25) | | 5 | Full source SHA requirement explicit? | YES — required; limitation is explicit + gated (§7.4–7.5, G-3) | | 6 | Canonical dataset reproducible? | YES — content-id contract independent of physical bytes (§12.4) | | 7 | Research datasets regenerable without re-reading raw ticks? | YES — canonical bar layer only (§15) | | 8 | Verifier independent of producer validation? | YES — separate implementation, no shared parse/aggregate code (§18) | | 9 | Discrepancy old vs new diagnosable? | YES — legacy comparison taxonomy (§24) | | 10 | Human decisions clearly identified? | YES — §29 gates G-1..G-17 | | 11 | Hidden dependency on future AI interaction? | NO — none | | 12 | Any path silently changing semantics? | NO — version gates, RESUME_BLOCKED, fail-closed states (§13, §20, §26) | Non-conformance items, if any, are filed in §29 rather than silently assumed. This table records DESIGN COMPLETE. ## Appendix B — Glossary | Term | Definition | |---|---| | chunk | line-safe byte range of the raw source (§9) | | workload | deterministic group of consecutive chunks (§9.2) | | content_id | canonical logical digest of a layer file (§12.4) | | journal | append-only hash-chained commit record (Layer 4/7) | | carry | in-progress partial bar per timeframe (§11.4) | | material condition | a failure that forces STOP and operator resolution (§20) | | expected class | a deterministic, counted, non-fatal data anomaly (§20) | --- *End of specification. This document is the only artifact produced by the design session. Nothing in it authorizes execution.* --- # Appendix C — P3-DATA-ENGINE-004 CONTROLLED AMENDMENT RECORD Session type: CONTROLLED SPECIFICATION AMENDMENT + IMPLEMENTATION RE-QUALIFICATION ONLY (authorized by the owner after the P3-DATA-ENGINE-003 pilot finding; DESIGN remains FROZEN; no unrelated governance decision is re-opened). All amendments below are traceable to the P3-DATA-ENGINE-003 evidence: source `XAUUSD_mt5_ticks.csv` (size 34,473,661,010 bytes; FULL SHA-256 `5252ce8f5b0e7d286b71a3172c7ef9c1009c8e4b1254ac67400a803fcd85cba1`; CRLF; ASCII; no header; first raw row `20030505,00:01:03,…`). ## C.1 Source grammar (authoritative; supersedes the 8.1 dotted primary) G_TICKSTORY_MT5 :: YYYYMMDD , HH:MM:SS , bid , ask , last , volume field count :: exactly 6 (else MALFORMED_FIELD_COUNT) date :: 8 digits, month 01..12, day 01..31 time :: HH:MM:SS, hh 00..23, mi 00..59, ss 00..59 (no ms) bid/ask/last :: decimal micro-price (PRICE_SCALE = 1e6; 0..6 fraction digits); non-positive -> MALFORMED_PRICE_NONPOSITIVE; ask < bid -> MALFORMED_BID_ASK_RELATION (mandatory) volume :: non-negative integer; 0 VALID; negative/non-integer -> MALFORMED_VOLUME (has_volume=true enforced) delimiter :: comma only line ending :: CRLF/LF/CR accepted deterministically; recorded header :: none in the real source; keyword-detected header (date|time|datetime prefix) skipped + has_header=true; absence verified by evidence Grammar selection is certificate-bound: `grammar_id` in `source_certificate.json` must equal the configured `source_grammar`, else `init` is BLOCKED (spec 8.1 certificate/config mismatch). No silent generic format acceptance. ## C.2 Timestamp `ts_ms = wallclock_epoch_ms(YYYYMMDD, HH:MM:SS, ms=0) - source_tz_offset_minutes * 60000`, integer civil-day math, deterministic. Production `source_tz_offset_minutes = 0` preserved (P3-DE-003 evidence chain: first row `20030505,00:01:03` -> epoch `1,052,092,863` s equivalent `1,052,092,863,000` ms == legacy chunk0 first_timestamp; G-6 policy; DST not modeled in v1). Regression-anchored by SG13. ## C.3 Volume / last-field / versions - Volume: see 8.2 amendment (mandatory source volume for G_TICKSTORY_MT5; `vol = 1` substitution FORBIDDEN for this source; zero valid). - `last`: validated; EXCLUDED from CTS_V1; preserved in `source_preservation/chunk_%06d.jsonl` (spec 8.6); promotion to the canonical schema = new CTS version + G-5 gate. - Versioning: ENGINE_VERSION 1.1.0; PARSER_VERSION PARSER_V1.1; ALGORITHM_VERSION ALG_V1 (unchanged); SCHEMA_VERSION CTS_V1/CBS_V1 (unchanged); DATASET_VERSION DS_V1.0.0 (unchanged; G-5). - G-9 (aggregation semantics) and G-10 (partial-bar policy): UNCHANGED. - G-14: UNCHANGED — BLOCKED / NOT AUTHORIZED. - Legacy state: checkpoint `759 -> 760 @ byte 18,442,355,762` UNCHANGED; workload 46 NOT prepared; legacy pipeline byte-identical and read-only. ## C.4 Implementation-visible changes - Parser/source-adapter: two named grammar presets (8.1 registry); compact timestamp parse; `last` validation + preservation sidecar; volume semantics; certificate `grammar_id` + `compact_date_time` sniffing. - CLI defect repair: `status`/`resume` on a run-less directory now exit 4 (BLOCKED) instead of exit 1 + traceback (P3-DE-003 finding; `NEW_ENGINE_DEFECT`, non-data-affecting; regression in CLI E2E). - Qualification suite extended: SG01–SG15 source-grammar golden cases + compact-grammar end-to-end + dependent acceptance criteria (27). ## C.5 Scope exclusions (unchanged) No pilot execution, no 34 GB source processing, no full-source hashing for ingestion purposes, no workload 46, no chunk 760+ preparation, no legacy checkpoint/pipeline modification, no research dataset generation, no forecasting, no G-14 authorization.