Watch

Chiki Bum cdf01aaabb P3-DATA-ENGINE-004: source-grammar correction + re-qualification (QUALIFIED) - spec 1.1.0 controlled amendment (Appendix C): G_TICKSTORY_MT5 six-col YYYYMMDD,HH:MM:SS,bid,ask,last,volume; compact timestamp; mandatory volume; last preserved via source-preservation sidecar (spec 8.6; CTS_V1 unchanged; G-4/G-5 intact); SHA effbaf2624cd46137c22a246fdabff4067052807b6b6224be464366bec4745d9 (machine-verified) - implementation: grammar registry, source_grammar config identity + fail-closed validation, compact timestamp parse, _parse_chunk_tickstory, preservation serialization + sidecars, certificate grammar_id + compact sniffing, init cert/config mismatch BLOCK, independent verifier v_parse_chunk_tickstory + preservation ids, evidence + verify_dataset preservation checks; ENGINE_VERSION 1.1.0, PARSER_V1.1 - CLI defect repair: status/resume on run-less dir exit 4 BLOCKED without traceback (P3-DE-003 NEW_ENGINE_DEFECT) + regression tests - qualification: unit, golden G01-G17, SG01-SG15 + e2e, property, adversarial, mutation 8/8, independent verifier dataset+run-complete ACCEPTED, legacy diff 0 unresolved legacy untouched, dataset-builder determinism, CLI E2E; VERDICT QUALIFIED - G-14 unchanged BLOCKED/NOT AUTHORIZED; no real-data ingestion; no workload 46; no chunk 760+; legacy checkpoint 759/18442355762 untouched

2026-09-07 17:34:57 +07:00

73 KiB

Raw Permalink Blame History

P3_DATA_ENGINE_V1_SPEC — SNIPERGOLD DATA ENGINE v1

Data Engine Architecture & Clean Reprocessing Specification

Field	Value
Document ID	P3_DATA_ENGINE_V1_SPEC
Version	1.1.0 (amended by P3-DATA-ENGINE-004; Appendix C authoritative for the amended clauses)
Status	DESIGN FROZEN + CONTROLLED AMENDMENT APPLIED (P3-DE-004); implemented & qualified
Session type	Amendment: source-grammar correction + implementation re-qualification
Authorized execution	Qualification only (NO real-data ingestion; NO G-14)
Classification	Internal research infrastructure

0. Executive Summary

This document defines SNIPERGOLD DATA ENGINE v1 ("the Engine"): a standalone, headless Python system that performs a clean, deterministic, resumable full reprocessing of the authoritative XAUUSD Tickstory tick source (XAUUSD_mt5_ticks.csv, current known size 34,473,661,010 bytes).

The Engine is designed so that the entire source processing lifecycle can complete with ChatGPT closed. The AI Assistant role is limited to research design, governance, audit, and interpretation. Python is the deterministic data processor.

This session produced this specification only. No real-data ingestion was performed, no checkpoint was modified, and no pipeline code was changed.

1. Purpose

Produce an implementation-ready architecture for a data engine that:

certifies the identity of the 34.5 GB raw tick source before any processing;
performs a clean full reprocessing of that source into a canonical, columnar, versioned, resumable dataset;
builds deterministic canonical time bars (M1, M5, M15, M30, H1);
builds independently reproducible research datasets from canonical data without ever re-reading the raw source;
runs headlessly, survives interruption (including Windows restarts), and resumes without AI assistance;
exposes full auditability: every layer has a deterministic identity and every artifact is independently verifiable.

2. Scope

The Engine covers:

source certification (size, mtime, full SHA-256, format facts);
deterministic line-safe chunking of the raw source;
canonical tick parsing, classification, serialization, hashing;
parallel canonical tick conversion with deterministic merge semantics;
canonical bar aggregation (M1, M5, M15, M30, H1);
persistent canonical storage (Parquet by default; see Human Decision Gate G-1);
standalone checkpoint / resume with atomic fail-closed transitions;
headless CLI (sniper-data), config, manifests, evidence and hashing;
independent verification tooling, golden test corpus, failure model, resource management, completion protocol, clean reprocessing protocol, legacy comparison protocol;
research dataset builder driven strictly by research configuration.

3. Non-Goals

The Engine MUST NOT:

require an AI Assistant session (or any human) to remain active;
make trading decisions, optimize a trading strategy, or select models;
chase a "successful" timeframe or perform forecasting;
read the legacy checkpoint as its own execution state (legacy checkpoint 759 is preserved and untouched; the new Engine starts its own lifecycle);
reuse the legacy parser/aggregator code as its implementation;
silently downgrade certification requirements;
auto-recover from material failures by reasoning;
use CSV as a default intermediate storage format;
write to, delete, or modify any legacy pipeline directories or evidence.

4. Architecture

4.1 Logical layers

#	Layer	Responsibility	Immutable contract
0	RAW SOURCE	immutable Tickstory CSV	never modified by the Engine
1	SOURCE CERTIFICATION	identity + format facts	certificate JSON
2	CANONICAL TICK REPRESENTATION	raw row → canonical record	canonical schema + hash
3	CHUNKING	line-safe byte ranges	chunk map
4	PROCESSING MODEL	parallel deterministic conversion	chunk output files
5	AGGREGATION	tick → M1/M5/M15/M30/H1 bars	bar files + carries
6	PERSISTENT CANONICAL DATASET	Parquet layer	dataset manifest
7	CHECKPOINT / RESUME	atomic state	checkpoint + commit journal
8	HEADLESS CLI	operator interface	exit codes
9	RESEARCH DATASET BUILDER	config-driven derived data	research dataset + manifest
10	DATASET MANIFEST	reproducibility record	manifest JSON
11	EVIDENCE & HASHING	identity at every layer	evidence manifest
12	INDEPENDENT VERIFICATION	non-producer verification	verification report
13	GOLDEN TEST CORPUS	adversarial micro-inputs	expected outputs
14	FAILURE MODEL	three-class fail-closed behavior	state transitions
15	RESOURCE MANAGEMENT	bounded local execution	telemetry
16	RUN COMPLETION PROTOCOL	machine-verifiable completion	RUN_COMPLETE.json
17	CLEAN REPROCESSING PROTOCOL	the authorized future procedure	step gates

4.2 Design invariants (binding)

Determinism: canonical output is a pure function of (source bytes, engine version, parser version, algorithm version, config snapshot). Worker count, scheduling, result arrival order, disk layout and wall-clock time NEVER affect output.
Fail-closed: every transition is atomic; on doubt the system stops in a documented state. There is no silent fallback and no autonomous recovery-by-reasoning.
Resumable: any chunk is either fully committed (file + journal record + digest) or not committed at all. A hard power-loss leaves at most an orphan .tmp file that resume deletes and re-dispatches.
Source immutability: the Engine opens the source read-only (rb), never writes it, and re-checks size+mtime at every resume.
Integer-only canonical math: the canonical path (parse → serialization → aggregation) uses integer arithmetic only. No floating point anywhere in canonical ingestion or bar aggregation.
Append-only outputs: committed chunk/bar files are never rewritten in place. Resume appends or skips; it never mutates committed data.
Independent verification: the verifier is a separate implementation (different parsing and aggregation code), not a re-run of producer code.

4.3 Execution topology

sniper-data CLI (parent/operator process)
  ├── chunkmap scanner        (reads source bytes for newline boundaries)
  ├── certification           (size/mtime/SHA-256 + format sniff)
  ├── dispatcher              (assigns chunk indices to workers)
  │     ├── worker process 1..N   (parse chunk -> chunk parquet + digests)
  │     └── ...                (spawn context; no shared state)
  ├── commit journal          (append-only hash-chained commit records)
  ├── checkpoint writer       (atomic tmp+rename)
  ├── sequential aggregator   (canonical ticks -> bars, carry-aware)
  ├── research builder        (canonical bars -> research datasets)
  ├── verifier                (independent implementation)
  └── completion/finalization (evidence + RUN_COMPLETE.json)

No component talks to a network. No component requires an interactive session.

5. Data Flow

RAW SOURCE (34.5 GB CSV, read-only)
   │  [certify]  size, mtime, FULL SHA-256, format facts
   ▼
source_certificate.json            (Layer 1)
   │  [chunkmap]  newline-scan only
   ▼
chunk_map.json                     (Layer 3)
   │  [parse+canonicalize, parallel]
   ▼
ticks/chunk_%06d.parquet  +  malformed/malformed_%06d.jsonl   (Layer 2/4/6)
   │  [ordered merge; sequential aggregator, carry-aware]
   ▼
bars/<TF>/wl_%03d.parquet          (Layer 5)
   │  [dataset manifest + content digests]
   ▼
dataset manifest (canonical)        (Layer 6/10)
   │  [research config]
   ▼
datasets/<dataset_id>/…            (Layer 9)
   │  [evidence + hash]
   ▼
evidence.json + RUN_COMPLETE.json  (Layer 11/16)

Every layer records its inputs (source_id, versions) so each stage can be recomputed and compared. Research datasets are built from the canonical bar layer ONLY — never re-reading the 34 GB raw source.

6. Component Design

6.1 Module map (implementation target)

engine/
  cli.py                 # argument parsing, command dispatch, exit codes
  config.py              # config load + strict schema validation
  versions.py            # canonical version constants (see §26)
  certify.py             # Layer 1
  chunkmap.py            # Layer 3 (newline scanning; pure function of bytes)
  parse.py               # Layer 2 producer parser (optimized, integer-only)
  canonical.py           # canonical record serialize/hash (Layer 2)
  worker.py              # Layer 4 worker entry point (spawn-safe)
  dispatcher.py          # Layer 4 dispatch + retry + telemetry
  journal.py             # commit journal (append-only, hash-chained)
  checkpoint.py          # Layer 7 atomic checkpoint read/write
  aggregate.py           # Layer 5 sequential deterministic aggregator
  storage.py             # Layer 6 parquet writers, naming, schema metadata
  dataset_builder.py     # Layer 9 research dataset builder
  manifest.py            # Layer 10 manifest build/verify
  evidence.py            # Layer 11 hashing + evidence manifest
  run_complete.py        # Layer 16 completion protocol
  verify/                # Layer 12 INDEPENDENT verifier (separate impl)
    __init__.py
    vparse.py            # separate CSV/line parser written from spec
    vaggregate.py        # separate aggregator written from spec
    vinvariants.py       # invariant checks
    vcompare.py          # differential comparison
  tests/golden/          # Layer 13 corpus + expected outputs
    cases/ expected/
  tests/mutations/       # mutation test driver + cases

6.2 Responsibilities matrix

Component	Reads	Writes	Fails closed by
certifier	source (rb)	`certification/source_certificate.json`	never writing source; recording status
chunkmapper	source (rb)	`chunkmap/chunkmap-<id>.json`	deterministic scan; no parsing
worker	source range, chunk map	`staging/chunk_%06d.parquet.tmp` → `ticks/chunk_%06d.parquet`; `malformed/...`	one chunk per process; tmp+rename; digest computed before rename
dispatcher	chunk map, journal	journal entries, `progress.json`	bounded retries then FAILED
aggregator	ticks parquet (ordered)	`bars/<TF>/wl_%03d.parquet`; carries in checkpoint	append-only; carry continuity check on resume
verifier	source ranges, artifacts	`verification/...`	never reusing producer parse/aggregate code
finalizer	everything	`evidence.json`, `RUN_COMPLETE.json`	only from COMPLETED + verified state

6.3 Configuration

Config is a strict-schema JSON file passed at init. Fields relevant to data identity: source_path, source_tz_offset_minutes, has_volume, timeframes. Fields relevant to operation: workers_requested, chunk_bytes_nominal, workload_bytes_nominal, checkpoint_every_workload, retry_limit, memory_limits, disk_floor_bytes, timeouts. Every config field that affects output is captured in the dataset manifest (config_snapshot

config_sha256). Operational-only fields are excluded from config_sha256.

7. Source Certification (Layer 1)

7.1 Purpose

Before any processing, the Engine certifies that the configured source file is the authorized, unmodified Tickstory source.

7.2 Certificate schema

certification/source_certificate.json:

schema_version        "CERT_V1"
certification_timestamp  ISO-8601 UTC
certification_status     FULLY_VERIFIED | VERIFIED_WITH_LIMITATION | FAILED
source_path              absolute path as configured
source_size_bytes        integer, stat size
source_mtime_ns          integer, stat mtime nanoseconds (Windows st_mtime_ns)
sha256_full              hex (only present when the full file was hashed)
sha256_mode              "full" | "not_computed"
limitation_reason        null | enum(see 7.4)
source_format            "csv_tickstory_mt5" | "unrecognized" | "mismatch"
grammar_id               "tickstory_mt5_v1" | "dotted_v1" | null (P3-DE-004: pins the source-adapter preset; init cross-check §8.1)
line_ending              "CRLF" | "LF" | "CR" | "MIXED" (with ratio record)
encoding                 detected (BOM-aware) ; "utf8" | "ascii" | other
timestamp_format_detected  "compact_date_time" | "dotted_ms" | "dotted_s" | "unrecognized"
delimiter                "," (only comma supported in v1)
has_header               bool (detection rule §8.3)
column_layout            {"fields":N,"names":[...]} observed; 3/4 => [datetime,bid,ask(,vol)]; 6 => [date,time,bid,ask,last,volume]
parser_version           (constants §26)
engine_version           (constants §26)

7.3 Certification procedure

Verify path exists and is a regular file.
Record size and mtime_ns.
Stream the file once, rb, in fixed 8 MiB buffers, computing SHA-256 over ALL bytes, and simultaneously sniffing: first 1 MiB (line terminator classification), first 1000 parseable line-shaped rows (timestamp/delimiter/ header detection), BOM presence.
On success: write certificate with status FULLY_VERIFIED and the full hash.
If the configurable time budget (certify_timeout_sec, default 3600; config option) elapses before the hash completes, the process stops cleanly and records VERIFIED_WITH_LIMITATION with sha256_mode="not_computed" and limitation_reason="timeout". It MUST NOT claim a full hash.
Write is atomic (*.tmp + fsync + rename). Certificate path is derived from source identity fields; re-certification is an allowed overwrite, always recorded with a new timestamp.

7.4 Certification statuses

Status	Meaning	Consequences
FULLY_VERIFIED	full file hashed; format facts consistent	canonical pipeline eligible (subject to Gate G-3)
VERIFIED_WITH_LIMITATION	full SHA-256 not completed within budget; only size/mtime/format facts guaranteed	pipeline MUST NOT claim full certification; starting a run requires explicit human authorization (Gate G-3)
FAILED	size/mtime changed mid-read, unreadable, or format facts self-contradictory	RESUME_BLOCKED; no processing

A size or mtime change detected between the pre-stat and post-stat is an automatic FAILED with limitation_reason="source_changed_during_certify".

7.5 Operational requirement

FULL SOURCE SHA-256 VERIFIED is the required precondition of the canonical pipeline. Any decision to proceed with VERIFIED_WITH_LIMITATION is a HUMAN DECISION GATE (G-3). The Engine never silently downgrades this requirement.

8. Canonical Tick Representation (Layer 2)

8.1 CSV interpretation (deterministic rules)

Encoding: UTF-8 (a leading UTF-8 BOM is stripped). Any undecodable byte sequence marks the row MALFORMED_ENCODING. ASCII is a strict subset and is processed identically.
Delimiter: comma only in v1. Detection result is recorded in the certificate; a mismatch between certificate and config blocks init.
Line endings: the row scanner splits on any of \r\n, \n, \r (deterministic; independent of the recorded classification). The recorded line_ending is diagnostic.
Blank lines: skipped, counted in malformed.blank_line (expected class).
Header: after BOM strip, if the first non-blank row has a first field matching ^(datetime|date|time)([^0-9]|$) (case-insensitive), that row is the header and is skipped with has_header=true recorded in the certificate. The header row boundary is deterministic.
Grammar presets (P3-DATA-ENGINE-004 source adapter): exactly two named, fully-specified, versioned grammar presets exist: G_TICKSTORY_MT5 (six-column YYYYMMDD,HH:MM:SS,bid,ask,last,volume; the authoritative production grammar, derived from the P3-DATA-ENGINE-003 pilot evidence) and G_DOTTED_V1 (dotted 3/4-column corpus grammar, retained for the synthetic G01–G17 fixtures). The certificate pins grammar_id from the observed facts; the operator config source_grammar MUST equal the certified grammar_id or init is BLOCKED (certificate/config mismatch, spec 8.1). There is no silent per-row format guessing.
Timestamp: accepted forms (recorded at certification):
- YYYYMMDD + HH:MM:SS (compact date + time fields; G_TICKSTORY_MT5) — PRIMARY;
- YYYY.MM.DD HH:MM:SS.mmm (dotted, 3-digit milliseconds; G_DOTTED_V1) — corpus preset;
- YYYY.MM.DD HH:MM:SS (no ms) — corpus preset, ms = 0. Any other shape → MALFORMED_TIMESTAMP_PARSE.
Timezone: source text is a wall-clock in the trader's server time. Canonical ts_ms = epoch milliseconds UTC computed as: ts_ms = julian_day_ms(wall_clock) - source_tz_offset_minutes * 60000. source_tz_offset_minutes is an explicit config value recorded in the manifest; DST is NOT modeled in v1 (see Open Decisions, risk R-TZ). The parse→canonical mapping is therefore a pure, documented function of the source text, so any future re-interpretation is a re-derivation with a new dataset version, never an in-place mutation.
Timestamp range: valid only if 2000-01-01T00:00:00Z <= ts_ms <= certify_time+7d; outside → MALFORMED_TIMESTAMP_RANGE.

8.2 Price, bid/ask, volume semantics

Canonical prices are scaled integers: PRICE_SCALE = 1_000_000 (micro-USD units). bid_u = int(decimal_text * PRICE_SCALE) computed by exact decimal-string parsing (no floats).
ask_u likewise. spread_u = ask_u - bid_u.
Invalid relationships:
- bid_u <= 0 or ask_u <= 0 → MALFORMED_PRICE_NONPOSITIVE;
- spread_u < 0 → MALFORMED_BID_ASK_RELATION;
- fractional digits beyond the scale → MALFORMED_PRICE_PRECISION.
Volume (amended P3-DATA-ENGINE-004): the grammar presets fix the volume semantics — for G_TICKSTORY_MT5 the source HAS an explicit mandatory volume column so has_volume=true is enforced by config validation; vol = 1 substitution (has_volume=false) is FORBIDDEN for this source. Volume is a non-negative integer token; 0 is VALID (observed in the real source: many canonical rows carry volume 0); negative or non-integer values are MALFORMED_VOLUME. The dotted preset retains has_volume=false (default) → vol = 1 tick-volume semantics. Any violation of the certificate's column_layout vs config → init block.
Duplicate detection: identical (ts_ms,bid_u,ask_u,vol) records within one chunk are flagged MALFORMED_DUPLICATE (bounded in-memory set per chunk).
Monotonicity: within a chunk, ts_ms must be non-decreasing; otherwise the offending row is MALFORMED_NON_MONOTONIC. Across a chunk boundary the first ts_ms of chunk N must be >= the last ts_ms of chunk N-1 (carried in checkpoint); a violation is counted as MALFORMED_NON_MONOTONIC_BOUNDARY and, if > threshold, becomes material (§20).

8.3 Malformed-record taxonomy (complete)

Class	Triggers
MALFORMED_EMPTY_LINE	blank row
MALFORMED_ENCODING	undecodable bytes
MALFORMED_FIELD_COUNT	field count ≠ grammar-expected (G_DOTTED_V1: 3/4; G_TICKSTORY_MT5: exactly 6)
MALFORMED_TIMESTAMP_PARSE	timestamp text unrecognized
MALFORMED_TIMESTAMP_RANGE	ts outside validity window
MALFORMED_PRICE_PARSE	bid/ask non-numeric
MALFORMED_PRICE_PRECISION	decimals exceed PRICE_SCALE precision
MALFORMED_PRICE_NONPOSITIVE	bid<=0 or ask<=0
MALFORMED_BID_ASK_RELATION	ask < bid
MALFORMED_VOLUME	volume invalid (when has_volume)
MALFORMED_DUPLICATE	exact duplicate within chunk
MALFORMED_NON_MONOTONIC	ts regress within chunk
MALFORMED_NON_MONOTONIC_BOUNDARY	ts regression across chunk boundary

Malformed rows are NEVER fabricated, skipped silently, or injected into the canonical stream. Each is appended to malformed/malformed_%06d.jsonl (chunk_index, byte_start, global_line_no, class, raw_sha256, reason). Counters per class accumulate in the checkpoint and in malformed_summary.json at completion. If malformed rows exceed malformed_rate_limit (default 0.001 of parsed rows in a workload) OR the absolute cap (default 10,000 in a workload), the run enters FAILED (§20).

8.4 Canonical record (CTS_V1)

Column	Type	Semantics
ts_ms	int64	epoch ms UTC
bid_u	int64	micro-USD × 1e6
ask_u	int64	micro-USD × 1e6
spread_u	int64	ask_u − bid_u (≥ 0)
vol	int64	source volume or 1
src_line	int64	global 1-based raw line number (audit anchor)
rec_ord	int64	0-based record ordinal within the chunk

All values are integers. No floats, no nulls, no strings in the canonical tick table.

8.5 Canonical serialization & hashing

The canonical serialization of a record r is the single line:

ts_ms|bid_u|ask_u|vol

(no spaces, \n terminated). The chunk content digest is:

chunk_content_id = SHA-256( concat(serialize(r_0), serialize(r_1), …, serialize(r_{n-1})) )

in rec_ord order. The chunk content id is the reproducibility identity of the chunk (Parquet physical bytes are not part of the reproducibility contract, because physical writer metadata may differ run-to-run; the logical serialization is fixed by this spec). The chunk physical SHA-256 is also recorded for integrity evidence.

8.6 LAST-field treatment and source-preservation layer (P3-DE-004)

The G_TICKSTORY_MT5 source carries a fourth price column last (last traded/ last quote price) that does NOT map into CTS_V1 (bid/ask/spread/vol). Decision chain, based on preservation, reproducibility, future research usefulness and schema integrity:

Validated at parse time with the same decimal micro-price grammar as bid/ask (PRICE_SCALE; non-positive or over-precision → the corresponding MALFORMED_PRICE_* class). Bid/ask relationship validation remains mandatory; last is deliberately NOT cross-validated against bid/ask (a last price may legally equal bid, ask or any value; imposing an invariant would be an unsupported assumption).
Deliberately EXCLUDED from CTS_V1: the canonical tick schema (and its serialization/content-id contract) is UNCHANGED — P3-DE-004 does not re-open G-4/G-5. Promoting last into the canonical schema would be a new CTS version and a new dataset version (human gate G-5).
Preserved in a source-preservation layer: source_preservation/chunk_%06d.jsonl, one line per canonical row in rec_ord order, src_line|last_u (e.g. 12|340345000). The canonical preservation serialization is the single line src_line|last_u\n; the preservation chunk content id is SHA-256 over the concatenation (micro-units, integer math only). The sidecar content id and physical SHA-256 are recorded in the evidence manifest and independently re-verified (spec 18).
Evidence basis: the immutable raw source (certified FULL SHA-256) remains the ultimate preservation record; the sidecar makes last recoverable from canonical-run artifacts without re-reading the 34 GB source, and keeps future research (e.g., last-price labels) derivable under the frozen canonical identity. Dotted-grammar runs produce no sidecars.

9. Chunking (Layer 3)

9.1 Evaluation of legacy values

Legacy atomic chunk = 24 MiB, workload = 512 MiB nominal. Evaluation:
- 24 MiB ⇒ ~1,370 chunks for the 34.5 GB source; each chunk holds ~0.6–0.9 M tick lines (~20–40 MB RSS per worker) — excellent checkpoint granularity, bounded worker memory, fine-grained resume.
- 512 MiB nominal workloads ⇒ ~67 workloads; matches legacy differential comparison granularity (chunk 759/760 boundary lies on the same grid).
- Both values are RETAINED as nominal defaults for v1. Actual boundaries are line-safe (below). These remain a human decision gate (G-7/G-8) because they affect the differential comparison surface.

9.2 Chunk map (deterministic)

A single newline-scan pass over the source (no parsing) produces chunkmap/chunkmap-<source_id>.json:

{"source_id", "chunk_bytes_nominal":25165824, "workload_bytes_nominal":536870912,
 "chunks":[ {"index":i, "byte_start":b, "byte_end":e, "line_count":n,
             "workload_index":w} , …],
 "total_bytes", "total_chunks", "total_workloads", "chunkmap_sha256"}

Rules:

byte_start is 0-based from file start. Chunk 0 starts at byte 0.
A chunk's nominal end = byte_start + chunk_bytes_nominal; the actual byte_end = first line-terminator position at or after the nominal end, INCLUSIVE of the terminator bytes. Consequence: byte_end is always a line boundary; no chunk ever splits a CSV record.
Chunk N+1 byte_start = chunk N byte_end. Boundaries are contiguous and cover [0, total_bytes) exactly.
If the source does not end with a newline, the final line is terminated by EOF; the last chunk ends at total_bytes.
A file smaller than one chunk yields exactly one chunk.
Workload grouping: accumulate whole chunks whose cumulative byte_end − byte_start first reaches or exceeds workload_bytes_nominal; the final workload contains the remaining chunks (partial final workload) and is recorded as partial=true. No padding.
chunkmap_sha256 = SHA-256 of the deterministic JSON serialization (sorted keys, fixed field order). The chunk map is cached per source_id; on resume it is recomputed cheaply only if the source identity changed (which blocks).

9.3 Byte cursor semantics

Checkpoint records next_chunk and next_byte_start (= byte_start of the next uncommitted chunk). They always agree with the chunk map; the chunk map is the sole authority for boundaries.

10. Processing Model (Layer 4)

10.1 Model selection

Canonical tick conversion: parallel multiprocessing (spawn context, Windows-safe), one chunk per worker task.
Aggregation: sequential deterministic pass over ordered canonical ticks (see §11). Parallel aggregation is explicitly NOT in v1; it is a documented future optimization that must preserve identical content ids and pass the reordered-execution determinism test before adoption.
Rationale: conversion is CPU/parse-bound (parallel-friendly, pure per chunk); aggregation is stateful and order-dependent (sequential is the safest deterministic design for v1).

10.2 Worker & dispatch model

workers_requested (config, default 24 — Gate G-2);
workers_created = min(requested, os.cpu_count() or 8, memory estimates) — recorded, never assumed;
workers_with_non_empty_partitions = workers that actually received ≥ 1 chunk before the queue drained;
cpu_utilization_pct = measured (parent-side polling), reported but never tuned automatically.
Dispatch: parent keeps a deterministic ordered work list (chunk indices 0..N−1) and a simple job queue (process pool). A worker pulls exactly one chunk at a time, processes it, writes staging/chunk_%06d.parquet.tmp, computes digests, fsyncs, renames to ticks/chunk_%06d.parquet, returns (chunk_index, stats). The parent records the commit in the journal (hash-chained). Completion order NEVER affects outputs: each chunk file is independently named and content-identified; the canonical reading layer iterates chunk indices in ascending order.
No shared memory between workers; no queues carrying tick data; workers only communicate via files.

10.3 Deterministic merge

"Merge" is a sorted concatenation by chunk_index (and by rec_ord within a chunk). The aggregator reads chunk files in ascending index order. Because chunk files are immutable once committed, merged output is identical for every arrival order.

10.4 Worker failure handling (bounded)

Event	Action (exactly one predefined recovery)
worker exits non-zero before rename	redispatch chunk (attempt counter); after `retry_limit=2` → FAILED
worker killed (crash/OOM)	redispatch (attempt counter as above); stale `.tmp` deleted by dispatcher before redispatch
chunk file exists but journal has no record	journal is truth; file re-verified; if digest matches → journal repaired to record commit (recorded `repaired=commit`); else re-dispatched
journal has record but file missing/corrupt	FAILED (material; no silent re-issue)
worker OOM repeatedly	attempt counter applies; then FAILED with `reason=worker_oom`

Commit is defined as: chunk parquet renamed and fsynced AND journal line appended and fsynced. Only both count.

11. Aggregation (Layer 5)

11.1 Timeframes and justification

Included: M1, M5, M15, M30, H1. Justification:

M1 is the finest bar technically justified by tick input and is the base unit for all coarser frames;
M5..H1 form the standard ladder required by the research plans for horizons of 15/30/60 min and by the legacy differential protocol (M15/M30 are legacy comparison anchors);
no other timeframe is added; "interesting" frames are explicitly out of scope until a research decision demands one (then a new dataset version).

Periods (ms): M1=60_000, M5=300_000, M15=900_000, M30=1_800_000, H1=3_600_000.

11.2 Boundary definition

Bar boundaries are UTC clock boundaries: period_id = floor(ts_ms / period_ms); start_ms = period_id * period_ms; end_ms = start_ms + period_ms.
A tick belongs to the bar of its period_id.
Ticks are consumed in canonical order (ts_ms ascending, then src_line ascending) — stable and deterministic.
Within identical ts_ms, order is by src_line (raw file order).

11.3 OHLC / volume / spread (integer-exact)

The representative bar price is the mid: mid_u2 = bid_u + ask_u (units: 2×PRICE_SCALE = 2,000,000 per USD). This keeps all OHLC math in integers with no truncation:
- open_u2 = first mid_u2 in the bar;
- high_u2 = max mid_u2; low_u2 = min mid_u2; close_u2 = last mid_u2;
- PRICE_SCALE_BAR = 2_000_000 is recorded in the bar schema metadata.
Volume: vol_sum = Σ tick vol; tick_count = number of ticks.
Spread: spread_min_u, spread_max_u, spread_sum_u, spread_avg_u = spread_sum_u // tick_count (floor division; defined).
Audit anchors: first_src_line, last_src_line.

11.4 Carry state

For each timeframe the aggregator keeps exactly one in-progress bar (the partial bar) as the carry:

TF carry = {period_id, start_ms, open_u2, high_u2, low_u2, close_u2,
            tick_count, vol_sum, spread_min_u, spread_max_u, spread_sum_u,
            last_ts_ms, last_src_line}

Carries are checkpointed with every commit (see §13). On resume the aggregator verifies that its first input ts_ms is ≥ the carried last_ts_ms; otherwise FAILED (carry mismatch = material).

11.5 Finalization rules

A bar is finalized when the first tick of a later period_id arrives. It is written to bars/<TF>/wl_%03d.parquet with is_final=true.
At EOF, all carried partial bars are flushed with is_final=false and the dataset manifest records final_bar_partial=true for each timeframe.
Bar uniqueness key: (TF, period_id). Append-only: a period_id is written at most once per run. On resume, the aggregator validates that the last written record for each TF equals the carried period_id (continuity check); mismatch → FAILED.
Bar global ordinal bar_idx (0-based per timeframe, contiguous across parts) makes reading order-independent.

11.6 Bar schema (CBS_V1)

Column	Type	Semantics
bar_idx	int64	global ordinal per timeframe
period_id	int64	start_ms / period_ms
start_ms / end_ms	int64	UTC boundaries
open_u2, high_u2, low_u2, close_u2	int64	units = 2 × PRICE_SCALE
tick_count, vol_sum	int64	counts
spread_min_u, spread_max_u, spread_sum_u, spread_avg_u	int64	spread in PRICE_SCALE units
is_final	bool	false only for EOF partial bars
first_src_line, last_src_line	int64	audit anchors

12. Storage Format & Persistent Canonical Dataset (Layer 6)

12.1 Format decision

Default: Apache Parquet, compression zstd level 3, write_page layout with pinned writer version (see §26 determinism table).

Rationale: columnar, typed (no float in canonical tabsh), compressed, stream-appendable by immutable files, universally readable, and row-group scannable. CSV is explicitly rejected as the default intermediate format (no types, slow, no random access, ambiguous escaping). PyArrow version is pinned and recorded in every manifest.

12.2 Layout

engine_output/
  certification/source_certificate.json
  chunkmap/chunkmap-<source_id>.json
  ticks/chunk_%06d.parquet              # final per-chunk columnar files
  staging/…                             # .tmp intermediates (prunable after verification)
  bars/M1/wl_%03d.parquet , bars/M5/… , bars/M15/… , bars/M30/… , bars/H1/…
  malformed/malformed_%06d.jsonl
  state/checkpoint.json, state/commits.jsonl, state/lock, state/control.json
  logs/*.jsonl
  progress.json
  evidence/evidence.json
  RUN_COMPLETE.json
  datasets/<dataset_id>/…
  verification/…

12.3 Naming rules

chunk_%06d (0-padded index), wl_%03d (workload index), part derived deterministically from workload index.
Every parquet file embeds schema key-value metadata: schema_version, engine_version, parser_version, algorithm_version, source_id, chunk_index (ticks) / timeframe (bars), price_scale, content_id is NOT embedded (it is computed over logical rows; see §12.4).

12.4 Dataset identity (content ids, not physical bytes)

content_id(ticks chunk i) = chunk_content_id from §8.5.
content_id(bars wl w, TF) = SHA-256 over canonical bar serialization: each row as bar_idx|period_id|open_u2|high_u2|low_u2|close_u2|tick_count|vol_sum|spread_min_u|spread_max_u|spread_sum_u|is_final|first_src_line|last_src_line\n concatenated in ascending bar_idx.
The canonical dataset hash = SHA-256 over the canonical concatenation (fixed field order) of content_id(ticks chunk 0) ‖ content_id(chunk 1) ‖ … ‖ content_id(bars files…) in sorted path order. Ordering is defined by the manifest file list.
Physical file SHA-256s are recorded alongside content ids in evidence.json; content ids are the reproducibility identity; physical hashes are integrity evidence. This distinction is explicit and non-negotiable (Parquet physical bytes are writer-dependent).

12.5 Schema metadata & versioning

Manifest record per dataset: {"dataset_version": "DS_V1.0.0", schema_version, engine_version, parser_version, algorithm_version, source_id, config snapshot}. Dataset version bumps are HUMAN DECISION GATES (G-5).

13. Checkpoint / Resume (Layer 7)

13.1 Files

state/checkpoint.json — atomic (tmp+fsync+rename+dir fsync).
state/commits.jsonl — append-only commit journal, each line hash-chained to the previous: {"seq":n,"chunk":i,"content_id":"…","file_sha256":"…","prev":"<hash of line n-1>","ts_utc":"…"}
state/lock — PID, run_id, heartbeat. Stale detection §21.

13.2 Checkpoint schema (CP_V1)

schema_version, run_id, status,
source_identity {path,size,mtime_ns,sha256_full,cert_status,cert_path},
versions {engine, parser, algorithm, dataset, manifest schema},
chunkmap_sha256,
journal_tail {seq, hash},
last_completed_chunk, next_chunk, next_byte_start,
cumulative {rows_parsed, rows_canonical, bytes_read, malformed:{class:count}, chunks_committed,
            bars_per_timeframe:{TF:count}},
carries {TF: carry object (§11.4)},
update_ts_utc,
status_reason (nullable),
checkpoint_hash   (SHA-256 over the canonical JSON serialization of all fields
                   EXCEPT checkpoint_hash — non-self-referential)

13.3 Status machine

INITIALIZED → RUNNING → CHUNK_COMMITTED* → PAUSED → RUNNING → … → COMPLETED
                  │                                          │
                  └──────────── FAILED ──────────────────────┘
INITIALIZED / RUNNING / PAUSED / FAILED ──(resume guard)──▶ RESUME_BLOCKED

INITIALIZED — after init, before first dispatch.
RUNNING — while workers are dispatched.
CHUNK_COMMITTED — recorded transition on every journal append (the durable state of progress; the file is rewritten with this status after each workload, see 13.4).
PAUSED — graceful stop (signal/control file), clean checkpoint, exit 0.
FAILED — material condition; processing stopped; resume is permitted only after operator resolution and explicit re-run (see §20).
COMPLETED — final after verification; final checkpoint written.
RESUME_BLOCKED — a resume attempt refused entry; no state mutation.

Transitions are atomic (single rename) and fail-closed: any exception during a checkpoint write leaves the previous valid checkpoint untouched.

13.4 Checkpoint frequency

Checkpoint + journal fsync after every workload (~67 writes for the full run) and always before graceful pause/stop and after any worker failure that causes re-dispatch. Chunk commit (journal append) is the fine-grained durable record; the checkpoint file is the coarse summary. On resume, journal replay (from the recorded tail) determines actual committed chunks — including chunks committed after the last checkpoint write.

13.5 Resume procedure (deterministic)

Load checkpoint; verify checkpoint_hash.
Verify source identity: recompute size+mtime; require certificate present and matching (sha256_full if FULLY_VERIFIED). Any mismatch → RESUME_BLOCKED (never auto-recovery).
Verify versions (engine/parser/algorithm/dataset/schema) match exactly → mismatch = RESUME_BLOCKED.
Verify chunkmap_sha256 matches the cached map → mismatch = RESUME_BLOCKED.
Replay journal from journal_tail.seq; validate chain; derive next_chunk; verify each committed chunk file exists and its content id matches the journal → any anomaly = FAILED (material), not auto-fixed.
Restore carries from checkpoint; verify aggregator continuity (§11.5).
Resume dispatch from next_chunk.

13.6 Source identity mismatch

source_identity mismatch → RESUME_BLOCKED with status_reason (source_size_changed | source_mtime_changed | certificate_missing | certificate_mismatch | sha256_mismatch). The operator must re-certify and decide (human gate) whether to start a new run.

14. Headless CLI (Layer 8)

14.1 Commands (semantics, not implemented this session)

sniper-data --version
sniper-data certify    --source <path> [--timeout-sec N] [--output <cert>]
sniper-data init       --config <engine_config.json> [--run-id <id>]
sniper-data ingest     [--limit-chunks N] [--pause-after <workload|chunk> N]
sniper-data resume     [--limit-chunks N]
sniper-data pause      | stop
sniper-data status     [--json]
sniper-data verify     --mode golden | range | dataset | run-complete | legacy
                       [--start-byte N --end-byte M] [--samples N]
sniper-data manifest   [--dataset <id>] [--print]
sniper-data build-bars [--timeframes M1,M5,M15,M30,H1]
sniper-data build-dataset --config <research_cfg.json>
sniper-data test       [--quick]
sniper-data audit-dataset --dataset <id>
sniper-data diff-legacy --legacy-evidence <path> --report <out.json>

14.2 Semantics and safety properties

Command	Meaning	Safety property
certify	compute certificate (read-only on source)	never opens source writable; atomic cert write
init	create a new run (new run_id) from a certified source; refuses if a live run/lock exists; archives prior run on explicit `--new-run`	refuses to touch legacy evidence; validates cert + config
ingest	start processing from INITIALIZED (or resume-equivalent allowed states)	idempotent dispatch; journal-before-checkpoint durability
resume	continue from PAUSED/FAILED-after-resolution; RESUME_BLOCKED otherwise	verifies every guard in §13.5; changes nothing on refusal
pause / stop	graceful stop at next chunk/workload boundary; checkpoint; exit	never kills a worker mid-chunk without completing its rename
status	print checkpoint, progress, lock, journal tail; exit code indicates completion	read-only
verify	run the independent verifier	refuses to run pipeline stages; only reads source/artifacts
manifest	build/print the dataset manifest	refuses to list missing/corrupt files
build-dataset	research dataset builder (§15)	reads canonical layers only; never the raw source
diff-legacy	differential comparison vs legacy evidence	read-only on legacy evidence
test	golden corpus runner	tiny synthetic inputs only

14.3 Exit codes

0  success / COMPLETED
1  operational error (I/O, config invalid)
2  usage error
3  not completed (status: RUNNING/PAUSED/INITIALIZED)
4  BLOCKED (RESUME_BLOCKED / lock held / certification missing)
5  verification or audit failed

The CLI is designed so start / pause / resume / verify / status / completion detection require no human conversation.

15. Research Dataset Builder (Layer 9)

15.1 Separation of concerns

DATA ENGINE → CANONICAL DATA → RESEARCH DATASET → FORECASTING EXPERIMENT → MODEL → TRADING DECISION

The Engine provides canonical ticks+bars and a fixed library of deterministic feature/label primitives. It holds NO forecasting hypothesis. Examples such as "M15 + 15m direction", "M15 + 30m return", "M30 + 60m return" are research configurations, never ingestion rules.

15.2 Research config schema (strict; hash → config_sha256)

{
 "dataset_id": "xau_m15_15m_dir_v1",
 "source_dataset_version": "DS_V1.0.0",
 "timeframe": "M15",
 "prediction_horizon_min": 15,
 "target": {"type": "direction|cls_3|return_reg", "horizon": "15m", "from": "close"},
 "features": [{"name":..., "primitive": "return|log_return|volatility|range|spread_stats|rolling_*",
               "window": k, "args": {...}}],
 "date_range": ["2015-01-01","2020-12-31"],
 "split": {"policy": "time_ordered", "train":0.7,"val":0.15,"test":0.15,
           "oos_barrier": true, "barrier_gap_min": 15},
 "cost_model": {"type":"fixed_spread","spread_units":…,"commission_per_lot":…,"lot_size":…},
 "label_rules": {"min_abs_move_units":…,"neutral_zone":…}}
}

Rules:

Inputs: the canonical bar layer of source_dataset_version ONLY.
Feature/label primitives are deterministic pure functions of bars (integer or pinned-float operations, fixed evaluation order).
Leakage rule: the label for row t uses bars (t, t+h] only; rows lacking h future bars are excluded and counted in the manifest (rows_excluded_no_future).
Split: strictly time-ordered; oos_barrier demands a gap of ≥ horizon between the last training bar and the first test bar (in bars), recorded in the manifest.
All boundaries are (row_id, bar_idx) based, never shuffled at row level.
Float math is allowed HERE (research layer), but the pipeline is deterministic: pinned library versions, fixed operation order, no hash randomization, no set-iteration in serialization paths.

15.3 Outputs

datasets/<dataset_id>/
  config.json, config_sha256,
  features.parquet, labels.parquet, dataset.parquet (row_id, ts, features, label, split),
  split.parquet (row_id → split), manifest.json

16. Dataset Manifest (Layer 10)

manifest.json schema (MS_V1) — applies to the canonical dataset and to every research dataset:

schema_version, dataset_id, dataset_version, engine_version, parser_version,
algorithm_version, source_identity {certificate reference},
timeframe (research only), prediction_horizon_min, target_definition,
features (list), date_range, split_definition, oos_barrier, cost_model,
row_counts {total, per_split, excluded_no_future, malformed…},
files [ {relpath, bytes, sha256_physical, content_id} ],   # sorted by relpath
config_snapshot, config_sha256,
creation_ts_utc,
manifest_hash,   # H(canonical JSON of manifest WITHOUT manifest_hash and dataset_hash)
dataset_hash     # H(canonical concat of (relpath|content_id) list) — excludes manifest_hash

Non-self-referential hashing: manifest_hash explicitly excludes both hash fields; dataset_hash is computed over file content ids only (plus version identity fields), never over the manifest that contains it. No hash cycle.

17. Evidence & Hashing Strategy (Layer 11)

17.1 Evidence hierarchy

SOURCE
  ↓ certification (FULLY_VERIFIED gate)
CANONICAL TICKS          content_id per chunk; dataset hash
CANONICAL BARS           content_id per workload/TF file
DERIVED FEATURES         content_id per dataset file
LABELS                   content_id per dataset file
RESEARCH DATASET         dataset_hash per manifest

17.2 Identity definitions

run_id: RUN-<YYYYMMDD-HHMMSS>-<uuid4 first 8>. Assigned at init.
file hash: SHA-256 of physical bytes (sha256_physical).
content_id: layer-specific canonical logical digest (§8.5, §12.4).
aggregate/layer hash: SHA-256 over the canonical concatenation of the layer's content ids in manifest-sorted order.
manifest hash / dataset hash: see §16 (no self-reference).
checkpoint hash: SHA-256 of the checkpoint payload excluding the hash field itself (§13.2).
evidence manifest (evidence/evidence.json): lists every produced file with both hashes, plus evidence_manifest_hash computed over itself minus the hash field. RUN_COMPLETE.json records the evidence_manifest_hash; no cycle (RUN_COMPLETE.json is not hashed by evidence.json).

17.3 Rules

All hash inputs use fixed canonical serialization (sorted keys at each JSON level, \n line endings, no trailing whitespace).
No hash is ever computed over a blob that contains that same hash.
PYTHONHASHSEED is irrelevant by construction (no set/dict iteration in any serialization path); the Engine additionally pins PYTHONHASHSEED=0 as a belt-and-braces control.

18. Independent Verification (Layer 12)

18.1 Principle

The verifier (engine/verify/) is a SEPARATE implementation written from this specification's tables — its own line splitter, timestamp parser, price parser, and aggregator. It does NOT import or call producer parsing, aggregation, or serialization code. Producer and verifier may share only the version constant tables.

18.2 Verification modes

Mode	What it does
golden	runs the golden corpus through BOTH implementations; byte-equal expected outputs
range	re-parses selected source byte ranges with the verifier parser; compares canonical rows, ts, bid/ask against chunk parquet logical rows
dataset	recomputes every content_id and dataset_hash from artifacts (independent of producer); checks manifest consistency
run-complete	validates RUN_COMPLETE.json plus evidence manifest against artifacts
legacy	differential comparison vs legacy evidence (§24)

18.3 Verification techniques

Golden vectors: hand-computed expected outputs for tiny inputs.
Deterministic invariants (at least): OHLC validity (low ≤ open,close ≤ high; counts ≥ 1), ts_ms non-decreasing across merged stream, Σ per-chunk tick counts == total tick count, bar continuity (close of bar k == first tick of bar k+1 order boundary, carry continuity), content_id stability across producer/verifier, append-only uniqueness of (TF, period_id).
Byte-range verification: independent parse+aggregate over chosen ranges, compared to chunk artifacts.
Hash verification: full recomputation from files.
Differential comparison vs legacy (§24).
Mutation tests: flip a byte / duplicate a row / drop a row / alter a timestamp in a golden corpus COPY (never the source) → engine must change digests or classify per taxonomy.
Boundary tests: chunk boundary exactly at a terminator; boundary inside a line; CRLF/LF/CR/mixed inputs.
Resume tests: kill+resume cycles must reproduce identical content ids.
Reordered-worker tests: same input dispatched in different orders → identical content ids.
Duplicated/missing data tests: injected duplicates/missing rows must be detected by invariants (counters/hash changes).

18.4 Verdicts

verification/report_<run>.json with per-check PASS|FAIL|SKIP(reason)|DIFF(classification) and an overall VERIFIER_ACCEPTED | REJECTED. A FAILED invariant or a failed independent content-id comparison is material (§20).

19. Golden Test Corpus (Layer 13)

Small, permanent, adversarial; each input < 100 KiB; full run < 10 s.

Case	Input characteristics	Expected effect
G01 CRLF	terminator `\r\n` everywhere	parse ok; counts exact
G02 LF	`\n` only	parse ok; identical canonical ids to G01
G03 header present	first row header	skipped; has_header=true
G04 header absent	data only	all rows data
G05 boundary at exact byte	chunk cut lands exactly on terminator	one line per chunk
G06 boundary inside line	chunk cut lands mid-line	chunk map extends boundary to line end; no split record
G07 malformed timestamp	`2023.01.01 12:00:00` variants / garbage	MALFORMED_TIMESTAMP_PARSE count
G08 malformed price	`abc` / signed / extra decimals	MALFORMED_PRICE_* classes
G09 invalid bid/ask	ask < bid	MALFORMED_BID_ASK_RELATION
G10 duplicated row	exact duplicate in chunk	MALFORMED_DUPLICATE + digest sensitivity
G11 dropped row	verifier-independent count mismatch	invariant FAIL (verification-time)
G12 non-monotonic	regressed ts	MALFORMED_NON_MONOTONIC
G13 worker reorder	shuffled dispatch of same input	identical content ids
G14 missing input carry	resume without carry state	RESUME_BLOCKED / continuity FAIL
G15 altered state	corrupt checkpoint hash / journal chain	RESUME_BLOCKED / FAILED
G16 repeated resume	pause+resume+resume	identical content ids (determinism proof)
G17 already-transformed data	canonical-shaped input fed as raw	format mismatch → certification fails / blocked

Source-grammar golden cases SG01–SG15 (P3-DATA-ENGINE-004) extend the corpus with dedicated synthetic fixtures for the authoritative six-column G_TICKSTORY_MT5 grammar: SG01 grammar acceptance, SG02 CRLF handling, SG03 valid six-column row, SG04 wrong field count, SG05 invalid date, SG06 invalid time, SG07 invalid bid, SG08 invalid ask, SG09 invalid last, SG10 invalid volume, SG11 bid>ask, SG12 zero/negative volume, SG13 timestamp conversion (regression anchor 20030505,00:01:03 → 1_052_092_863_000 ms), SG14 decimal precision, SG15 header absence/presence semantics. Like G01–G17, every SG case requires producer == independent verifier == expected values.

Each case has an expected/ JSON (counters, classes, content ids) that both the producer and the verifier must reproduce.

20. Failure Model (Layer 14)

Three classes. The Engine's explicit action set is: continue (with counter), apply ONE predefined recovery, or STOP.

#	Condition	Class	Action	Result state
F01	malformed row (any taxon)	EXPECTED	classify + counter + sidecar	continue
F02	malformed rate > 0.001 or >10,000 in a workload	MATERIAL	STOP; record reason	FAILED
F03	blank lines / BOM / header	EXPECTED	deterministic skip	continue
F04	worker non-zero exit	BOUNDED	ONE recovery: redispatch, ≤2 attempts	continue/FAILED after limit
F05	worker OOM	BOUNDED	ONE recovery: redispatch ≤2	FAILED(`worker_oom`) after limit
F06	chunk file present, no journal	BOUNDED	ONE recovery: verify digest, repair journal OR redispatch	continue
F07	journal says committed, file missing/digest mismatch	MATERIAL	STOP	FAILED
F08	checkpoint hash mismatch	MATERIAL	STOP; no auto-recovery	RESUME_BLOCKED
F09	source size/mtime/cert mismatch	MATERIAL	STOP	RESUME_BLOCKED
F10	version mismatch (engine/parser/algorithm/dataset/schema)	MATERIAL	STOP	RESUME_BLOCKED
F11	carry continuity / bar append mismatch	MATERIAL	STOP	FAILED
F12	disk free below floor at runtime	BOUNDED	ONE recovery: graceful pause	PAUSED(reason=disk_space); resume after human frees space
F13	disk free below floor at init	MATERIAL	STOP	init blocked
F14	lock held by live process	MATERIAL	STOP (no kill)	exit 4 (locked)
F15	unknown exception in canonical writer/aggregator	MATERIAL	STOP	FAILED
F16	verifier verdict REJECTED	MATERIAL	STOP before COMPLETED	FAILED(verification)
F17	malformed threshold inside chunk (pre-workload)	EXPECTED	carry counters	continue (workload check at end)

Rules:

No autonomous recovery-by-reasoning. No silent fallback. No broad self-repair.
Every STOP writes a checkpoint (atomic), a status_reason, and a logs/failure_<run>.jsonl entry with the exact condition code.
FAILED runs resume only after an operator (human) resolves the root cause and re-invokes resume or starts a new run; the Engine never decides to bypass F07–F16.

21. Resource Management (Layer 15)

Resource	Requirement
Max memory, parent	≤ 512 MB RSS (dispatcher/coordinator only)
Max memory, worker	≤ 1 GiB RSS per worker (chunk-bounded; one chunk in memory)
Total memory target	requested workers × per-worker bound; never exceeds measured machine memory / 2
Worker isolation	separate processes (`spawn`); worker crash cannot corrupt parent
Logging	JSONL, one operator log + per-worker logs; rotation at 100 MB, 5 files; logs never influence outputs
Progress	`progress.json`: run_id, status, chunks done/total, bytes, workloads, workers{requested, created, with_data, busy, cpu_util_pct}, started/updated ts, eta_sec
Checkpoint frequency	journal per chunk commit; checkpoint file per workload; always at pause/fail
Graceful shutdown	SIGINT/SIGTERM or `state/control.json` → finish current chunk, join workers (≤300 s), checkpoint PAUSED, exit 0
OS restart recovery	any interruption leaves only orphan `.tmp`; resume (§13.5) restores exactly
Disk-space checks	preflight: free ≥ max(2×estimated_canonical_size, 20 GiB); per-workload: free ≥ 4 GiB else PAUSED(disk_space)
Output-space checks	staging cleaned after successful verification; `--prune-staging` explicit only
Process cleanup	atexit + signal handlers remove lock; children joined; no zombie policy beyond retry
Stale-process detection	lock heartbeat every 60 s; stale if >300 s old; resume with a stale lock → exit 4 and `status` reports `lock_stale`; releasing requires explicit `--force-release-lock` (human gate G-11) — default: refuse

The Engine never relies on AI session lifetime; all of the above executes under the OS scheduler.

22. Run Completion Protocol (Layer 16)

RUN_COMPLETE.json is produced ONLY after: all chunks committed, aggregator flushed, evidence manifest computed, and the independent verifier (verify --mode dataset and --mode run-complete) returned ACCEPTED.

Schema (RC_V1):

status: "COMPLETED",
run_id, source_identity {ref to certificate},
versions {engine, parser, algorithm, dataset, manifest schema},
rows {parsed, canonical, per-chunk-min/max},
malformed {per class, total},
outputs {ticks_files, bar_files_per_TF, dataset_files},
checkpoint {last_completed_chunk, next_chunk(=total), status},
hashes {canonical_dataset_hash, manifest_hash, evidence_manifest_hash,
        per-layer aggregate hashes},
completion_ts_utc, verifier_report (path + verdict)

Machine-verifiability: sniper-data verify --mode run-complete re-checks every hash and file from the disk; exit code 0 ⇔ genuine completion.

Operator flow: run the Engine headlessly; hours later, independently of any AI session, run sniper-data status (→ completion detection), then hand the result package (RUN_COMPLETE.json + evidence + manifests) to ChatGPT for audit.

23. Clean Reprocessing Protocol (Layer 17)

Exact future procedure (NOT executed in this session; each step has an authorization requirement):

#	Step	Entry condition	Authority
1	Freeze legacy pipeline	approved; legacy remains reference-only, evidence preserved	G-12
2	Certify source	step 1; certificate FULLY_VERIFIED preferred	G-3
3	Initialize NEW engine	certificate accepted; new run_id (not checkpoint 759)	G-4
4	Run golden tests	`sniper-data test` green (producer + verifier)	—
5	Run small real-data pilot	e.g., first 2 workloads or a configurable 512 MiB window	G-13 (pilot scope)
6	Independently verify pilot	verifier PASS on pilot ranges	—
7	START full 34 GB processing	step 6 accepted	G-14 (full reprocessing)
8	Allow Python to run independently	no AI required; resume as needed	—
9	Resume if interrupted	§13.5; no AI required	—
10	Complete full dataset	status COMPLETED + RUN_COMPLETE.json	—
11	Independently certify final dataset	verifier ACCEPTED on full dataset	—
12	Freeze canonical research dataset	mark frozen in manifest; publish hashes	G-15 (freeze)

The legacy pipeline is not extended during or after this procedure. Legacy checkpoint 759 and all workload evidence remain untouched.

24. Legacy Comparison Protocol

24.1 Role of the legacy pipeline

The P3-S25 pipeline is a REFERENCE, not an unquestionable oracle. Legacy cursor (chunks 738–759 / workload 45 completed, next chunk 760 at byte 18,442,355,762) is preserved and read-only. The new Engine does not inherit it.

24.2 Differential verification scope

NEW ENGINE ── selected byte ranges ──▶ canonical ticks (ts, bid, ask)
          ── selected ticks ─────────▶ row-level samples
          ── timestamps ─────────────▶ M15/M30 bar boundaries
          ── M15/M30 bars ───────────▶ OHLC/tick counts
          ── cumulative counters ────▶ parsed/canonical/malformed totals
          ── hashes ─────────────────▶ layer digests
                                        ▼
                              LEGACY EVIDENCE

Selection set (deterministic): first 64 KiB; the 24 MiB range preceding legacy chunk 759/760 boundary; and 8 evenly spaced sample ranges across the file (config-driven; each ≤ 24 MiB).

24.3 Discrepancy classification (mandatory taxonomy)

Class	Meaning	Auto-action
EXPECTED_SEMANTIC	documented spec difference (e.g., mid-based OHLC, UTC alignment, carry policy)	record, no alarm
IMPLEMENTATION_DIFFERENCE	same semantics, different internals	record; verify both
LEGACY_DEFECT	legacy evidence contradicts the source within known legacy scope	record; escalate (G-16)
NEW_ENGINE_DEFECT	new Engine deviates from its own spec	FAILED; fix before proceeding
UNRESOLVED	cannot classify from available evidence	record; escalate (G-16); never auto-assume

Output: verification/legacy_diff_<run>.json with per-item range/tick/bar/counter/hash evidence and a classification. The Legacy Comparison stage never modifies legacy files and never chooses an answer on the operator's behalf; UNRESOLVED entries are G-16 material.

25. Security & Integrity Controls

Source is opened rb only; the Engine never holds a writable handle to the raw source and fails if stat changes during certification (F09).
All outputs live under engine_output/; path validation (resolved absolute paths under the configured write root) rejects .., symlink/junction escape, and absolute paths outside the root.
No network access, no credentials, no account data, no secrets, no eval of config (config validated against JSON schema).
Checkpoint and journal are hash-chained (checkpoint_hash, per-line prev); optional HMAC layer (key file outside repo) is a NON-BLOCKING future hardening (G-17).
Every produced artifact carries sha256_physical + content_id (§17); evidence manifest enumerates all files with both.
Legacy evidence directories are excluded from every access path the Engine may write; a runtime guard refuses any write whose target path is not under the Engine's own output root.
Logs are sanitized (no raw bid/ask values, no account identifiers).
Malformed sidecar rows contain hashes (raw_sha256), not raw text.

26. Versioning

Constant	Value (v1)	Bump rule
ENGINE_VERSION	1.1.0	any Engine behavior change (semver)
PARSER_VERSION	PARSER_V1.1	any change to CSV/timestamp/price parsing rules (P3-DE-004: extended with G_TICKSTORY_MT5; unambiguous vs the failed PARSER_V1)
GRAMMAR REGISTRY	G_TICKSTORY_MT5 / G_DOTTED_V1	named presets, certificate-pinned (P3-DE-004; §8.1)
ALGORITHM_VERSION	ALG_V1	any change to canonical serialization, chunking rule, or aggregation semantics (unchanged by P3-DE-004)
SCHEMA_VERSION (ticks)	CTS_V1	schema change
SCHEMA_VERSION (bars)	CBS_V1	bar schema change
DATASET_VERSION	DS_V1.0.0	human decision gate only (G-5)
MANIFEST_SCHEMA_VERSION	MS_V1	manifest schema change
CHECKPOINT_SCHEMA_VERSION	CP_V1	checkpoint schema change
RUN_COMPLETE_SCHEMA_VERSION	RC_V1	completion schema change

Determinism pins (recorded in every manifest): Python ≥ 3.11, pyarrow exact version, pandas (research layer) exact version, PYTHONHASHSEED=0, Parquet writer version string. Any pin change bumps ENGINE_VERSION and requires re-running the golden corpus and determinism tests before further runs.

Version mismatch on resume = RESUME_BLOCKED (F10). No automatic data migration; schema migrations are new dataset versions (G-5).

27. Acceptance Criteria

The Engine is accepted for full reprocessing only when ALL hold:

Golden corpus G01–G17 passes under both producer and verifier. 1a. Source-grammar corpus SG01–SG15 (G_TICKSTORY_MT5 six-column grammar) passes under both producer and verifier with the regression timestamp anchor (20030505,00:01:03 → 1_052_092_863_000 ms at tz offset 0), the compact-grammar end-to-end pipeline run, and the certificate/grammar fail-closed cross-check all green. 1b. CLI headless E2E includes the run-less-directory regression: status and resume on an uninitialized output directory MUST both exit 4 (BLOCKED) without a traceback.
Independent verifier accepts the pilot (step 6).
Resume invariance: content ids of a run interrupted 3× and resumed are identical to an uninterrupted run of the same range.
verify --mode dataset recomputes all content ids and the dataset hash from disk independently and matches.
verify --mode run-complete returns exit 0 only on a genuine COMPLETED.
Differential report vs legacy is produced with every item classified.
Source certificate: FULL_SOURCE_SHA256_VERIFIED achieved or the run did not start without G-3 authorization.
Malformed rates within limits; malformed sidecar complete.
Research dataset regeneration: two runs of build-dataset with the same config produce identical dataset hashes without touching the raw source.
Engine completes a full 34.5 GB pass headlessly (AI closed), including at least one forced resume.
Legacy evidence directories byte-identical before/after full run.
Source file size and mtime unchanged by the full run.

28. Risks & Mitigations

#	Risk	Mitigation
R-TZ	Server-time/timezone ambiguity (DST) shifts bar boundaries	canonical ts is a pure function of text; re-derivation with new offset = new dataset version; manifest records offset; G-6
R-DISK	34.5 GB source + canonical outputs exceed free space	preflight floor; per-workload floor; PAUSED(disk_space)
R-HW	Disk/CPU failure mid-run	atomic commits; journal truth; resume
R-AV	Antivirus/OneDrive locking files	rename+fsync pattern; documented deployment requirement; retry on transient EACCES (bounded)
R-DETS	Library/dependency drift breaks determinism	pinned versions recorded; golden corpus gate; upgrade bumps ENGINE_VERSION
R-MEM	Parent or worker memory blowup	chunk-bounded workers; memory telemetry; retry/FAILED policy
R-DUP	Duplicate/overlapping data in source (Tickstory known cases)	taxonomy counters; verifier invariants; research layer documents dedup policy per dataset config
R-LEG	Legacy evidence drift/incomplete	legacy dirs read-only; differential classification taxonomy
R-GAP	Weekend/illiquid gaps produce zero-tick bars	bars exist only when ticks exist (no empty-bar inflation); gap analysis is research-layer
R-CERT	Full SHA-256 too slow on target hardware	certification budget; VERIFIED_WITH_LIMITATION is explicit and gated (G-3)
R-SRC	Source replaced/moved by another process	size/mtime re-checks; cert binding; RESUME_BLOCKED
R-SPEC	Spec ambiguity discovered at implementation	every ambiguity must be filed as an Open Decision, not silently assumed

29. Open Decisions Requiring Human Authorization

Every item below is a HUMAN DECISION GATE. The Engine must NOT proceed past the gate without an explicit owner decision recorded in docs/CURRENT_DECISION_GATE.md (or successor register).

#	Decision	Recommended default	Consequence if not authorized
G-1	Final storage format (Parquet + zstd-3 + pinned pyarrow)	accept default	init blocked
G-2	Default worker count (24; created = min(24, cpu_count, mem-constrained))	accept default	ingest uses 1 worker until decided
G-3	Source certification acceptance: require FULLY_VERIFIED; allow VERIFIED_WITH_LIMITATION only with explicit authorization	FULLY_VERIFIED only	no run under limitation
G-4	Canonical schema CTS_V1/CBS_V1 (columns, PRICE_SCALE=1e6, mid-based OHLC with PRICE_SCALE_BAR=2e6, vol semantics)	accept default	canonicalization blocked
G-5	Dataset version identity & bump policy (DS_X.Y.Z, human-only bumps)	accept default	freeze/dataset publish blocked
G-6	Source timezone offset (default config `source_tz_offset_minutes`; DST not modeled in v1)	record + document	canonicalization blocked
G-7	Atomic chunk size (24 MiB nominal, line-safe)	accept default	chunkmap frozen with legacy grid
G-8	Workload size (512 MiB nominal)	accept default	workload grouping frozen
G-9	Aggregation semantics (UTC clock boundaries, mid OHLC, spread min/max/avg floor, carry policy)	accept default	bar layer blocked
G-10	Bar partial policy at EOF (flush as is_final=false) & no empty-bar inflation	accept default	bar layer publish blocked
G-11	Stale-lock release policy (default: refuse; `--force-release-lock` explicit)	refuse by default	blocked while stale lock present
G-12	Freeze legacy pipeline as reference-only	after G-1..G-10	reprocessing not authorized
G-13	Pilot scope (default first 512 MiB window)	accept default	pilot not run
G-14	FULL 34 GB REPROCESSING authorization	after pilot verification	step 7 never starts
G-15	Freeze of canonical research dataset + publish hashes	after independent certification	step 12 not executed
G-16	Legacy-vs-new discrepancy resolution (per UNRESOLVED item)	adjudicate per item	run stays paused
G-17	Optional checkpoint HMAC hardening	non-blocking future	no impact on v1

Additionally: any research target definition (Layer 9 configs such as M15+15m direction) is owner-authorized research configuration and is out of engine scope until G-14 completes. No gate is decided by the Engine.

Approved governance clarifications (P3-DATA-ENGINE-001 GOV-CLOSE, human decision baseline 2026-09-07):

G-14 remains NOT AUTHORIZED even after design freeze.
G-6 approval does not define the actual UTC offset value; source_tz_offset_minutes must be established explicitly at source initialization.
G-12 is a legacy freeze/reference decision and does not mean the legacy checkpoint becomes a new-engine checkpoint.

Appendix A — Final Review Gate (design self-check)

#	Question	Answer
1	Works with ChatGPT completely closed?	YES — headless CLI, no AI dependency anywhere in the pipeline (§4.3, §14)
2	Windows restart interrupts without corrupting state?	YES — atomic tmp+rename, journal truth, orphan-tmp cleanup (§13)
3	Resume without AI intervention?	YES — §13.5 procedure is deterministic `sniper-data resume`
4	Source identity immutable?	YES — read-only handles, cert binding, F09 (§7, §25)
5	Full source SHA requirement explicit?	YES — required; limitation is explicit + gated (§7.4–7.5, G-3)
6	Canonical dataset reproducible?	YES — content-id contract independent of physical bytes (§12.4)
7	Research datasets regenerable without re-reading raw ticks?	YES — canonical bar layer only (§15)
8	Verifier independent of producer validation?	YES — separate implementation, no shared parse/aggregate code (§18)
9	Discrepancy old vs new diagnosable?	YES — legacy comparison taxonomy (§24)
10	Human decisions clearly identified?	YES — §29 gates G-1..G-17
11	Hidden dependency on future AI interaction?	NO — none
12	Any path silently changing semantics?	NO — version gates, RESUME_BLOCKED, fail-closed states (§13, §20, §26)

Non-conformance items, if any, are filed in §29 rather than silently assumed. This table records DESIGN COMPLETE.

Appendix B — Glossary

Term	Definition
chunk	line-safe byte range of the raw source (§9)
workload	deterministic group of consecutive chunks (§9.2)
content_id	canonical logical digest of a layer file (§12.4)
journal	append-only hash-chained commit record (Layer 4/7)
carry	in-progress partial bar per timeframe (§11.4)
material condition	a failure that forces STOP and operator resolution (§20)
expected class	a deterministic, counted, non-fatal data anomaly (§20)

End of specification. This document is the only artifact produced by the design session. Nothing in it authorizes execution.

Appendix C — P3-DATA-ENGINE-004 CONTROLLED AMENDMENT RECORD

Session type: CONTROLLED SPECIFICATION AMENDMENT + IMPLEMENTATION RE-QUALIFICATION ONLY (authorized by the owner after the P3-DATA-ENGINE-003 pilot finding; DESIGN remains FROZEN; no unrelated governance decision is re-opened). All amendments below are traceable to the P3-DATA-ENGINE-003 evidence: source XAUUSD_mt5_ticks.csv (size 34,473,661,010 bytes; FULL SHA-256 5252ce8f5b0e7d286b71a3172c7ef9c1009c8e4b1254ac67400a803fcd85cba1; CRLF; ASCII; no header; first raw row 20030505,00:01:03,…).

C.1 Source grammar (authoritative; supersedes the 8.1 dotted primary)

G_TICKSTORY_MT5  ::  YYYYMMDD , HH:MM:SS , bid , ask , last , volume
field count      ::  exactly 6 (else MALFORMED_FIELD_COUNT)
date             ::  8 digits, month 01..12, day 01..31
time             ::  HH:MM:SS, hh 00..23, mi 00..59, ss 00..59 (no ms)
bid/ask/last     ::  decimal micro-price (PRICE_SCALE = 1e6; 0..6 fraction
                     digits); non-positive -> MALFORMED_PRICE_NONPOSITIVE;
                     ask < bid -> MALFORMED_BID_ASK_RELATION (mandatory)
volume           ::  non-negative integer; 0 VALID; negative/non-integer
                     -> MALFORMED_VOLUME (has_volume=true enforced)
delimiter        ::  comma only
line ending      ::  CRLF/LF/CR accepted deterministically; recorded
header           ::  none in the real source; keyword-detected header
                     (date|time|datetime prefix) skipped + has_header=true;
                     absence verified by evidence

Grammar selection is certificate-bound: grammar_id in source_certificate.json must equal the configured source_grammar, else init is BLOCKED (spec 8.1 certificate/config mismatch). No silent generic format acceptance.

C.2 Timestamp

ts_ms = wallclock_epoch_ms(YYYYMMDD, HH:MM:SS, ms=0) - source_tz_offset_minutes * 60000, integer civil-day math, deterministic. Production source_tz_offset_minutes = 0 preserved (P3-DE-003 evidence chain: first row 20030505,00:01:03 -> epoch 1,052,092,863 s equivalent 1,052,092,863,000 ms == legacy chunk0 first_timestamp; G-6 policy; DST not modeled in v1). Regression-anchored by SG13.

C.3 Volume / last-field / versions

Volume: see 8.2 amendment (mandatory source volume for G_TICKSTORY_MT5; vol = 1 substitution FORBIDDEN for this source; zero valid).
last: validated; EXCLUDED from CTS_V1; preserved in source_preservation/chunk_%06d.jsonl (spec 8.6); promotion to the canonical schema = new CTS version + G-5 gate.
Versioning: ENGINE_VERSION 1.1.0; PARSER_VERSION PARSER_V1.1; ALGORITHM_VERSION ALG_V1 (unchanged); SCHEMA_VERSION CTS_V1/CBS_V1 (unchanged); DATASET_VERSION DS_V1.0.0 (unchanged; G-5).
G-9 (aggregation semantics) and G-10 (partial-bar policy): UNCHANGED.
G-14: UNCHANGED — BLOCKED / NOT AUTHORIZED.
Legacy state: checkpoint 759 -> 760 @ byte 18,442,355,762 UNCHANGED; workload 46 NOT prepared; legacy pipeline byte-identical and read-only.

C.4 Implementation-visible changes

Parser/source-adapter: two named grammar presets (8.1 registry); compact timestamp parse; last validation + preservation sidecar; volume semantics; certificate grammar_id + compact_date_time sniffing.
CLI defect repair: status/resume on a run-less directory now exit 4 (BLOCKED) instead of exit 1 + traceback (P3-DE-003 finding; NEW_ENGINE_DEFECT, non-data-affecting; regression in CLI E2E).
Qualification suite extended: SG01–SG15 source-grammar golden cases + compact-grammar end-to-end + dependent acceptance criteria (27).

C.5 Scope exclusions (unchanged)

No pilot execution, no 34 GB source processing, no full-source hashing for ingestion purposes, no workload 46, no chunk 760+ preparation, no legacy checkpoint/pipeline modification, no research dataset generation, no forecasting, no G-14 authorization.

73 KiB Raw Permalink Blame History

P3_DATA_ENGINE_V1_SPEC — SNIPERGOLD DATA ENGINE v1

Data Engine Architecture & Clean Reprocessing Specification

0. Executive Summary

1. Purpose

2. Scope

3. Non-Goals

4. Architecture

4.1 Logical layers

4.2 Design invariants (binding)

4.3 Execution topology

5. Data Flow

6. Component Design

6.1 Module map (implementation target)

6.2 Responsibilities matrix

6.3 Configuration

7. Source Certification (Layer 1)

7.1 Purpose

7.2 Certificate schema

7.3 Certification procedure

7.4 Certification statuses

7.5 Operational requirement

8. Canonical Tick Representation (Layer 2)

8.1 CSV interpretation (deterministic rules)

8.2 Price, bid/ask, volume semantics

8.3 Malformed-record taxonomy (complete)

8.4 Canonical record (CTS_V1)

8.5 Canonical serialization & hashing

8.6 LAST-field treatment and source-preservation layer (P3-DE-004)

9. Chunking (Layer 3)

9.1 Evaluation of legacy values

9.2 Chunk map (deterministic)

9.3 Byte cursor semantics

10. Processing Model (Layer 4)

10.1 Model selection

10.2 Worker & dispatch model

10.3 Deterministic merge

10.4 Worker failure handling (bounded)

11. Aggregation (Layer 5)

11.1 Timeframes and justification

11.2 Boundary definition

11.3 OHLC / volume / spread (integer-exact)

11.4 Carry state

11.5 Finalization rules

11.6 Bar schema (CBS_V1)

12. Storage Format & Persistent Canonical Dataset (Layer 6)

12.1 Format decision

12.2 Layout

12.3 Naming rules

12.4 Dataset identity (content ids, not physical bytes)

12.5 Schema metadata & versioning

13. Checkpoint / Resume (Layer 7)

13.1 Files

13.2 Checkpoint schema (CP_V1)

13.3 Status machine

13.4 Checkpoint frequency

13.5 Resume procedure (deterministic)

13.6 Source identity mismatch

14. Headless CLI (Layer 8)

14.1 Commands (semantics, not implemented this session)

14.2 Semantics and safety properties

14.3 Exit codes

15. Research Dataset Builder (Layer 9)

15.1 Separation of concerns

15.2 Research config schema (strict; hash → config_sha256)

15.3 Outputs

16. Dataset Manifest (Layer 10)

17. Evidence & Hashing Strategy (Layer 11)

17.1 Evidence hierarchy

17.2 Identity definitions

17.3 Rules

18. Independent Verification (Layer 12)

18.1 Principle

18.2 Verification modes

18.3 Verification techniques

18.4 Verdicts

19. Golden Test Corpus (Layer 13)

20. Failure Model (Layer 14)

21. Resource Management (Layer 15)

22. Run Completion Protocol (Layer 16)

73 KiB

Raw Permalink Blame History