Aliwkarimi98/SniperGold_ML
Watch
1
0
Fork
You've already forked SniperGold_ML
 1
forked from chiki2bum2/SniperGold_ML
Code Pull requests Activity
SniperGold_ML/docs/P3_DATA_ENGINE_V1_SPEC.md

1429 lines
73 KiB
Markdown
Raw Permalink Blame History

# P3_DATA_ENGINE_V1_SPEC — SNIPERGOLD DATA ENGINE v1
## Data Engine Architecture & Clean Reprocessing Specification
| Field | Value |
|---|---|
| Document ID | P3_DATA_ENGINE_V1_SPEC |
| Version | 1.1.0 (amended by P3-DATA-ENGINE-004; Appendix C authoritative for the amended clauses) |
| Status | DESIGN FROZEN + CONTROLLED AMENDMENT APPLIED (P3-DE-004); implemented & qualified |
| Session type | Amendment: source-grammar correction + implementation re-qualification |
| Authorized execution | Qualification only (NO real-data ingestion; NO G-14) |
| Classification | Internal research infrastructure |
---
## 0. Executive Summary
This document defines SNIPERGOLD DATA ENGINE v1 ("the Engine"): a standalone,
headless Python system that performs a clean, deterministic, resumable full
reprocessing of the authoritative XAUUSD Tickstory tick source
(`XAUUSD_mt5_ticks.csv`, current known size 34,473,661,010 bytes).
The Engine is designed so that the entire source processing lifecycle can
complete with ChatGPT closed. The AI Assistant role is limited to research
design, governance, audit, and interpretation. Python is the deterministic
data processor.
This session produced this specification only. No real-data ingestion was
performed, no checkpoint was modified, and no pipeline code was changed.
---
## 1. Purpose
Produce an implementation-ready architecture for a data engine that:
1. certifies the identity of the 34.5 GB raw tick source before any processing;
2. performs a clean full reprocessing of that source into a canonical,
columnar, versioned, resumable dataset;
3. builds deterministic canonical time bars (M1, M5, M15, M30, H1);
4. builds independently reproducible research datasets from canonical data
without ever re-reading the raw source;
5. runs headlessly, survives interruption (including Windows restarts),
and resumes without AI assistance;
6. exposes full auditability: every layer has a deterministic identity and
every artifact is independently verifiable.
## 2. Scope
The Engine covers:
- source certification (size, mtime, full SHA-256, format facts);
- deterministic line-safe chunking of the raw source;
- canonical tick parsing, classification, serialization, hashing;
- parallel canonical tick conversion with deterministic merge semantics;
- canonical bar aggregation (M1, M5, M15, M30, H1);
- persistent canonical storage (Parquet by default; see Human Decision Gate G-1);
- standalone checkpoint / resume with atomic fail-closed transitions;
- headless CLI (`sniper-data`), config, manifests, evidence and hashing;
- independent verification tooling, golden test corpus, failure model,
resource management, completion protocol, clean reprocessing protocol,
legacy comparison protocol;
- research dataset builder driven strictly by research configuration.
## 3. Non-Goals
The Engine MUST NOT:
- require an AI Assistant session (or any human) to remain active;
- make trading decisions, optimize a trading strategy, or select models;
- chase a "successful" timeframe or perform forecasting;
- read the legacy checkpoint as its own execution state (legacy checkpoint 759
is preserved and untouched; the new Engine starts its own lifecycle);
- reuse the legacy parser/aggregator code as its implementation;
- silently downgrade certification requirements;
- auto-recover from material failures by reasoning;
- use CSV as a default intermediate storage format;
- write to, delete, or modify any legacy pipeline directories or evidence.
## 4. Architecture
### 4.1 Logical layers
| # | Layer | Responsibility | Immutable contract |
|---|---|---|---|
| 0 | RAW SOURCE | immutable Tickstory CSV | never modified by the Engine |
| 1 | SOURCE CERTIFICATION | identity + format facts | certificate JSON |
| 2 | CANONICAL TICK REPRESENTATION | raw row → canonical record | canonical schema + hash |
| 3 | CHUNKING | line-safe byte ranges | chunk map |
| 4 | PROCESSING MODEL | parallel deterministic conversion | chunk output files |
| 5 | AGGREGATION | tick → M1/M5/M15/M30/H1 bars | bar files + carries |
| 6 | PERSISTENT CANONICAL DATASET | Parquet layer | dataset manifest |
| 7 | CHECKPOINT / RESUME | atomic state | checkpoint + commit journal |
| 8 | HEADLESS CLI | operator interface | exit codes |
| 9 | RESEARCH DATASET BUILDER | config-driven derived data | research dataset + manifest |
| 10 | DATASET MANIFEST | reproducibility record | manifest JSON |
| 11 | EVIDENCE & HASHING | identity at every layer | evidence manifest |
| 12 | INDEPENDENT VERIFICATION | non-producer verification | verification report |
| 13 | GOLDEN TEST CORPUS | adversarial micro-inputs | expected outputs |
| 14 | FAILURE MODEL | three-class fail-closed behavior | state transitions |
| 15 | RESOURCE MANAGEMENT | bounded local execution | telemetry |
| 16 | RUN COMPLETION PROTOCOL | machine-verifiable completion | RUN_COMPLETE.json |
| 17 | CLEAN REPROCESSING PROTOCOL | the authorized future procedure | step gates |
### 4.2 Design invariants (binding)
1. **Determinism**: canonical output is a pure function of
(source bytes, engine version, parser version, algorithm version, config
snapshot). Worker count, scheduling, result arrival order, disk layout and
wall-clock time NEVER affect output.
2. **Fail-closed**: every transition is atomic; on doubt the system stops in a
documented state. There is no silent fallback and no autonomous
recovery-by-reasoning.
3. **Resumable**: any chunk is either fully committed (file + journal
record + digest) or not committed at all. A hard power-loss leaves at most
an orphan `.tmp` file that resume deletes and re-dispatches.
4. **Source immutability**: the Engine opens the source read-only (`rb`),
never writes it, and re-checks size+mtime at every resume.
5. **Integer-only canonical math**: the canonical path (parse → serialization →
aggregation) uses integer arithmetic only. No floating point anywhere in
canonical ingestion or bar aggregation.
6. **Append-only outputs**: committed chunk/bar files are never rewritten in
place. Resume appends or skips; it never mutates committed data.
7. **Independent verification**: the verifier is a separate implementation
(different parsing and aggregation code), not a re-run of producer code.
### 4.3 Execution topology
```
sniper-data CLI (parent/operator process)
├── chunkmap scanner (reads source bytes for newline boundaries)
├── certification (size/mtime/SHA-256 + format sniff)
├── dispatcher (assigns chunk indices to workers)
│ ├── worker process 1..N (parse chunk -> chunk parquet + digests)
│ └── ... (spawn context; no shared state)
├── commit journal (append-only hash-chained commit records)
├── checkpoint writer (atomic tmp+rename)
├── sequential aggregator (canonical ticks -> bars, carry-aware)
├── research builder (canonical bars -> research datasets)
├── verifier (independent implementation)
└── completion/finalization (evidence + RUN_COMPLETE.json)
```
No component talks to a network. No component requires an interactive session.
## 5. Data Flow
```
RAW SOURCE (34.5 GB CSV, read-only)
│ [certify] size, mtime, FULL SHA-256, format facts
▼
source_certificate.json (Layer 1)
│ [chunkmap] newline-scan only
▼
chunk_map.json (Layer 3)
│ [parse+canonicalize, parallel]
▼
ticks/chunk_%06d.parquet + malformed/malformed_%06d.jsonl (Layer 2/4/6)
│ [ordered merge; sequential aggregator, carry-aware]
▼
bars/<TF>/wl_%03d.parquet (Layer 5)
│ [dataset manifest + content digests]
▼
dataset manifest (canonical) (Layer 6/10)
│ [research config]
▼
datasets/<dataset_id>/… (Layer 9)
│ [evidence + hash]
▼
evidence.json + RUN_COMPLETE.json (Layer 11/16)
```
Every layer records its inputs (`source_id`, versions) so each stage can be
recomputed and compared. Research datasets are built from the canonical bar
layer ONLY — never re-reading the 34 GB raw source.
## 6. Component Design
### 6.1 Module map (implementation target)
```
engine/
cli.py # argument parsing, command dispatch, exit codes
config.py # config load + strict schema validation
versions.py # canonical version constants (see §26)
certify.py # Layer 1
chunkmap.py # Layer 3 (newline scanning; pure function of bytes)
parse.py # Layer 2 producer parser (optimized, integer-only)
canonical.py # canonical record serialize/hash (Layer 2)
worker.py # Layer 4 worker entry point (spawn-safe)
dispatcher.py # Layer 4 dispatch + retry + telemetry
journal.py # commit journal (append-only, hash-chained)
checkpoint.py # Layer 7 atomic checkpoint read/write
aggregate.py # Layer 5 sequential deterministic aggregator
storage.py # Layer 6 parquet writers, naming, schema metadata
dataset_builder.py # Layer 9 research dataset builder
manifest.py # Layer 10 manifest build/verify
evidence.py # Layer 11 hashing + evidence manifest
run_complete.py # Layer 16 completion protocol
verify/ # Layer 12 INDEPENDENT verifier (separate impl)
__init__.py
vparse.py # separate CSV/line parser written from spec
vaggregate.py # separate aggregator written from spec
vinvariants.py # invariant checks
vcompare.py # differential comparison
tests/golden/ # Layer 13 corpus + expected outputs
cases/ expected/
tests/mutations/ # mutation test driver + cases
```
### 6.2 Responsibilities matrix
| Component | Reads | Writes | Fails closed by |
|---|---|---|---|
| certifier | source (rb) | `certification/source_certificate.json` | never writing source; recording status |
| chunkmapper | source (rb) | `chunkmap/chunkmap-<id>.json` | deterministic scan; no parsing |
| worker | source range, chunk map | `staging/chunk_%06d.parquet.tmp` → `ticks/chunk_%06d.parquet`; `malformed/...` | one chunk per process; tmp+rename; digest computed before rename |
| dispatcher | chunk map, journal | journal entries, `progress.json` | bounded retries then FAILED |
| aggregator | ticks parquet (ordered) | `bars/<TF>/wl_%03d.parquet`; carries in checkpoint | append-only; carry continuity check on resume |
| verifier | source ranges, artifacts | `verification/...` | never reusing producer parse/aggregate code |
| finalizer | everything | `evidence.json`, `RUN_COMPLETE.json` | only from COMPLETED + verified state |
### 6.3 Configuration
Config is a strict-schema JSON file passed at `init`. Fields relevant to data
identity: `source_path`, `source_tz_offset_minutes`, `has_volume`,
`timeframes`. Fields relevant to operation: `workers_requested`,
`chunk_bytes_nominal`, `workload_bytes_nominal`, `checkpoint_every_workload`,
`retry_limit`, `memory_limits`, `disk_floor_bytes`, `timeouts`. Every config
field that affects output is captured in the dataset manifest (`config_snapshot`
+ `config_sha256`). Operational-only fields are excluded from `config_sha256`.
## 7. Source Certification (Layer 1)
### 7.1 Purpose
Before any processing, the Engine certifies that the configured source file is
the authorized, unmodified Tickstory source.
### 7.2 Certificate schema
`certification/source_certificate.json`:
```
schema_version "CERT_V1"
certification_timestamp ISO-8601 UTC
certification_status FULLY_VERIFIED | VERIFIED_WITH_LIMITATION | FAILED
source_path absolute path as configured
source_size_bytes integer, stat size
source_mtime_ns integer, stat mtime nanoseconds (Windows st_mtime_ns)
sha256_full hex (only present when the full file was hashed)
sha256_mode "full" | "not_computed"
limitation_reason null | enum(see 7.4)
source_format "csv_tickstory_mt5" | "unrecognized" | "mismatch"
grammar_id "tickstory_mt5_v1" | "dotted_v1" | null (P3-DE-004: pins the source-adapter preset; init cross-check §8.1)
line_ending "CRLF" | "LF" | "CR" | "MIXED" (with ratio record)
encoding detected (BOM-aware) ; "utf8" | "ascii" | other
timestamp_format_detected "compact_date_time" | "dotted_ms" | "dotted_s" | "unrecognized"
delimiter "," (only comma supported in v1)
has_header bool (detection rule §8.3)
column_layout {"fields":N,"names":[...]} observed; 3/4 => [datetime,bid,ask(,vol)]; 6 => [date,time,bid,ask,last,volume]
parser_version (constants §26)
engine_version (constants §26)
```
### 7.3 Certification procedure
1. Verify path exists and is a regular file.
2. Record `size` and `mtime_ns`.
3. Stream the file once, `rb`, in fixed 8 MiB buffers, computing SHA-256 over
ALL bytes, and simultaneously sniffing: first 1 MiB (line terminator
classification), first 1000 parseable line-shaped rows (timestamp/delimiter/
header detection), BOM presence.
4. On success: write certificate with status `FULLY_VERIFIED` and the full hash.
5. If the configurable time budget (`certify_timeout_sec`, default 3600; config
option) elapses before the hash completes, the process stops cleanly and
records `VERIFIED_WITH_LIMITATION` with `sha256_mode="not_computed"` and
`limitation_reason="timeout"`. It MUST NOT claim a full hash.
6. Write is atomic (`*.tmp` + fsync + rename). Certificate path is derived
from source identity fields; re-certification is an allowed overwrite,
always recorded with a new timestamp.
### 7.4 Certification statuses
| Status | Meaning | Consequences |
|---|---|---|
| FULLY_VERIFIED | full file hashed; format facts consistent | canonical pipeline eligible (subject to Gate G-3) |
| VERIFIED_WITH_LIMITATION | full SHA-256 not completed within budget; only size/mtime/format facts guaranteed | pipeline MUST NOT claim full certification; starting a run requires explicit human authorization (Gate G-3) |
| FAILED | size/mtime changed mid-read, unreadable, or format facts self-contradictory | RESUME_BLOCKED; no processing |
A size or mtime change detected between the pre-stat and post-stat is an
automatic FAILED with `limitation_reason="source_changed_during_certify"`.
### 7.5 Operational requirement
FULL SOURCE SHA-256 VERIFIED is the required precondition of the canonical
pipeline. Any decision to proceed with `VERIFIED_WITH_LIMITATION` is a HUMAN
DECISION GATE (G-3). The Engine never silently downgrades this requirement.
## 8. Canonical Tick Representation (Layer 2)
### 8.1 CSV interpretation (deterministic rules)
- **Encoding**: UTF-8 (a leading UTF-8 BOM is stripped). Any undecodable byte
sequence marks the row `MALFORMED_ENCODING`. ASCII is a strict subset and is
processed identically.
- **Delimiter**: comma only in v1. Detection result is recorded in the
certificate; a mismatch between certificate and config blocks init.
- **Line endings**: the row scanner splits on any of `\r\n`, `\n`, `\r`
(deterministic; independent of the recorded classification). The recorded
`line_ending` is diagnostic.
- **Blank lines**: skipped, counted in `malformed.blank_line` (expected class).
- **Header**: after BOM strip, if the first non-blank row has a first field
matching `^(datetime|date|time)([^0-9]|$)` (case-insensitive), that row is
the header and is skipped with `has_header=true` recorded in the certificate.
The header row boundary is deterministic.
- **Grammar presets (P3-DATA-ENGINE-004 source adapter)**: exactly two named,
fully-specified, versioned grammar presets exist: `G_TICKSTORY_MT5`
(six-column `YYYYMMDD,HH:MM:SS,bid,ask,last,volume`; the authoritative
production grammar, derived from the P3-DATA-ENGINE-003 pilot evidence) and
`G_DOTTED_V1` (dotted 3/4-column corpus grammar, retained for the synthetic
G01–G17 fixtures). The certificate pins `grammar_id` from the observed
facts; the operator config `source_grammar` MUST equal the certified
`grammar_id` or `init` is BLOCKED (certificate/config mismatch, spec 8.1).
There is no silent per-row format guessing.
- **Timestamp**: accepted forms (recorded at certification):
- `YYYYMMDD` + `HH:MM:SS` (compact date + time fields; G_TICKSTORY_MT5) — PRIMARY;
- `YYYY.MM.DD HH:MM:SS.mmm` (dotted, 3-digit milliseconds; G_DOTTED_V1) — corpus preset;
- `YYYY.MM.DD HH:MM:SS` (no ms) — corpus preset, ms = 0.
Any other shape → `MALFORMED_TIMESTAMP_PARSE`.
- **Timezone**: source text is a wall-clock in the trader's server time.
Canonical `ts_ms` = epoch milliseconds UTC computed as:
`ts_ms = julian_day_ms(wall_clock) - source_tz_offset_minutes * 60000`.
`source_tz_offset_minutes` is an explicit config value recorded in the
manifest; DST is NOT modeled in v1 (see Open Decisions, risk R-TZ). The
parse→canonical mapping is therefore a pure, documented function of the
source text, so any future re-interpretation is a re-derivation with a new
dataset version, never an in-place mutation.
- **Timestamp range**: valid only if
`2000-01-01T00:00:00Z <= ts_ms <= certify_time+7d`; outside → `MALFORMED_TIMESTAMP_RANGE`.
### 8.2 Price, bid/ask, volume semantics
- Canonical prices are **scaled integers**: `PRICE_SCALE = 1_000_000`
(micro-USD units). `bid_u = int(decimal_text * PRICE_SCALE)` computed by
exact decimal-string parsing (no floats).
- `ask_u` likewise. `spread_u = ask_u - bid_u`.
- Invalid relationships:
- `bid_u <= 0` or `ask_u <= 0` → `MALFORMED_PRICE_NONPOSITIVE`;
- `spread_u < 0` → `MALFORMED_BID_ASK_RELATION`;
- fractional digits beyond the scale → `MALFORMED_PRICE_PRECISION`.
- **Volume (amended P3-DATA-ENGINE-004)**: the grammar presets fix the volume
semantics — for `G_TICKSTORY_MT5` the source HAS an explicit mandatory
volume column so `has_volume=true` is enforced by config validation;
`vol = 1` substitution (`has_volume=false`) is FORBIDDEN for this source.
Volume is a non-negative integer token; `0` is VALID (observed in the real
source: many canonical rows carry volume 0); negative or non-integer
values are `MALFORMED_VOLUME`. The dotted preset retains `has_volume=false`
(default) → `vol = 1` tick-volume semantics. Any violation of the
certificate's `column_layout` vs config → init block.
- Duplicate detection: identical (`ts_ms,bid_u,ask_u,vol`) records within one
chunk are flagged `MALFORMED_DUPLICATE` (bounded in-memory set per chunk).
- Monotonicity: within a chunk, `ts_ms` must be non-decreasing; otherwise the
offending row is `MALFORMED_NON_MONOTONIC`. Across a chunk boundary the
first `ts_ms` of chunk N must be >= the last `ts_ms` of chunk N-1 (carried
in checkpoint); a violation is counted as
`MALFORMED_NON_MONOTONIC_BOUNDARY` and, if > threshold, becomes material (§20).
### 8.3 Malformed-record taxonomy (complete)
| Class | Triggers |
|---|---|
| MALFORMED_EMPTY_LINE | blank row |
| MALFORMED_ENCODING | undecodable bytes |
| MALFORMED_FIELD_COUNT | field count ≠ grammar-expected (G_DOTTED_V1: 3/4; G_TICKSTORY_MT5: exactly 6) |
| MALFORMED_TIMESTAMP_PARSE | timestamp text unrecognized |
| MALFORMED_TIMESTAMP_RANGE | ts outside validity window |
| MALFORMED_PRICE_PARSE | bid/ask non-numeric |
| MALFORMED_PRICE_PRECISION | decimals exceed PRICE_SCALE precision |
| MALFORMED_PRICE_NONPOSITIVE | bid<=0 or ask<=0 |
| MALFORMED_BID_ASK_RELATION | ask < bid |
| MALFORMED_VOLUME | volume invalid (when has_volume) |
| MALFORMED_DUPLICATE | exact duplicate within chunk |
| MALFORMED_NON_MONOTONIC | ts regress within chunk |
| MALFORMED_NON_MONOTONIC_BOUNDARY | ts regression across chunk boundary |
Malformed rows are NEVER fabricated, skipped silently, or injected into the
canonical stream. Each is appended to `malformed/malformed_%06d.jsonl`
(`chunk_index`, `byte_start`, `global_line_no`, `class`, `raw_sha256`,
`reason`). Counters per class accumulate in the checkpoint and in
`malformed_summary.json` at completion. If malformed rows exceed
`malformed_rate_limit` (default 0.001 of parsed rows in a workload) OR the
absolute cap (default 10,000 in a workload), the run enters FAILED (§20).
### 8.4 Canonical record (CTS_V1)
| Column | Type | Semantics |
|---|---|---|
| ts_ms | int64 | epoch ms UTC |
| bid_u | int64 | micro-USD × 1e6 |
| ask_u | int64 | micro-USD × 1e6 |
| spread_u | int64 | ask_u − bid_u (≥ 0) |
| vol | int64 | source volume or 1 |
| src_line | int64 | global 1-based raw line number (audit anchor) |
| rec_ord | int64 | 0-based record ordinal within the chunk |
All values are integers. No floats, no nulls, no strings in the canonical tick
table.
### 8.5 Canonical serialization & hashing
The canonical serialization of a record r is the single line:
```
ts_ms|bid_u|ask_u|vol
```
(no spaces, `\n` terminated). The **chunk content digest** is:
```
chunk_content_id = SHA-256( concat(serialize(r_0), serialize(r_1), …, serialize(r_{n-1})) )
```
in `rec_ord` order. The chunk content id is the reproducibility identity of the
chunk (Parquet physical bytes are not part of the reproducibility contract,
because physical writer metadata may differ run-to-run; the logical
serialization is fixed by this spec). The chunk physical SHA-256 is also
recorded for integrity evidence.
### 8.6 LAST-field treatment and source-preservation layer (P3-DE-004)
The G_TICKSTORY_MT5 source carries a fourth price column `last` (last traded/
last quote price) that does NOT map into CTS_V1 (bid/ask/spread/vol). Decision
chain, based on preservation, reproducibility, future research usefulness and
schema integrity:
1. **Validated** at parse time with the same decimal micro-price grammar as
bid/ask (`PRICE_SCALE`; non-positive or over-precision → the corresponding
MALFORMED_PRICE_* class). Bid/ask relationship validation remains
mandatory; `last` is deliberately NOT cross-validated against bid/ask
(a last price may legally equal bid, ask or any value; imposing an
invariant would be an unsupported assumption).
2. **Deliberately EXCLUDED from CTS_V1**: the canonical tick schema
(and its serialization/content-id contract) is UNCHANGED — P3-DE-004 does
not re-open G-4/G-5. Promoting `last` into the canonical schema would be a
new CTS version and a new dataset version (human gate G-5).
3. **Preserved in a source-preservation layer**:
`source_preservation/chunk_%06d.jsonl`, one line per canonical row in
rec_ord order, `src_line|last_u` (e.g. `12|340345000`). The canonical preservation
serialization is the single line `src_line|last_u\n`; the preservation
chunk content id is SHA-256 over the concatenation (micro-units, integer
math only). The sidecar content id and physical SHA-256 are recorded in
the evidence manifest and independently re-verified (spec 18).
4. **Evidence basis**: the immutable raw source (certified FULL SHA-256)
remains the ultimate preservation record; the sidecar makes `last`
recoverable from canonical-run artifacts without re-reading the 34 GB
source, and keeps future research (e.g., last-price labels) derivable
under the frozen canonical identity. Dotted-grammar runs produce no
sidecars.
## 9. Chunking (Layer 3)
### 9.1 Evaluation of legacy values
- Legacy `atomic chunk = 24 MiB`, `workload = 512 MiB nominal`. Evaluation:
- 24 MiB ⇒ ~1,370 chunks for the 34.5 GB source; each chunk holds
~0.6–0.9 M tick lines (~20–40 MB RSS per worker) — excellent
checkpoint granularity, bounded worker memory, fine-grained resume.
- 512 MiB nominal workloads ⇒ ~67 workloads; matches legacy differential
comparison granularity (chunk 759/760 boundary lies on the same grid).
- Both values are RETAINED as nominal defaults for v1. Actual boundaries are
line-safe (below). These remain a human decision gate (G-7/G-8) because
they affect the differential comparison surface.
### 9.2 Chunk map (deterministic)
A single newline-scan pass over the source (no parsing) produces
`chunkmap/chunkmap-<source_id>.json`:
```
{"source_id", "chunk_bytes_nominal":25165824, "workload_bytes_nominal":536870912,
"chunks":[ {"index":i, "byte_start":b, "byte_end":e, "line_count":n,
"workload_index":w} , …],
"total_bytes", "total_chunks", "total_workloads", "chunkmap_sha256"}
```
Rules:
- `byte_start` is 0-based from file start. Chunk 0 starts at byte 0.
- A chunk's nominal end = `byte_start + chunk_bytes_nominal`; the actual
`byte_end` = first line-terminator position at or after the nominal end,
INCLUSIVE of the terminator bytes. Consequence: `byte_end` is always a line
boundary; no chunk ever splits a CSV record.
- Chunk N+1 `byte_start` = chunk N `byte_end`. Boundaries are contiguous and
cover `[0, total_bytes)` exactly.
- If the source does not end with a newline, the final line is terminated by
EOF; the last chunk ends at `total_bytes`.
- A file smaller than one chunk yields exactly one chunk.
- Workload grouping: accumulate whole chunks whose cumulative `byte_end −
byte_start` first reaches or exceeds `workload_bytes_nominal`; the final
workload contains the remaining chunks (partial final workload) and is
recorded as `partial=true`. No padding.
- `chunkmap_sha256` = SHA-256 of the deterministic JSON serialization (sorted
keys, fixed field order). The chunk map is cached per `source_id`; on resume
it is recomputed cheaply only if the source identity changed (which blocks).
### 9.3 Byte cursor semantics
- Checkpoint records `next_chunk` and `next_byte_start` (= byte_start of the
next uncommitted chunk). They always agree with the chunk map; the chunk map
is the sole authority for boundaries.
## 10. Processing Model (Layer 4)
### 10.1 Model selection
- **Canonical tick conversion**: parallel multiprocessing (spawn context,
Windows-safe), one chunk per worker task.
- **Aggregation**: sequential deterministic pass over ordered canonical ticks
(see §11). Parallel aggregation is explicitly NOT in v1; it is a documented
future optimization that must preserve identical content ids and pass the
reordered-execution determinism test before adoption.
- Rationale: conversion is CPU/parse-bound (parallel-friendly, pure per
chunk); aggregation is stateful and order-dependent (sequential is the
safest deterministic design for v1).
### 10.2 Worker & dispatch model
- `workers_requested` (config, default 24 — Gate G-2);
- `workers_created` = min(requested, `os.cpu_count()` or 8, memory
estimates) — recorded, never assumed;
- `workers_with_non_empty_partitions` = workers that actually received ≥ 1
chunk before the queue drained;
- `cpu_utilization_pct` = measured (parent-side polling), reported but never
tuned automatically.
- Dispatch: parent keeps a deterministic ordered work list (chunk indices
0..N−1) and a simple job queue (process pool). A worker pulls exactly one
chunk at a time, processes it, writes `staging/chunk_%06d.parquet.tmp`,
computes digests, fsyncs, renames to `ticks/chunk_%06d.parquet`, returns
(chunk_index, stats). The parent records the commit in the journal
(hash-chained). Completion order NEVER affects outputs: each chunk file is
independently named and content-identified; the canonical reading layer
iterates chunk indices in ascending order.
- No shared memory between workers; no queues carrying tick data; workers
only communicate via files.
### 10.3 Deterministic merge
"Merge" is a sorted concatenation by `chunk_index` (and by `rec_ord` within a
chunk). The aggregator reads chunk files in ascending index order. Because
chunk files are immutable once committed, merged output is identical for every
arrival order.
### 10.4 Worker failure handling (bounded)
| Event | Action (exactly one predefined recovery) |
|---|---|
| worker exits non-zero before rename | redispatch chunk (attempt counter); after `retry_limit=2` → FAILED |
| worker killed (crash/OOM) | redispatch (attempt counter as above); stale `.tmp` deleted by dispatcher before redispatch |
| chunk file exists but journal has no record | journal is truth; file re-verified; if digest matches → journal repaired to record commit (recorded `repaired=commit`); else re-dispatched |
| journal has record but file missing/corrupt | FAILED (material; no silent re-issue) |
| worker OOM repeatedly | attempt counter applies; then FAILED with `reason=worker_oom` |
Commit is defined as: chunk parquet renamed and fsynced AND journal line
appended and fsynced. Only both count.
## 11. Aggregation (Layer 5)
### 11.1 Timeframes and justification
Included: **M1, M5, M15, M30, H1**. Justification:
- M1 is the finest bar technically justified by tick input and is the base
unit for all coarser frames;
- M5..H1 form the standard ladder required by the research plans for
horizons of 15/30/60 min and by the legacy differential protocol (M15/M30
are legacy comparison anchors);
- no other timeframe is added; "interesting" frames are explicitly out of
scope until a research decision demands one (then a new dataset version).
Periods (ms): M1=60_000, M5=300_000, M15=900_000, M30=1_800_000, H1=3_600_000.
### 11.2 Boundary definition
- Bar boundaries are UTC clock boundaries: `period_id = floor(ts_ms / period_ms)`;
`start_ms = period_id * period_ms`; `end_ms = start_ms + period_ms`.
- A tick belongs to the bar of its `period_id`.
- Ticks are consumed in canonical order `(ts_ms ascending, then src_line
ascending)` — stable and deterministic.
- Within identical `ts_ms`, order is by `src_line` (raw file order).
### 11.3 OHLC / volume / spread (integer-exact)
- The representative bar price is the mid: `mid_u2 = bid_u + ask_u`
(units: 2×PRICE_SCALE = 2,000,000 per USD). This keeps all OHLC math in
integers with no truncation:
- `open_u2` = first mid_u2 in the bar;
- `high_u2` = max mid_u2; `low_u2` = min mid_u2; `close_u2` = last mid_u2;
- `PRICE_SCALE_BAR = 2_000_000` is recorded in the bar schema metadata.
- Volume: `vol_sum` = Σ tick vol; `tick_count` = number of ticks.
- Spread: `spread_min_u`, `spread_max_u`, `spread_sum_u`,
`spread_avg_u = spread_sum_u // tick_count` (floor division; defined).
- Audit anchors: `first_src_line`, `last_src_line`.
### 11.4 Carry state
For each timeframe the aggregator keeps exactly one in-progress bar (the
partial bar) as the carry:
```
TF carry = {period_id, start_ms, open_u2, high_u2, low_u2, close_u2,
tick_count, vol_sum, spread_min_u, spread_max_u, spread_sum_u,
last_ts_ms, last_src_line}
```
Carries are checkpointed with every commit (see §13). On resume the aggregator
verifies that its first input `ts_ms` is ≥ the carried `last_ts_ms`; otherwise
FAILED (carry mismatch = material).
### 11.5 Finalization rules
- A bar is finalized when the first tick of a later period_id arrives. It is
written to `bars/<TF>/wl_%03d.parquet` with `is_final=true`.
- At EOF, all carried partial bars are flushed with `is_final=false` and the
dataset manifest records `final_bar_partial=true` for each timeframe.
- Bar uniqueness key: `(TF, period_id)`. Append-only: a period_id is written
at most once per run. On resume, the aggregator validates that the last
written record for each TF equals the carried period_id (continuity check);
mismatch → FAILED.
- Bar global ordinal `bar_idx` (0-based per timeframe, contiguous across
parts) makes reading order-independent.
### 11.6 Bar schema (CBS_V1)
| Column | Type | Semantics |
|---|---|---|
| bar_idx | int64 | global ordinal per timeframe |
| period_id | int64 | start_ms / period_ms |
| start_ms / end_ms | int64 | UTC boundaries |
| open_u2, high_u2, low_u2, close_u2 | int64 | units = 2 × PRICE_SCALE |
| tick_count, vol_sum | int64 | counts |
| spread_min_u, spread_max_u, spread_sum_u, spread_avg_u | int64 | spread in PRICE_SCALE units |
| is_final | bool | false only for EOF partial bars |
| first_src_line, last_src_line | int64 | audit anchors |
## 12. Storage Format & Persistent Canonical Dataset (Layer 6)
### 12.1 Format decision
Default: **Apache Parquet**, compression **zstd** level 3, `write_page` layout
with pinned writer version (see §26 determinism table).
Rationale: columnar, typed (no float in canonical tabsh), compressed,
stream-appendable by immutable files, universally readable, and row-group
scannable. CSV is explicitly rejected as the default intermediate format
(no types, slow, no random access, ambiguous escaping). PyArrow version is
pinned and recorded in every manifest.
### 12.2 Layout
```
engine_output/
certification/source_certificate.json
chunkmap/chunkmap-<source_id>.json
ticks/chunk_%06d.parquet # final per-chunk columnar files
staging/… # .tmp intermediates (prunable after verification)
bars/M1/wl_%03d.parquet , bars/M5/… , bars/M15/… , bars/M30/… , bars/H1/…
malformed/malformed_%06d.jsonl
state/checkpoint.json, state/commits.jsonl, state/lock, state/control.json
logs/*.jsonl
progress.json
evidence/evidence.json
RUN_COMPLETE.json
datasets/<dataset_id>/…
verification/…
```
### 12.3 Naming rules
- `chunk_%06d` (0-padded index), `wl_%03d` (workload index), `part` derived
deterministically from workload index.
- Every parquet file embeds schema key-value metadata: `schema_version`,
`engine_version`, `parser_version`, `algorithm_version`, `source_id`,
`chunk_index` (ticks) / `timeframe` (bars), `price_scale`,
`content_id` is NOT embedded (it is computed over logical rows; see §12.4).
### 12.4 Dataset identity (content ids, not physical bytes)
- `content_id(ticks chunk i)` = `chunk_content_id` from §8.5.
- `content_id(bars wl w, TF)` = SHA-256 over canonical bar serialization:
each row as `bar_idx|period_id|open_u2|high_u2|low_u2|close_u2|tick_count|vol_sum|spread_min_u|spread_max_u|spread_sum_u|is_final|first_src_line|last_src_line\n`
concatenated in ascending `bar_idx`.
- The **canonical dataset hash** = SHA-256 over the canonical concatenation
(fixed field order) of
`content_id(ticks chunk 0) ‖ content_id(chunk 1) ‖ … ‖ content_id(bars files…)`
in sorted path order. Ordering is defined by the manifest file list.
- Physical file SHA-256s are recorded alongside content ids in `evidence.json`;
content ids are the reproducibility identity; physical hashes are integrity
evidence. This distinction is explicit and non-negotiable (Parquet physical
bytes are writer-dependent).
### 12.5 Schema metadata & versioning
Manifest record per dataset: `{"dataset_version": "DS_V1.0.0", schema_version,
engine_version, parser_version, algorithm_version, source_id, config snapshot}`.
Dataset version bumps are HUMAN DECISION GATES (G-5).
## 13. Checkpoint / Resume (Layer 7)
### 13.1 Files
- `state/checkpoint.json` — atomic (tmp+fsync+rename+dir fsync).
- `state/commits.jsonl` — append-only commit journal, each line
hash-chained to the previous:
`{"seq":n,"chunk":i,"content_id":"…","file_sha256":"…","prev":"<hash of line n-1>","ts_utc":"…"}`
- `state/lock` — PID, run_id, heartbeat. Stale detection §21.
### 13.2 Checkpoint schema (CP_V1)
```
schema_version, run_id, status,
source_identity {path,size,mtime_ns,sha256_full,cert_status,cert_path},
versions {engine, parser, algorithm, dataset, manifest schema},
chunkmap_sha256,
journal_tail {seq, hash},
last_completed_chunk, next_chunk, next_byte_start,
cumulative {rows_parsed, rows_canonical, bytes_read, malformed:{class:count}, chunks_committed,
bars_per_timeframe:{TF:count}},
carries {TF: carry object (§11.4)},
update_ts_utc,
status_reason (nullable),
checkpoint_hash (SHA-256 over the canonical JSON serialization of all fields
EXCEPT checkpoint_hash — non-self-referential)
```
### 13.3 Status machine
```
INITIALIZED → RUNNING → CHUNK_COMMITTED* → PAUSED → RUNNING → … → COMPLETED
│ │
└──────────── FAILED ──────────────────────┘
INITIALIZED / RUNNING / PAUSED / FAILED ──(resume guard)──▶ RESUME_BLOCKED
```
- `INITIALIZED` — after `init`, before first dispatch.
- `RUNNING` — while workers are dispatched.
- `CHUNK_COMMITTED` — recorded transition on every journal append (the durable
state of progress; the file is rewritten with this status after each
workload, see 13.4).
- `PAUSED` — graceful stop (signal/control file), clean checkpoint, exit 0.
- `FAILED` — material condition; processing stopped; resume is permitted only
after operator resolution and explicit re-run (see §20).
- `COMPLETED` — final after verification; final checkpoint written.
- `RESUME_BLOCKED` — a resume attempt refused entry; no state mutation.
Transitions are atomic (single rename) and fail-closed: any exception during a
checkpoint write leaves the previous valid checkpoint untouched.
### 13.4 Checkpoint frequency
Checkpoint + journal fsync after **every workload** (~67 writes for the full
run) and always before graceful pause/stop and after any worker failure that
causes re-dispatch. Chunk commit (journal append) is the fine-grained durable
record; the checkpoint file is the coarse summary. On resume, journal replay
(from the recorded tail) determines actual committed chunks — including chunks
committed after the last checkpoint write.
### 13.5 Resume procedure (deterministic)
1. Load checkpoint; verify `checkpoint_hash`.
2. Verify source identity: recompute size+mtime; require certificate present
and matching (`sha256_full` if FULLY_VERIFIED). Any mismatch →
**RESUME_BLOCKED** (never auto-recovery).
3. Verify versions (engine/parser/algorithm/dataset/schema) match exactly →
mismatch = RESUME_BLOCKED.
4. Verify `chunkmap_sha256` matches the cached map → mismatch = RESUME_BLOCKED.
5. Replay journal from `journal_tail.seq`; validate chain; derive
`next_chunk`; verify each committed chunk file exists and its content id
matches the journal → any anomaly = FAILED (material), not auto-fixed.
6. Restore carries from checkpoint; verify aggregator continuity (§11.5).
7. Resume dispatch from `next_chunk`.
### 13.6 Source identity mismatch
`source_identity` mismatch → `RESUME_BLOCKED` with `status_reason`
(`source_size_changed` | `source_mtime_changed` | `certificate_missing` |
`certificate_mismatch` | `sha256_mismatch`). The operator must re-certify and
decide (human gate) whether to start a new run.
## 14. Headless CLI (Layer 8)
### 14.1 Commands (semantics, not implemented this session)
```
sniper-data --version
sniper-data certify --source <path> [--timeout-sec N] [--output <cert>]
sniper-data init --config <engine_config.json> [--run-id <id>]
sniper-data ingest [--limit-chunks N] [--pause-after <workload|chunk> N]
sniper-data resume [--limit-chunks N]
sniper-data pause | stop
sniper-data status [--json]
sniper-data verify --mode golden | range | dataset | run-complete | legacy
[--start-byte N --end-byte M] [--samples N]
sniper-data manifest [--dataset <id>] [--print]
sniper-data build-bars [--timeframes M1,M5,M15,M30,H1]
sniper-data build-dataset --config <research_cfg.json>
sniper-data test [--quick]
sniper-data audit-dataset --dataset <id>
sniper-data diff-legacy --legacy-evidence <path> --report <out.json>
```
### 14.2 Semantics and safety properties
| Command | Meaning | Safety property |
|---|---|---|
| certify | compute certificate (read-only on source) | never opens source writable; atomic cert write |
| init | create a new run (new run_id) from a certified source; refuses if a live run/lock exists; archives prior run on explicit `--new-run` | refuses to touch legacy evidence; validates cert + config |
| ingest | start processing from INITIALIZED (or resume-equivalent allowed states) | idempotent dispatch; journal-before-checkpoint durability |
| resume | continue from PAUSED/FAILED-after-resolution; RESUME_BLOCKED otherwise | verifies every guard in §13.5; changes nothing on refusal |
| pause / stop | graceful stop at next chunk/workload boundary; checkpoint; exit | never kills a worker mid-chunk without completing its rename |
| status | print checkpoint, progress, lock, journal tail; exit code indicates completion | read-only |
| verify | run the independent verifier | refuses to run pipeline stages; only reads source/artifacts |
| manifest | build/print the dataset manifest | refuses to list missing/corrupt files |
| build-dataset | research dataset builder (§15) | reads canonical layers only; never the raw source |
| diff-legacy | differential comparison vs legacy evidence | read-only on legacy evidence |
| test | golden corpus runner | tiny synthetic inputs only |
### 14.3 Exit codes
```
0 success / COMPLETED
1 operational error (I/O, config invalid)
2 usage error
3 not completed (status: RUNNING/PAUSED/INITIALIZED)
4 BLOCKED (RESUME_BLOCKED / lock held / certification missing)
5 verification or audit failed
```
The CLI is designed so start / pause / resume / verify / status / completion
detection require no human conversation.
## 15. Research Dataset Builder (Layer 9)
### 15.1 Separation of concerns
```
DATA ENGINE → CANONICAL DATA → RESEARCH DATASET → FORECASTING EXPERIMENT → MODEL → TRADING DECISION
```
The Engine provides canonical ticks+bars and a fixed library of deterministic
feature/label primitives. It holds NO forecasting hypothesis. Examples such as
"M15 + 15m direction", "M15 + 30m return", "M30 + 60m return" are research
configurations, never ingestion rules.
### 15.2 Research config schema (strict; hash → config_sha256)
```
{
"dataset_id": "xau_m15_15m_dir_v1",
"source_dataset_version": "DS_V1.0.0",
"timeframe": "M15",
"prediction_horizon_min": 15,
"target": {"type": "direction|cls_3|return_reg", "horizon": "15m", "from": "close"},
"features": [{"name":..., "primitive": "return|log_return|volatility|range|spread_stats|rolling_*",
"window": k, "args": {...}}],
"date_range": ["2015-01-01","2020-12-31"],
"split": {"policy": "time_ordered", "train":0.7,"val":0.15,"test":0.15,
"oos_barrier": true, "barrier_gap_min": 15},
"cost_model": {"type":"fixed_spread","spread_units":…,"commission_per_lot":…,"lot_size":…},
"label_rules": {"min_abs_move_units":…,"neutral_zone":…}}
}
```
Rules:
- Inputs: the canonical bar layer of `source_dataset_version` ONLY.
- Feature/label primitives are deterministic pure functions of bars
(integer or pinned-float operations, fixed evaluation order).
- Leakage rule: the label for row t uses bars `(t, t+h]` only; rows lacking
`h` future bars are excluded and counted in the manifest
(`rows_excluded_no_future`).
- Split: strictly time-ordered; `oos_barrier` demands a gap of
≥ horizon between the last training bar and the first test bar (in bars),
recorded in the manifest.
- All boundaries are (row_id, bar_idx) based, never shuffled at row level.
- Float math is allowed HERE (research layer), but the pipeline is
deterministic: pinned library versions, fixed operation order, no hash
randomization, no set-iteration in serialization paths.
### 15.3 Outputs
```
datasets/<dataset_id>/
config.json, config_sha256,
features.parquet, labels.parquet, dataset.parquet (row_id, ts, features, label, split),
split.parquet (row_id → split), manifest.json
```
## 16. Dataset Manifest (Layer 10)
`manifest.json` schema (MS_V1) — applies to the canonical dataset and to every
research dataset:
```
schema_version, dataset_id, dataset_version, engine_version, parser_version,
algorithm_version, source_identity {certificate reference},
timeframe (research only), prediction_horizon_min, target_definition,
features (list), date_range, split_definition, oos_barrier, cost_model,
row_counts {total, per_split, excluded_no_future, malformed…},
files [ {relpath, bytes, sha256_physical, content_id} ], # sorted by relpath
config_snapshot, config_sha256,
creation_ts_utc,
manifest_hash, # H(canonical JSON of manifest WITHOUT manifest_hash and dataset_hash)
dataset_hash # H(canonical concat of (relpath|content_id) list) — excludes manifest_hash
```
Non-self-referential hashing: `manifest_hash` explicitly excludes both hash
fields; `dataset_hash` is computed over file content ids only (plus version
identity fields), never over the manifest that contains it. No hash cycle.
## 17. Evidence & Hashing Strategy (Layer 11)
### 17.1 Evidence hierarchy
```
SOURCE
↓ certification (FULLY_VERIFIED gate)
CANONICAL TICKS content_id per chunk; dataset hash
CANONICAL BARS content_id per workload/TF file
DERIVED FEATURES content_id per dataset file
LABELS content_id per dataset file
RESEARCH DATASET dataset_hash per manifest
```
### 17.2 Identity definitions
- **run_id**: `RUN-<YYYYMMDD-HHMMSS>-<uuid4 first 8>`. Assigned at `init`.
- **file hash**: SHA-256 of physical bytes (`sha256_physical`).
- **content_id**: layer-specific canonical logical digest (§8.5, §12.4).
- **aggregate/layer hash**: SHA-256 over the canonical concatenation of the
layer's content ids in manifest-sorted order.
- **manifest hash** / **dataset hash**: see §16 (no self-reference).
- **checkpoint hash**: SHA-256 of the checkpoint payload excluding the hash
field itself (§13.2).
- **evidence manifest** (`evidence/evidence.json`): lists every produced file
with both hashes, plus `evidence_manifest_hash` computed over itself minus
the hash field. `RUN_COMPLETE.json` records the `evidence_manifest_hash`;
no cycle (RUN_COMPLETE.json is not hashed by evidence.json).
### 17.3 Rules
- All hash inputs use fixed canonical serialization (sorted keys at each JSON
level, `\n` line endings, no trailing whitespace).
- No hash is ever computed over a blob that contains that same hash.
- `PYTHONHASHSEED` is irrelevant by construction (no set/dict iteration in any
serialization path); the Engine additionally pins `PYTHONHASHSEED=0` as a
belt-and-braces control.
## 18. Independent Verification (Layer 12)
### 18.1 Principle
The verifier (`engine/verify/`) is a SEPARATE implementation written from this
specification's tables — its own line splitter, timestamp parser, price
parser, and aggregator. It does NOT import or call producer parsing,
aggregation, or serialization code. Producer and verifier may share only the
version constant tables.
### 18.2 Verification modes
| Mode | What it does |
|---|---|
| golden | runs the golden corpus through BOTH implementations; byte-equal expected outputs |
| range | re-parses selected source byte ranges with the verifier parser; compares canonical rows, ts, bid/ask against chunk parquet logical rows |
| dataset | recomputes every content_id and dataset_hash from artifacts (independent of producer); checks manifest consistency |
| run-complete | validates RUN_COMPLETE.json plus evidence manifest against artifacts |
| legacy | differential comparison vs legacy evidence (§24) |
### 18.3 Verification techniques
- **Golden vectors**: hand-computed expected outputs for tiny inputs.
- **Deterministic invariants** (at least): OHLC validity
(low ≤ open,close ≤ high; counts ≥ 1), ts_ms non-decreasing across merged
stream, Σ per-chunk tick counts == total tick count, bar continuity
(close of bar k == first tick of bar k+1 order boundary, carry continuity),
content_id stability across producer/verifier, append-only uniqueness of
(TF, period_id).
- **Byte-range verification**: independent parse+aggregate over chosen ranges,
compared to chunk artifacts.
- **Hash verification**: full recomputation from files.
- **Differential comparison** vs legacy (§24).
- **Mutation tests**: flip a byte / duplicate a row / drop a row / alter a
timestamp in a golden corpus COPY (never the source) → engine must change
digests or classify per taxonomy.
- **Boundary tests**: chunk boundary exactly at a terminator; boundary inside
a line; CRLF/LF/CR/mixed inputs.
- **Resume tests**: kill+resume cycles must reproduce identical content ids.
- **Reordered-worker tests**: same input dispatched in different orders →
identical content ids.
- **Duplicated/missing data tests**: injected duplicates/missing rows must be
detected by invariants (counters/hash changes).
### 18.4 Verdicts
`verification/report_<run>.json` with per-check
`PASS|FAIL|SKIP(reason)|DIFF(classification)` and an overall
`VERIFIER_ACCEPTED | REJECTED`. A FAILED invariant or a failed independent
content-id comparison is material (§20).
## 19. Golden Test Corpus (Layer 13)
Small, permanent, adversarial; each input < 100 KiB; full run < 10 s.
| Case | Input characteristics | Expected effect |
|---|---|---|
| G01 CRLF | terminator `\r\n` everywhere | parse ok; counts exact |
| G02 LF | `\n` only | parse ok; identical canonical ids to G01 |
| G03 header present | first row header | skipped; has_header=true |
| G04 header absent | data only | all rows data |
| G05 boundary at exact byte | chunk cut lands exactly on terminator | one line per chunk |
| G06 boundary inside line | chunk cut lands mid-line | chunk map extends boundary to line end; no split record |
| G07 malformed timestamp | `2023.01.01 12:00:00` variants / garbage | MALFORMED_TIMESTAMP_PARSE count |
| G08 malformed price | `abc` / signed / extra decimals | MALFORMED_PRICE_* classes |
| G09 invalid bid/ask | ask < bid | MALFORMED_BID_ASK_RELATION |
| G10 duplicated row | exact duplicate in chunk | MALFORMED_DUPLICATE + digest sensitivity |
| G11 dropped row | verifier-independent count mismatch | invariant FAIL (verification-time) |
| G12 non-monotonic | regressed ts | MALFORMED_NON_MONOTONIC |
| G13 worker reorder | shuffled dispatch of same input | identical content ids |
| G14 missing input carry | resume without carry state | RESUME_BLOCKED / continuity FAIL |
| G15 altered state | corrupt checkpoint hash / journal chain | RESUME_BLOCKED / FAILED |
| G16 repeated resume | pause+resume+resume | identical content ids (determinism proof) |
| G17 already-transformed data | canonical-shaped input fed as raw | format mismatch → certification fails / blocked |
Source-grammar golden cases SG01–SG15 (P3-DATA-ENGINE-004) extend the corpus
with dedicated synthetic fixtures for the authoritative six-column
G_TICKSTORY_MT5 grammar: SG01 grammar acceptance, SG02 CRLF handling,
SG03 valid six-column row, SG04 wrong field count, SG05 invalid date,
SG06 invalid time, SG07 invalid bid, SG08 invalid ask, SG09 invalid last,
SG10 invalid volume, SG11 bid>ask, SG12 zero/negative volume, SG13 timestamp
conversion (regression anchor `20030505,00:01:03` → `1_052_092_863_000` ms),
SG14 decimal precision, SG15 header absence/presence semantics. Like G01–G17,
every SG case requires producer == independent verifier == expected values.
Each case has an `expected/` JSON (counters, classes, content ids) that both
the producer and the verifier must reproduce.
## 20. Failure Model (Layer 14)
Three classes. The Engine's explicit action set is: continue (with counter),
apply ONE predefined recovery, or STOP.
| # | Condition | Class | Action | Result state |
|---|---|---|---|---|
| F01 | malformed row (any taxon) | EXPECTED | classify + counter + sidecar | continue |
| F02 | malformed rate > 0.001 or >10,000 in a workload | MATERIAL | STOP; record reason | FAILED |
| F03 | blank lines / BOM / header | EXPECTED | deterministic skip | continue |
| F04 | worker non-zero exit | BOUNDED | ONE recovery: redispatch, ≤2 attempts | continue/FAILED after limit |
| F05 | worker OOM | BOUNDED | ONE recovery: redispatch ≤2 | FAILED(`worker_oom`) after limit |
| F06 | chunk file present, no journal | BOUNDED | ONE recovery: verify digest, repair journal OR redispatch | continue |
| F07 | journal says committed, file missing/digest mismatch | MATERIAL | STOP | FAILED |
| F08 | checkpoint hash mismatch | MATERIAL | STOP; no auto-recovery | RESUME_BLOCKED |
| F09 | source size/mtime/cert mismatch | MATERIAL | STOP | RESUME_BLOCKED |
| F10 | version mismatch (engine/parser/algorithm/dataset/schema) | MATERIAL | STOP | RESUME_BLOCKED |
| F11 | carry continuity / bar append mismatch | MATERIAL | STOP | FAILED |
| F12 | disk free below floor at runtime | BOUNDED | ONE recovery: graceful pause | PAUSED(reason=disk_space); resume after human frees space |
| F13 | disk free below floor at init | MATERIAL | STOP | init blocked |
| F14 | lock held by live process | MATERIAL | STOP (no kill) | exit 4 (locked) |
| F15 | unknown exception in canonical writer/aggregator | MATERIAL | STOP | FAILED |
| F16 | verifier verdict REJECTED | MATERIAL | STOP before COMPLETED | FAILED(verification) |
| F17 | malformed threshold inside chunk (pre-workload) | EXPECTED | carry counters | continue (workload check at end) |
Rules:
- No autonomous recovery-by-reasoning. No silent fallback. No broad self-repair.
- Every STOP writes a checkpoint (atomic), a `status_reason`, and a
`logs/failure_<run>.jsonl` entry with the exact condition code.
- `FAILED` runs resume only after an operator (human) resolves the root cause
and re-invokes `resume` or starts a new run; the Engine never decides to
bypass F07–F16.
## 21. Resource Management (Layer 15)
| Resource | Requirement |
|---|---|
| Max memory, parent | ≤ 512 MB RSS (dispatcher/coordinator only) |
| Max memory, worker | ≤ 1 GiB RSS per worker (chunk-bounded; one chunk in memory) |
| Total memory target | requested workers × per-worker bound; never exceeds measured machine memory / 2 |
| Worker isolation | separate processes (`spawn`); worker crash cannot corrupt parent |
| Logging | JSONL, one operator log + per-worker logs; rotation at 100 MB, 5 files; logs never influence outputs |
| Progress | `progress.json`: run_id, status, chunks done/total, bytes, workloads, workers{requested, created, with_data, busy, cpu_util_pct}, started/updated ts, eta_sec |
| Checkpoint frequency | journal per chunk commit; checkpoint file per workload; always at pause/fail |
| Graceful shutdown | SIGINT/SIGTERM or `state/control.json` → finish current chunk, join workers (≤300 s), checkpoint PAUSED, exit 0 |
| OS restart recovery | any interruption leaves only orphan `.tmp`; resume (§13.5) restores exactly |
| Disk-space checks | preflight: free ≥ max(2×estimated_canonical_size, 20 GiB); per-workload: free ≥ 4 GiB else PAUSED(disk_space) |
| Output-space checks | staging cleaned after successful verification; `--prune-staging` explicit only |
| Process cleanup | atexit + signal handlers remove lock; children joined; no zombie policy beyond retry |
| Stale-process detection | lock heartbeat every 60 s; stale if >300 s old; resume with a stale lock → exit 4 and `status` reports `lock_stale`; releasing requires explicit `--force-release-lock` (human gate G-11) — default: refuse |
The Engine never relies on AI session lifetime; all of the above executes
under the OS scheduler.
## 22. Run Completion Protocol (Layer 16)
`RUN_COMPLETE.json` is produced ONLY after: all chunks committed, aggregator
flushed, evidence manifest computed, and the independent verifier
(`verify --mode dataset` and `--mode run-complete`) returned ACCEPTED.
Schema (RC_V1):
```
status: "COMPLETED",
run_id, source_identity {ref to certificate},
versions {engine, parser, algorithm, dataset, manifest schema},
rows {parsed, canonical, per-chunk-min/max},
malformed {per class, total},
outputs {ticks_files, bar_files_per_TF, dataset_files},
checkpoint {last_completed_chunk, next_chunk(=total), status},
hashes {canonical_dataset_hash, manifest_hash, evidence_manifest_hash,
per-layer aggregate hashes},
completion_ts_utc, verifier_report (path + verdict)
```
Machine-verifiability: `sniper-data verify --mode run-complete` re-checks every
hash and file from the disk; exit code 0 ⇔ genuine completion.
Operator flow: run the Engine headlessly; hours later, independently of any AI
session, run `sniper-data status` (→ completion detection), then hand the
result package (RUN_COMPLETE.json + evidence + manifests) to ChatGPT for audit.
## 23. Clean Reprocessing Protocol (Layer 17)
Exact future procedure (NOT executed in this session; each step has an
authorization requirement):
| # | Step | Entry condition | Authority |
|---|---|---|---|
| 1 | Freeze legacy pipeline | approved; legacy remains reference-only, evidence preserved | G-12 |
| 2 | Certify source | step 1; certificate FULLY_VERIFIED preferred | G-3 |
| 3 | Initialize NEW engine | certificate accepted; new run_id (not checkpoint 759) | G-4 |
| 4 | Run golden tests | `sniper-data test` green (producer + verifier) | — |
| 5 | Run small real-data pilot | e.g., first 2 workloads or a configurable 512 MiB window | G-13 (pilot scope) |
| 6 | Independently verify pilot | verifier PASS on pilot ranges | — |
| 7 | START full 34 GB processing | step 6 accepted | G-14 (full reprocessing) |
| 8 | Allow Python to run independently | no AI required; resume as needed | — |
| 9 | Resume if interrupted | §13.5; no AI required | — |
| 10 | Complete full dataset | status COMPLETED + RUN_COMPLETE.json | — |
| 11 | Independently certify final dataset | verifier ACCEPTED on full dataset | — |
| 12 | Freeze canonical research dataset | mark frozen in manifest; publish hashes | G-15 (freeze) |
The legacy pipeline is not extended during or after this procedure. Legacy
checkpoint 759 and all workload evidence remain untouched.
## 24. Legacy Comparison Protocol
### 24.1 Role of the legacy pipeline
The P3-S25 pipeline is a REFERENCE, not an unquestionable oracle. Legacy cursor
(chunks 738–759 / workload 45 completed, next chunk 760 at byte
18,442,355,762) is preserved and read-only. The new Engine does not inherit it.
### 24.2 Differential verification scope
```
NEW ENGINE ── selected byte ranges ──▶ canonical ticks (ts, bid, ask)
── selected ticks ─────────▶ row-level samples
── timestamps ─────────────▶ M15/M30 bar boundaries
── M15/M30 bars ───────────▶ OHLC/tick counts
── cumulative counters ────▶ parsed/canonical/malformed totals
── hashes ─────────────────▶ layer digests
▼
LEGACY EVIDENCE
```
Selection set (deterministic): first 64 KiB; the 24 MiB range preceding legacy
chunk 759/760 boundary; and 8 evenly spaced sample ranges across the file
(config-driven; each ≤ 24 MiB).
### 24.3 Discrepancy classification (mandatory taxonomy)
| Class | Meaning | Auto-action |
|---|---|---|
| EXPECTED_SEMANTIC | documented spec difference (e.g., mid-based OHLC, UTC alignment, carry policy) | record, no alarm |
| IMPLEMENTATION_DIFFERENCE | same semantics, different internals | record; verify both |
| LEGACY_DEFECT | legacy evidence contradicts the source within known legacy scope | record; escalate (G-16) |
| NEW_ENGINE_DEFECT | new Engine deviates from its own spec | FAILED; fix before proceeding |
| UNRESOLVED | cannot classify from available evidence | record; escalate (G-16); never auto-assume |
Output: `verification/legacy_diff_<run>.json` with per-item
range/tick/bar/counter/hash evidence and a classification. The Legacy
Comparison stage never modifies legacy files and never chooses an answer on
the operator's behalf; UNRESOLVED entries are G-16 material.
## 25. Security & Integrity Controls
1. Source is opened `rb` only; the Engine never holds a writable handle to the
raw source and fails if `stat` changes during certification (`F09`).
2. All outputs live under `engine_output/`; path validation (resolved absolute
paths under the configured write root) rejects `..`, symlink/junction
escape, and absolute paths outside the root.
3. No network access, no credentials, no account data, no secrets, no eval of
config (config validated against JSON schema).
4. Checkpoint and journal are hash-chained (`checkpoint_hash`, per-line
`prev`); optional HMAC layer (key file outside repo) is a NON-BLOCKING
future hardening (G-17).
5. Every produced artifact carries `sha256_physical` + `content_id`
(§17); evidence manifest enumerates all files with both.
6. Legacy evidence directories are excluded from every access path the Engine
may write; a runtime guard refuses any write whose target path is not under
the Engine's own output root.
7. Logs are sanitized (no raw bid/ask values, no account identifiers).
8. Malformed sidecar rows contain hashes (`raw_sha256`), not raw text.
## 26. Versioning
| Constant | Value (v1) | Bump rule |
|---|---|---|
| ENGINE_VERSION | 1.1.0 | any Engine behavior change (semver) |
| PARSER_VERSION | PARSER_V1.1 | any change to CSV/timestamp/price parsing rules (P3-DE-004: extended with G_TICKSTORY_MT5; unambiguous vs the failed PARSER_V1) |
| GRAMMAR REGISTRY | G_TICKSTORY_MT5 / G_DOTTED_V1 | named presets, certificate-pinned (P3-DE-004; §8.1) |
| ALGORITHM_VERSION | ALG_V1 | any change to canonical serialization, chunking rule, or aggregation semantics (unchanged by P3-DE-004) |
| SCHEMA_VERSION (ticks) | CTS_V1 | schema change |
| SCHEMA_VERSION (bars) | CBS_V1 | bar schema change |
| DATASET_VERSION | DS_V1.0.0 | human decision gate only (G-5) |
| MANIFEST_SCHEMA_VERSION | MS_V1 | manifest schema change |
| CHECKPOINT_SCHEMA_VERSION | CP_V1 | checkpoint schema change |
| RUN_COMPLETE_SCHEMA_VERSION | RC_V1 | completion schema change |
Determinism pins (recorded in every manifest): Python ≥ 3.11, pyarrow exact
version, pandas (research layer) exact version, `PYTHONHASHSEED=0`, Parquet
writer version string. Any pin change bumps ENGINE_VERSION and requires
re-running the golden corpus and determinism tests before further runs.
Version mismatch on resume = RESUME_BLOCKED (F10). No automatic data migration;
schema migrations are new dataset versions (G-5).
## 27. Acceptance Criteria
The Engine is accepted for full reprocessing only when ALL hold:
1. Golden corpus G01–G17 passes under both producer and verifier.
1a. Source-grammar corpus SG01–SG15 (G_TICKSTORY_MT5 six-column grammar) passes
under both producer and verifier with the regression timestamp anchor
(`20030505,00:01:03` → `1_052_092_863_000` ms at tz offset 0), the
compact-grammar end-to-end pipeline run, and the certificate/grammar
fail-closed cross-check all green.
1b. CLI headless E2E includes the run-less-directory regression: `status` and
`resume` on an uninitialized output directory MUST both exit 4 (BLOCKED)
without a traceback.
2. Independent verifier accepts the pilot (step 6).
3. Resume invariance: content ids of a run interrupted 3× and resumed are
identical to an uninterrupted run of the same range.
4. `verify --mode dataset` recomputes all content ids and the dataset hash
from disk independently and matches.
5. `verify --mode run-complete` returns exit 0 only on a genuine COMPLETED.
6. Differential report vs legacy is produced with every item classified.
7. Source certificate: FULL_SOURCE_SHA256_VERIFIED achieved or the run did not
start without G-3 authorization.
8. Malformed rates within limits; malformed sidecar complete.
9. Research dataset regeneration: two runs of `build-dataset` with the same
config produce identical dataset hashes without touching the raw source.
10. Engine completes a full 34.5 GB pass headlessly (AI closed), including at
least one forced resume.
11. Legacy evidence directories byte-identical before/after full run.
12. Source file size and mtime unchanged by the full run.
## 28. Risks & Mitigations
| # | Risk | Mitigation |
|---|---|---|
| R-TZ | Server-time/timezone ambiguity (DST) shifts bar boundaries | canonical ts is a pure function of text; re-derivation with new offset = new dataset version; manifest records offset; G-6 |
| R-DISK | 34.5 GB source + canonical outputs exceed free space | preflight floor; per-workload floor; PAUSED(disk_space) |
| R-HW | Disk/CPU failure mid-run | atomic commits; journal truth; resume |
| R-AV | Antivirus/OneDrive locking files | rename+fsync pattern; documented deployment requirement; retry on transient EACCES (bounded) |
| R-DETS | Library/dependency drift breaks determinism | pinned versions recorded; golden corpus gate; upgrade bumps ENGINE_VERSION |
| R-MEM | Parent or worker memory blowup | chunk-bounded workers; memory telemetry; retry/FAILED policy |
| R-DUP | Duplicate/overlapping data in source (Tickstory known cases) | taxonomy counters; verifier invariants; research layer documents dedup policy per dataset config |
| R-LEG | Legacy evidence drift/incomplete | legacy dirs read-only; differential classification taxonomy |
| R-GAP | Weekend/illiquid gaps produce zero-tick bars | bars exist only when ticks exist (no empty-bar inflation); gap analysis is research-layer |
| R-CERT | Full SHA-256 too slow on target hardware | certification budget; VERIFIED_WITH_LIMITATION is explicit and gated (G-3) |
| R-SRC | Source replaced/moved by another process | size/mtime re-checks; cert binding; RESUME_BLOCKED |
| R-SPEC | Spec ambiguity discovered at implementation | every ambiguity must be filed as an Open Decision, not silently assumed |
## 29. Open Decisions Requiring Human Authorization
Every item below is a HUMAN DECISION GATE. The Engine must NOT proceed past
the gate without an explicit owner decision recorded in
`docs/CURRENT_DECISION_GATE.md` (or successor register).
| # | Decision | Recommended default | Consequence if not authorized |
|---|---|---|---|
| G-1 | Final storage format (Parquet + zstd-3 + pinned pyarrow) | accept default | init blocked |
| G-2 | Default worker count (24; created = min(24, cpu_count, mem-constrained)) | accept default | ingest uses 1 worker until decided |
| G-3 | Source certification acceptance: require FULLY_VERIFIED; allow VERIFIED_WITH_LIMITATION only with explicit authorization | FULLY_VERIFIED only | no run under limitation |
| G-4 | Canonical schema CTS_V1/CBS_V1 (columns, PRICE_SCALE=1e6, mid-based OHLC with PRICE_SCALE_BAR=2e6, vol semantics) | accept default | canonicalization blocked |
| G-5 | Dataset version identity & bump policy (DS_X.Y.Z, human-only bumps) | accept default | freeze/dataset publish blocked |
| G-6 | Source timezone offset (default config `source_tz_offset_minutes`; DST not modeled in v1) | record + document | canonicalization blocked |
| G-7 | Atomic chunk size (24 MiB nominal, line-safe) | accept default | chunkmap frozen with legacy grid |
| G-8 | Workload size (512 MiB nominal) | accept default | workload grouping frozen |
| G-9 | Aggregation semantics (UTC clock boundaries, mid OHLC, spread min/max/avg floor, carry policy) | accept default | bar layer blocked |
| G-10 | Bar partial policy at EOF (flush as is_final=false) & no empty-bar inflation | accept default | bar layer publish blocked |
| G-11 | Stale-lock release policy (default: refuse; `--force-release-lock` explicit) | refuse by default | blocked while stale lock present |
| G-12 | Freeze legacy pipeline as reference-only | after G-1..G-10 | reprocessing not authorized |
| G-13 | Pilot scope (default first 512 MiB window) | accept default | pilot not run |
| G-14 | FULL 34 GB REPROCESSING authorization | after pilot verification | step 7 never starts |
| G-15 | Freeze of canonical research dataset + publish hashes | after independent certification | step 12 not executed |
| G-16 | Legacy-vs-new discrepancy resolution (per UNRESOLVED item) | adjudicate per item | run stays paused |
| G-17 | Optional checkpoint HMAC hardening | non-blocking future | no impact on v1 |
Additionally: any research target definition (Layer 9 configs such as
M15+15m direction) is owner-authorized research configuration and is out of
engine scope until G-14 completes. No gate is decided by the Engine.
Approved governance clarifications (P3-DATA-ENGINE-001 GOV-CLOSE, human decision baseline 2026-09-07):
- G-14 remains NOT AUTHORIZED even after design freeze.
- G-6 approval does not define the actual UTC offset value; `source_tz_offset_minutes` must be established explicitly at source initialization.
- G-12 is a legacy freeze/reference decision and does not mean the legacy checkpoint becomes a new-engine checkpoint.
---
## Appendix A — Final Review Gate (design self-check)
| # | Question | Answer |
|---|---|---|
| 1 | Works with ChatGPT completely closed? | YES — headless CLI, no AI dependency anywhere in the pipeline (§4.3, §14) |
| 2 | Windows restart interrupts without corrupting state? | YES — atomic tmp+rename, journal truth, orphan-tmp cleanup (§13) |
| 3 | Resume without AI intervention? | YES — §13.5 procedure is deterministic `sniper-data resume` |
| 4 | Source identity immutable? | YES — read-only handles, cert binding, F09 (§7, §25) |
| 5 | Full source SHA requirement explicit? | YES — required; limitation is explicit + gated (§7.4–7.5, G-3) |
| 6 | Canonical dataset reproducible? | YES — content-id contract independent of physical bytes (§12.4) |
| 7 | Research datasets regenerable without re-reading raw ticks? | YES — canonical bar layer only (§15) |
| 8 | Verifier independent of producer validation? | YES — separate implementation, no shared parse/aggregate code (§18) |
| 9 | Discrepancy old vs new diagnosable? | YES — legacy comparison taxonomy (§24) |
| 10 | Human decisions clearly identified? | YES — §29 gates G-1..G-17 |
| 11 | Hidden dependency on future AI interaction? | NO — none |
| 12 | Any path silently changing semantics? | NO — version gates, RESUME_BLOCKED, fail-closed states (§13, §20, §26) |
Non-conformance items, if any, are filed in §29 rather than silently assumed.
This table records DESIGN COMPLETE.
## Appendix B — Glossary
| Term | Definition |
|---|---|
| chunk | line-safe byte range of the raw source (§9) |
| workload | deterministic group of consecutive chunks (§9.2) |
| content_id | canonical logical digest of a layer file (§12.4) |
| journal | append-only hash-chained commit record (Layer 4/7) |
| carry | in-progress partial bar per timeframe (§11.4) |
| material condition | a failure that forces STOP and operator resolution (§20) |
| expected class | a deterministic, counted, non-fatal data anomaly (§20) |
---
*End of specification. This document is the only artifact produced by the design session. Nothing in it authorizes execution.*
---
# Appendix C — P3-DATA-ENGINE-004 CONTROLLED AMENDMENT RECORD
Session type: CONTROLLED SPECIFICATION AMENDMENT + IMPLEMENTATION
RE-QUALIFICATION ONLY (authorized by the owner after the P3-DATA-ENGINE-003
pilot finding; DESIGN remains FROZEN; no unrelated governance decision is
re-opened). All amendments below are traceable to the P3-DATA-ENGINE-003
evidence: source `XAUUSD_mt5_ticks.csv` (size 34,473,661,010 bytes; FULL
SHA-256 `5252ce8f5b0e7d286b71a3172c7ef9c1009c8e4b1254ac67400a803fcd85cba1`;
CRLF; ASCII; no header; first raw row `20030505,00:01:03,…`).
## C.1 Source grammar (authoritative; supersedes the 8.1 dotted primary)
G_TICKSTORY_MT5 :: YYYYMMDD , HH:MM:SS , bid , ask , last , volume
field count :: exactly 6 (else MALFORMED_FIELD_COUNT)
date :: 8 digits, month 01..12, day 01..31
time :: HH:MM:SS, hh 00..23, mi 00..59, ss 00..59 (no ms)
bid/ask/last :: decimal micro-price (PRICE_SCALE = 1e6; 0..6 fraction
digits); non-positive -> MALFORMED_PRICE_NONPOSITIVE;
ask < bid -> MALFORMED_BID_ASK_RELATION (mandatory)
volume :: non-negative integer; 0 VALID; negative/non-integer
-> MALFORMED_VOLUME (has_volume=true enforced)
delimiter :: comma only
line ending :: CRLF/LF/CR accepted deterministically; recorded
header :: none in the real source; keyword-detected header
(date|time|datetime prefix) skipped + has_header=true;
absence verified by evidence
Grammar selection is certificate-bound: `grammar_id` in
`source_certificate.json` must equal the configured `source_grammar`, else
`init` is BLOCKED (spec 8.1 certificate/config mismatch). No silent generic
format acceptance.
## C.2 Timestamp
`ts_ms = wallclock_epoch_ms(YYYYMMDD, HH:MM:SS, ms=0) -
source_tz_offset_minutes * 60000`, integer civil-day math, deterministic.
Production `source_tz_offset_minutes = 0` preserved (P3-DE-003 evidence
chain: first row `20030505,00:01:03` -> epoch `1,052,092,863` s equivalent
`1,052,092,863,000` ms == legacy chunk0 first_timestamp; G-6 policy; DST not
modeled in v1). Regression-anchored by SG13.
## C.3 Volume / last-field / versions
- Volume: see 8.2 amendment (mandatory source volume for G_TICKSTORY_MT5;
`vol = 1` substitution FORBIDDEN for this source; zero valid).
- `last`: validated; EXCLUDED from CTS_V1; preserved in
`source_preservation/chunk_%06d.jsonl` (spec 8.6); promotion to the
canonical schema = new CTS version + G-5 gate.
- Versioning: ENGINE_VERSION 1.1.0; PARSER_VERSION PARSER_V1.1;
ALGORITHM_VERSION ALG_V1 (unchanged); SCHEMA_VERSION CTS_V1/CBS_V1
(unchanged); DATASET_VERSION DS_V1.0.0 (unchanged; G-5).
- G-9 (aggregation semantics) and G-10 (partial-bar policy): UNCHANGED.
- G-14: UNCHANGED — BLOCKED / NOT AUTHORIZED.
- Legacy state: checkpoint `759 -> 760 @ byte 18,442,355,762` UNCHANGED;
workload 46 NOT prepared; legacy pipeline byte-identical and read-only.
## C.4 Implementation-visible changes
- Parser/source-adapter: two named grammar presets (8.1 registry); compact
timestamp parse; `last` validation + preservation sidecar; volume
semantics; certificate `grammar_id` + `compact_date_time` sniffing.
- CLI defect repair: `status`/`resume` on a run-less directory now exit 4
(BLOCKED) instead of exit 1 + traceback (P3-DE-003 finding;
`NEW_ENGINE_DEFECT`, non-data-affecting; regression in CLI E2E).
- Qualification suite extended: SG01–SG15 source-grammar golden cases +
compact-grammar end-to-end + dependent acceptance criteria (27).
## C.5 Scope exclusions (unchanged)
No pilot execution, no 34 GB source processing, no full-source hashing for
ingestion purposes, no workload 46, no chunk 760+ preparation, no legacy
checkpoint/pipeline modification, no research dataset generation, no
forecasting, no G-14 authorization.
View git blame Copy permalink