SniperGold_ML/docs/P3_DATA_ENGINE_002_IMPLEMENTATION_MAP.md

P3_DATA_ENGINE_002_IMPLEMENTATION_MAP

Data Engine v1 — Requirements to Implementation Responsibility Map

Session: P3-DATA-ENGINE-002 (IMPLEMENTATION + QUALIFICATION ONLY) Specification: docs/P3_DATA_ENGINE_V1_SPEC.md (DESIGN FROZEN, SHA-256 84bf0f217ffba51197112a6bbacbcc297058e04b5ca47f0459028fe33e0321e5) Governance: docs/P3_DATA_ENGINE_V1_GOV_DECISION_RECORD.md (G-1..G-13, G-15, G-16 APPROVED; G-14 NOT AUTHORIZED; G-17 DEFERRED)

Every implementation responsibility below is traceable to a specification section. Every critical behavior has at least one test (unit / golden / property / adversarial / mutation / independent verifier).

---

1. Requirement → implementation responsibility

Spec ref	Requirement	Implementation responsibility (file)	Tests
7.2–7.5 (Layer 1)	Source certification: size, mtime_ns, full SHA-256, format facts; atomic cert; FULLY_VERIFIED vs VERIFIED_WITH_LIMITATION vs FAILED; no silent claim of full hash	engine/certify.py	unit ts/format sniff; golden G17 (unrecognized format); timeout path unit
8.1 (Layer 2)	Deterministic CSV interpretation: UTF-8/BOM strip, comma delimiter, CRLF/LF/CR splitting, blank-line skip, header detection rule	engine/parse.py + engine/chunkmap.py (line detector)	unit; golden G01..G06; adversarial CRLF/LF/mixed
8.1	Timestamp parsing (dotted ms primary, no-ms secondary) + epoch ms int math + range rule	engine/parse.py + engine/util.py	unit; golden G07; adversarial malformed ts
8.2	Decimal price parsing to int (PRICE_SCALE=1_000_000), no floats; precision rule; nonpositive; bid/ask relation; volume rules	engine/util.py + engine/parse.py	unit; golden G08/G09; adversarial
8.3	Complete malformed taxonomy + sidecar (chunk_index, byte_start, global_line_no, class, raw_sha256, reason) + counters; never fabricate/skip silently	engine/parse.py (taxonomy + sidecar records)	unit; golden G07..G12; property "no silent malformed transformation"
8.4	CTS_V1 tick schema (all int64; no floats/nulls/strings)	engine/storage.py (parquet schema)	unit; verifier row-level compare
8.5	Canonical serialization `ts_ms	bid_u	ask_u
9.2–9.3 (Layer 3)	Line-safe chunk map; nominal 24 MiB / 512 MiB; inclusive terminator byte_end; workload grouping (partial final); chunkmap_sha256	engine/chunkmap.py	unit; golden G05/G06; property no-gap/no-overlap; adversarial boundary cases
10.1–10.4 (Layer 4)	Worker model: spawn, one chunk/worker, staging tmp+rename, digests before rename, bounded redispatch (retry_limit=2), journal truth, output-order independence	engine/worker.py + engine/dispatcher.py + engine/journal.py	golden G13 (worker reorder); property worker-schedule invariance; adversarial worker reorder/crash
11.1–11.6 (Layer 5)	M1/M5/M15/M30/H1; UTC clock boundaries; mid_u2 OHLC; spread min/max/sum/avg(floor); carry per TF; is_final=false EOF partial; no empty-bar inflation; bar_idx continuous; (TF,period_id) uniqueness	engine/aggregate.py	unit bar-period; unit aggregation; golden (bar ids); property no-gap/no-empty-bar; partial-bar unit
12.1–12.5 (Layer 6)	Parquet + zstd-3 + pinned pyarrow; layout/naming; schema KV metadata; content-id identity (not physical bytes); dataset hash (sorted path order)	engine/storage.py + engine/evidence.py + engine/manifest.py	golden content ids; property content-id stability; verifier dataset mode
13.1–13.6 (Layer 7)	CP_V1 checkpoint; hash-chained journal; atomic transitions (INITIALIZED→…→COMPLETED, FAILED, RESUME_BLOCKED); checkpoint_hash non-self-ref; resume guards §13.5; source/version/chunkmap mismatch → RESUME_BLOCKED	engine/checkpoint.py + engine/journal.py	unit state transitions; golden G15 (corrupt state); adversarial repeated resume / already-processed; property checkpoint monotonicity + resume equivalence
14 (Layer 8)	Headless CLI: certify, init, ingest, resume, pause/stop, status, verify, manifest, build-bars, build-dataset, test, diff-legacy; exit codes 0..5	engine/cli.py + sniper-data launcher	CLI E2E on synthetic fixtures; exit-code assertions
15 (Layer 9)	Research dataset builder: canonical-bar-only inputs, deterministic primitives, leakage rule, time-ordered split, no raw-source reads	engine/dataset_builder.py	unit; property regeneration determinism
16 (Layer 10)	MS_V1 manifest: non-self-referential manifest_hash/dataset_hash; sorted file list	engine/manifest.py	unit; verifier dataset mode
17 (Layer 11)	Evidence hashing: sha256_physical + content_id everywhere; evidence_manifest_hash; no hash cycles; PYTHONHASHSEED=0	engine/evidence.py + engine/util.py	verifier dataset/run-complete modes
18 (Layer 12)	INDEPENDENT verifier (separate parsing/aggregation code; only version tables shared)	engine/verify/ (vparse, vaggregate, vinvariants, vcompare)	golden (both impl), property, mutation rejection
19 (Layer 13)	Golden corpus G01–G17 with expected outputs	tests/golden/	all G01–G17 PASS (producer + verifier)
20 (Layer 14)	Failure model: EXPECTED/BOUNDED/MATERIAL classes, fail-closed STOP, status_reason, failure log	engine/dispatcher.py + engine/checkpoint.py + engine/aggregate.py	adversarial; mutation; F-gate tests
21 (Layer 15)	Resource mgmt: bounded workers; progress.json; lock heartbeat 60 s / stale 300 s; default refuse; --force-release-lock human-only; graceful shutdown; no kill of live process	engine/lock.py + engine/dispatcher.py	unit lock transitions; adversarial stale-lock
22 (Layer 16)	RUN_COMPLETE.json only after COMPLETED + verified; RC_V1 fields; machine-verifiable re-check	engine/run_complete.py + engine/verify/	verifier run-complete mode; artifact read-back
23 (Layer 17)	Clean reprocessing protocol steps recorded; not executed	engine/cli.py (init/new-run guard)	— (no execution)
24 (Layers legacy)	Legacy comparison protocol + five-class taxonomy; legacy data read-only	engine/verify/vcompare.py + engine/cli.py diff-legacy	differential classification (evidence-level this session)
25 (Security)	rb-only source; output-root path validation; no network/secrets/eval; malformed sidecar hashes not raw text	engine/util.py + engine/config.py + engine/parse.py	unit path-validation; adversarial
26 (Versions)	Version constants; determinism pins; mismatch → RESUME_BLOCKED	engine/versions.py + engine/checkpoint.py	unit; golden G15
29 (Gates)	G-1..G-17 decisions encoded; G-14 NOT AUTHORIZED honored; no silent downgrade	engine/versions.py + engine/cli.py	CLI refusal paths

2. Deterministic choices established at implementation (documented, from spec)

Choice	Value	Traceability
rec_ord	0-based ordinal of canonical records within its chunk (accepted rows only)	spec 8.4
global_line_no	1-based raw line number in the whole source	spec 8.3
line splitter	first of \r\n (2B), \n (1B), \r (1B)	spec 8.1
price token grammar	[0-9]+(\.[0-9]{1,6})?; more than 6 frac digits → PRECISION; any other non-digit/sign char → PARSE; value ≤ 0 → NONPOSITIVE	spec 8.2 / G08
volume grammar (has_volume)	integer token; non-numeric or negative → MALFORMED_VOLUME; 0 accepted	spec 8.2
timestamp grammar	YYYY.MM.DD HH:MM:SS.mmm (ms=000..999) or YYYY.MM.DD HH:MM:SS (ms=0); field ranges validated; else PARSE; out of [2000-01-01, certify+7d] → RANGE	spec 8.1
journal line hash	SHA-256 over utf8 canonical JSON of the line text (no trailing NL); seq0 prev = SHA-256("")	spec 13.1
chunkmap_sha256 / checkpoint_hash / manifest_hash / evidence hash	SHA-256 over canonical JSON excluding the hash field itself (non-self-referential)	spec 13.2 / 16 / 17.2
bar canonical serialization	`bar_idx	period_id
dataset_hash / layer hash	SHA-256 over concat of `relpath	content_id\n` in sorted relpath order
duplicate tracker	in-memory set of (ts_ms,bid_u,ask_u,vol) per chunk, capped at 4,000,000 entries (chunk-bounded)	spec 8.2
malformed rate	workload-level: malformed/parsed > 0.001 OR abs cap 10,000 → FAILED	spec 8.3 / F02
first_src_line in carry	Spec §11.4 carry field list omits first_src_line, yet CBS_V1 §11.3 requires the first_src_line audit anchor and resume equivalence (§13.5/§18.3) requires a partial bar's first tick to survive a pause. Deterministic resolution: the checkpointed carry object additionally stores first_src_line (set when the carry is created). No external contract, canonical serialization, or schema changes; the bar schema field is unchanged.	spec 11.3/11.4/13.5
source_tz_offset_minutes	For synthetic/golden fixtures only: 0 (UTC), explicitly recorded in fixture configs. Production value is NOT established in this session (G-6; must be established at source initialization in the pilot phase)	spec 8.1 / G-6
output root	engine_output/ under the authoritative repo; every write path validated against the root	spec 12.2 / 25

3. Module map (files to be created)

engine/          __init__.py, versions.py, util.py, config.py, canonical.py,
                 parse.py, chunkmap.py, certify.py, storage.py, journal.py,
                 checkpoint.py, lock.py, aggregate.py, worker.py, dispatcher.py,
                 evidence.py, manifest.py, run_complete.py, dataset_builder.py,
                 cli.py
engine/verify/   __init__.py, vparse.py, vaggregate.py, vinvariants.py, vcompare.py
tests/           harness.py, unit_*.py, golden/ (cases + expected + runner),
                 property/, adversarial/, mutations/, legacy_diff/,
                 qualification_main.py
sniper-data.py   headless CLI launcher
docs/            this implementation map; P3_DATA_ENGINE_002_QUALIFICATION_REPORT.*
                 (produced at the end of qualification)

No real source is opened by any test. No production run, checkpoint, workload 46 preparation, or chunk 760 access occurs in this session.