# P3_DATA_ENGINE_002_IMPLEMENTATION_MAP
## Data Engine v1 — Requirements to Implementation Responsibility Map

Session: P3-DATA-ENGINE-002 (IMPLEMENTATION + QUALIFICATION ONLY)
Specification: docs/P3_DATA_ENGINE_V1_SPEC.md (DESIGN FROZEN, SHA-256
  84bf0f217ffba51197112a6bbacbcc297058e04b5ca47f0459028fe33e0321e5)
Governance: docs/P3_DATA_ENGINE_V1_GOV_DECISION_RECORD.md (G-1..G-13, G-15, G-16
  APPROVED; G-14 NOT AUTHORIZED; G-17 DEFERRED)

Every implementation responsibility below is traceable to a specification
section. Every critical behavior has at least one test (unit / golden /
property / adversarial / mutation / independent verifier).

---

## 1. Requirement → implementation responsibility

| Spec ref | Requirement | Implementation responsibility (file) | Tests |
|---|---|---|---|
| 7.2–7.5 (Layer 1) | Source certification: size, mtime_ns, full SHA-256, format facts; atomic cert; FULLY_VERIFIED vs VERIFIED_WITH_LIMITATION vs FAILED; no silent claim of full hash | engine/certify.py | unit ts/format sniff; golden G17 (unrecognized format); timeout path unit |
| 8.1 (Layer 2) | Deterministic CSV interpretation: UTF-8/BOM strip, comma delimiter, CRLF/LF/CR splitting, blank-line skip, header detection rule | engine/parse.py + engine/chunkmap.py (line detector) | unit; golden G01..G06; adversarial CRLF/LF/mixed |
| 8.1 | Timestamp parsing (dotted ms primary, no-ms secondary) + epoch ms int math + range rule | engine/parse.py + engine/util.py | unit; golden G07; adversarial malformed ts |
| 8.2 | Decimal price parsing to int (PRICE_SCALE=1_000_000), no floats; precision rule; nonpositive; bid/ask relation; volume rules | engine/util.py + engine/parse.py | unit; golden G08/G09; adversarial |
| 8.3 | Complete malformed taxonomy + sidecar (chunk_index, byte_start, global_line_no, class, raw_sha256, reason) + counters; never fabricate/skip silently | engine/parse.py (taxonomy + sidecar records) | unit; golden G07..G12; property "no silent malformed transformation" |
| 8.4 | CTS_V1 tick schema (all int64; no floats/nulls/strings) | engine/storage.py (parquet schema) | unit; verifier row-level compare |
| 8.5 | Canonical serialization `ts_ms|bid_u|ask_u|vol\n`; chunk_content_id over rec_ord order | engine/canonical.py | unit; golden ids; property determinism; mutation |
| 9.2–9.3 (Layer 3) | Line-safe chunk map; nominal 24 MiB / 512 MiB; inclusive terminator byte_end; workload grouping (partial final); chunkmap_sha256 | engine/chunkmap.py | unit; golden G05/G06; property no-gap/no-overlap; adversarial boundary cases |
| 10.1–10.4 (Layer 4) | Worker model: spawn, one chunk/worker, staging tmp+rename, digests before rename, bounded redispatch (retry_limit=2), journal truth, output-order independence | engine/worker.py + engine/dispatcher.py + engine/journal.py | golden G13 (worker reorder); property worker-schedule invariance; adversarial worker reorder/crash |
| 11.1–11.6 (Layer 5) | M1/M5/M15/M30/H1; UTC clock boundaries; mid_u2 OHLC; spread min/max/sum/avg(floor); carry per TF; is_final=false EOF partial; no empty-bar inflation; bar_idx continuous; (TF,period_id) uniqueness | engine/aggregate.py | unit bar-period; unit aggregation; golden (bar ids); property no-gap/no-empty-bar; partial-bar unit |
| 12.1–12.5 (Layer 6) | Parquet + zstd-3 + pinned pyarrow; layout/naming; schema KV metadata; content-id identity (not physical bytes); dataset hash (sorted path order) | engine/storage.py + engine/evidence.py + engine/manifest.py | golden content ids; property content-id stability; verifier dataset mode |
| 13.1–13.6 (Layer 7) | CP_V1 checkpoint; hash-chained journal; atomic transitions (INITIALIZED→…→COMPLETED, FAILED, RESUME_BLOCKED); checkpoint_hash non-self-ref; resume guards §13.5; source/version/chunkmap mismatch → RESUME_BLOCKED | engine/checkpoint.py + engine/journal.py | unit state transitions; golden G15 (corrupt state); adversarial repeated resume / already-processed; property checkpoint monotonicity + resume equivalence |
| 14 (Layer 8) | Headless CLI: certify, init, ingest, resume, pause/stop, status, verify, manifest, build-bars, build-dataset, test, diff-legacy; exit codes 0..5 | engine/cli.py + sniper-data launcher | CLI E2E on synthetic fixtures; exit-code assertions |
| 15 (Layer 9) | Research dataset builder: canonical-bar-only inputs, deterministic primitives, leakage rule, time-ordered split, no raw-source reads | engine/dataset_builder.py | unit; property regeneration determinism |
| 16 (Layer 10) | MS_V1 manifest: non-self-referential manifest_hash/dataset_hash; sorted file list | engine/manifest.py | unit; verifier dataset mode |
| 17 (Layer 11) | Evidence hashing: sha256_physical + content_id everywhere; evidence_manifest_hash; no hash cycles; PYTHONHASHSEED=0 | engine/evidence.py + engine/util.py | verifier dataset/run-complete modes |
| 18 (Layer 12) | INDEPENDENT verifier (separate parsing/aggregation code; only version tables shared) | engine/verify/ (vparse, vaggregate, vinvariants, vcompare) | golden (both impl), property, mutation rejection |
| 19 (Layer 13) | Golden corpus G01–G17 with expected outputs | tests/golden/ | all G01–G17 PASS (producer + verifier) |
| 20 (Layer 14) | Failure model: EXPECTED/BOUNDED/MATERIAL classes, fail-closed STOP, status_reason, failure log | engine/dispatcher.py + engine/checkpoint.py + engine/aggregate.py | adversarial; mutation; F-gate tests |
| 21 (Layer 15) | Resource mgmt: bounded workers; progress.json; lock heartbeat 60 s / stale 300 s; default refuse; --force-release-lock human-only; graceful shutdown; no kill of live process | engine/lock.py + engine/dispatcher.py | unit lock transitions; adversarial stale-lock |
| 22 (Layer 16) | RUN_COMPLETE.json only after COMPLETED + verified; RC_V1 fields; machine-verifiable re-check | engine/run_complete.py + engine/verify/ | verifier run-complete mode; artifact read-back |
| 23 (Layer 17) | Clean reprocessing protocol steps recorded; not executed | engine/cli.py (init/new-run guard) | — (no execution) |
| 24 (Layers legacy) | Legacy comparison protocol + five-class taxonomy; legacy data read-only | engine/verify/vcompare.py + engine/cli.py diff-legacy | differential classification (evidence-level this session) |
| 25 (Security) | rb-only source; output-root path validation; no network/secrets/eval; malformed sidecar hashes not raw text | engine/util.py + engine/config.py + engine/parse.py | unit path-validation; adversarial |
| 26 (Versions) | Version constants; determinism pins; mismatch → RESUME_BLOCKED | engine/versions.py + engine/checkpoint.py | unit; golden G15 |
| 29 (Gates) | G-1..G-17 decisions encoded; G-14 NOT AUTHORIZED honored; no silent downgrade | engine/versions.py + engine/cli.py | CLI refusal paths |

## 2. Deterministic choices established at implementation (documented, from spec)

| Choice | Value | Traceability |
|---|---|---|
| rec_ord | 0-based ordinal of canonical records within its chunk (accepted rows only) | spec 8.4 |
| global_line_no | 1-based raw line number in the whole source | spec 8.3 |
| line splitter | first of `\r\n` (2B), `\n` (1B), `\r` (1B) | spec 8.1 |
| price token grammar | `[0-9]+(\.[0-9]{1,6})?`; more than 6 frac digits → PRECISION; any other non-digit/sign char → PARSE; value ≤ 0 → NONPOSITIVE | spec 8.2 / G08 |
| volume grammar (has_volume) | integer token; non-numeric or negative → MALFORMED_VOLUME; 0 accepted | spec 8.2 |
| timestamp grammar | `YYYY.MM.DD HH:MM:SS.mmm` (ms=000..999) or `YYYY.MM.DD HH:MM:SS` (ms=0); field ranges validated; else PARSE; out of [2000-01-01, certify+7d] → RANGE | spec 8.1 |
| journal line hash | SHA-256 over utf8 canonical JSON of the line text (no trailing NL); seq0 prev = SHA-256("") | spec 13.1 |
| chunkmap_sha256 / checkpoint_hash / manifest_hash / evidence hash | SHA-256 over canonical JSON excluding the hash field itself (non-self-referential) | spec 13.2 / 16 / 17.2 |
| bar canonical serialization | `bar_idx|period_id|open_u2|high_u2|low_u2|close_u2|tick_count|vol_sum|spread_min_u|spread_max_u|spread_sum_u|is_final|first_src_line|last_src_line\n` with is_final = 1/0 | spec 12.4 |
| dataset_hash / layer hash | SHA-256 over concat of `relpath|content_id\n` in sorted relpath order | spec 12.4 / 16 / 17.2 |
| duplicate tracker | in-memory set of `(ts_ms,bid_u,ask_u,vol)` per chunk, capped at 4,000,000 entries (chunk-bounded) | spec 8.2 |
| malformed rate | workload-level: malformed/parsed > 0.001 OR abs cap 10,000 → FAILED | spec 8.3 / F02 |
| first_src_line in carry | Spec §11.4 carry field list omits `first_src_line`, yet CBS_V1 §11.3 requires the `first_src_line` audit anchor and resume equivalence (§13.5/§18.3) requires a partial bar's first tick to survive a pause. Deterministic resolution: the checkpointed carry object additionally stores `first_src_line` (set when the carry is created). No external contract, canonical serialization, or schema changes; the bar schema field is unchanged. | spec 11.3/11.4/13.5 |
| source_tz_offset_minutes | For synthetic/golden fixtures only: 0 (UTC), explicitly recorded in fixture configs. Production value is NOT established in this session (G-6; must be established at source initialization in the pilot phase) | spec 8.1 / G-6 |
| output root | `engine_output/` under the authoritative repo; every write path validated against the root | spec 12.2 / 25 |

## 3. Module map (files to be created)

```
engine/          __init__.py, versions.py, util.py, config.py, canonical.py,
                 parse.py, chunkmap.py, certify.py, storage.py, journal.py,
                 checkpoint.py, lock.py, aggregate.py, worker.py, dispatcher.py,
                 evidence.py, manifest.py, run_complete.py, dataset_builder.py,
                 cli.py
engine/verify/   __init__.py, vparse.py, vaggregate.py, vinvariants.py, vcompare.py
tests/           harness.py, unit_*.py, golden/ (cases + expected + runner),
                 property/, adversarial/, mutations/, legacy_diff/,
                 qualification_main.py
sniper-data.py   headless CLI launcher
docs/            this implementation map; P3_DATA_ENGINE_002_QUALIFICATION_REPORT.*
                 (produced at the end of qualification)
```

No real source is opened by any test. No production run, checkpoint, workload
46 preparation, or chunk 760 access occurs in this session.