P3-DATA-ENGINE-002: Data Engine v1 implementation + qualification suite (QUALIFIED)
Implements the frozen P3_DATA_ENGINE_V1_SPEC (SHA 84bf0f217ffba51197112a6bbacbcc297058e04b5ca47f0459028fe33e0321e5).
Components: engine/ producer (certify, chunkmap, parse, canonical, worker,
dispatcher, aggregate, storage, journal, checkpoint, lock, evidence, manifest,
run_complete, dataset_builder, cli) + engine/verify independent verifier
(vparse, vaggregate, vinvariants, vcompare); headless CLI sniper-data; golden
corpus G01-G17; unit/property/adversarial/mutation/legacy-diff/CLI suites.
Qualification verdict: QUALIFIED (all 9 mandatory gates pass; independent
verifier accepted). Spec, governance record, and legacy checkpoint untouched.
G-14 NOT AUTHORIZED honored; no real-data processing, no pilot, no workload 46,
no chunk 760 access.
2026-09-07 14:15:25 +07:00 | | | """Independent invariant checks (spec 18.3).
|
| | |
|
| | | All checks are implemented against the specification text; they never call
|
| | | producer validation functions. Each returns (ok, details).
|
| | | """
|
| | |
|
| | | from ..util import sha256_bytes
|
| | |
|
| | |
|
| | | def check_monotonic_stream(records):
|
| | | """ts_ms must be non-decreasing across the merged stream."""
|
| | | prev = None
|
| | | bad = 0
|
| | | first_bad = None
|
| | | for rec in records:
|
| | | ts = rec[0]
|
| | | if prev is not None and ts < prev:
|
| | | bad += 1
|
| | | if first_bad is None:
|
| | | first_bad = (prev, ts)
|
| | | prev = ts
|
| | | return bad == 0, {"regressions": bad, "first": first_bad}
|
| | |
|
| | |
|
| | | def check_ohlc_validity(bar_rows):
|
| | | """low <= open,close <= high; counts >= 1; avg = sum // count."""
|
| | | bad = 0
|
| | | for row in bar_rows:
|
| | | (bar_idx, _pid, _st, _en, o, h, l, c, n, _v, _smn, _smx, ss,
|
| | | avg, _final, _f, _l) = row
|
| | | if not (l <= o <= h and l <= c <= h and n >= 1):
|
| | | bad += 1
|
| | | if avg != (ss // n):
|
| | | bad += 1
|
| | | return bad == 0, {"invalid": bad}
|
| | |
|
| | |
|
| | | def check_bar_uniqueness(bar_rows):
|
| | | """(TF period) uniqueness is implied per stream; bar_idx must be strictly
|
| | | increasing; period_id strictly increasing within one stream."""
|
| | | seen = {}
|
| | | bad = 0
|
| | | for row in bar_rows:
|
| | | bar_idx, period_id = row[0], row[1]
|
| | | if seen.get(period_id):
|
| | | bad += 1
|
| | | seen[period_id] = True
|
| | | return bad == 0, {"duplicates": bad}
|
| | |
|
| | |
|
| | | def check_chunk_counts_sum(per_chunk_counts, total):
|
| | | return sum(per_chunk_counts) == total, {"sum": sum(per_chunk_counts),
|
| | | "total": total}
|
| | |
|
| | |
|
| | | def check_carry_continuity(carry, first_tick):
|
| | | """Resume continuity: first ts must be >= carried last_ts; carried
|
| | | period must not be re-emitted."""
|
| | | if carry is None:
|
| | | return True, {}
|
| | | if first_tick[0] < carry["last_ts_ms"]:
|
| | | return False, {"first_ts": first_tick[0], "carried_last": carry["last_ts_ms"]}
|
| | | return True, {}
|
| | |
|
| | |
|
| | | def independent_serialize_ticks(records):
|
| | | """Independent canonical tick serialization (spec 8.5).
|
| | |
|
| | | ``records`` is an iterable of (ts_ms, bid_u, ask_u, vol) tuples in
|
| | | rec_ord order."""
|
| | | from .vaggregate import v_tick_serialize
|
| | | return b"".join(v_tick_serialize(r[0], r[1], r[2], r[3]) for r in records)
|
| | |
|
| | |
|
| | | def independent_ticks_content_id(records):
|
| | | return sha256_bytes(independent_serialize_ticks(records))
|
| | |
|
| | |
|
| | | def independent_bars_content_id(rows):
|
| | | from .vaggregate import v_bar_serialize
|
 P3-DATA-ENGINE-004: source-grammar correction + re-qualification (QUALIFIED) - spec 1.1.0 controlled amendment (Appendix C): G_TICKSTORY_MT5 six-col YYYYMMDD,HH:MM:SS,bid,ask,last,volume; compact timestamp; mandatory volume; last preserved via source-preservation sidecar (spec 8.6; CTS_V1 unchanged; G-4/G-5 intact); SHA effbaf2624cd46137c22a246fdabff4067052807b6b6224be464366bec4745d9 (machine-verified) - implementation: grammar registry, source_grammar config identity + fail-closed validation, compact timestamp parse, _parse_chunk_tickstory, preservation serialization + sidecars, certificate grammar_id + compact sniffing, init cert/config mismatch BLOCK, independent verifier v_parse_chunk_tickstory + preservation ids, evidence + verify_dataset preservation checks; ENGINE_VERSION 1.1.0, PARSER_V1.1 - CLI defect repair: status/resume on run-less dir exit 4 BLOCKED without traceback (P3-DE-003 NEW_ENGINE_DEFECT) + regression tests - qualification: unit, golden G01-G17, SG01-SG15 + e2e, property, adversarial, mutation 8/8, independent verifier dataset+run-complete ACCEPTED, legacy diff 0 unresolved legacy untouched, dataset-builder determinism, CLI E2E; VERDICT QUALIFIED - G-14 unchanged BLOCKED/NOT AUTHORIZED; no real-data ingestion; no workload 46; no chunk 760+; legacy checkpoint 759/18442355762 untouched
2026-09-07 17:34:57 +07:00 | | | return sha256_bytes(b"".join(v_bar_serialize(r) for r in rows))
|
| | |
|
| | |
|
| | | def independent_preservation_content_id(pairs):
|
| | | """Independent source-preservation content id (spec 8.6).
|
| | |
|
| | | ``pairs`` is an iterable of (src_line, last_u) integers in rec_ord
|
| | | order, extracted independently from the sidecar; the verifier re-derives
|
| | | the identity from its own row extraction with the fixed canonical
|
| | | serialization ``src_line|last_u\n`` (no floats)."""
|
| | | parts = []
|
| | | for src_line, last_u in pairs:
|
| | | parts.append(("%d|%d" % (src_line, last_u)).encode("ascii") + b"\n")
|
| | | return sha256_bytes(b"".join(parts))
|