P3-DATA-ENGINE-002: Data Engine v1 implementation + qualification suite (QUALIFIED)
Implements the frozen P3_DATA_ENGINE_V1_SPEC (SHA 84bf0f217ffba51197112a6bbacbcc297058e04b5ca47f0459028fe33e0321e5).
Components: engine/ producer (certify, chunkmap, parse, canonical, worker,
dispatcher, aggregate, storage, journal, checkpoint, lock, evidence, manifest,
run_complete, dataset_builder, cli) + engine/verify independent verifier
(vparse, vaggregate, vinvariants, vcompare); headless CLI sniper-data; golden
corpus G01-G17; unit/property/adversarial/mutation/legacy-diff/CLI suites.
Qualification verdict: QUALIFIED (all 9 mandatory gates pass; independent
verifier accepted). Spec, governance record, and legacy checkpoint untouched.
G-14 NOT AUTHORIZED honored; no real-data processing, no pilot, no workload 46,
no chunk 760 access.
2026-09-07 14:15:25 +07:00 | | | """Canonical version constants for SniperGold Data Engine v1.
|
| | |
|
| | | Every constant in this module is binding for v1 (spec section 26). No
|
| | | constant here may change without a DATASET/schema version bump governed by
|
| | | the human decision gates (G-4/G-5). Producer and independent verifier share
|
| | | ONLY this table (spec 18.1).
|
| | | """
|
| | |
|
| | | # --- Engine / algorithm identity -------------------------------------------
|
 P3-DATA-ENGINE-004: source-grammar correction + re-qualification (QUALIFIED) - spec 1.1.0 controlled amendment (Appendix C): G_TICKSTORY_MT5 six-col YYYYMMDD,HH:MM:SS,bid,ask,last,volume; compact timestamp; mandatory volume; last preserved via source-preservation sidecar (spec 8.6; CTS_V1 unchanged; G-4/G-5 intact); SHA effbaf2624cd46137c22a246fdabff4067052807b6b6224be464366bec4745d9 (machine-verified) - implementation: grammar registry, source_grammar config identity + fail-closed validation, compact timestamp parse, _parse_chunk_tickstory, preservation serialization + sidecars, certificate grammar_id + compact sniffing, init cert/config mismatch BLOCK, independent verifier v_parse_chunk_tickstory + preservation ids, evidence + verify_dataset preservation checks; ENGINE_VERSION 1.1.0, PARSER_V1.1 - CLI defect repair: status/resume on run-less dir exit 4 BLOCKED without traceback (P3-DE-003 NEW_ENGINE_DEFECT) + regression tests - qualification: unit, golden G01-G17, SG01-SG15 + e2e, property, adversarial, mutation 8/8, independent verifier dataset+run-complete ACCEPTED, legacy diff 0 unresolved legacy untouched, dataset-builder determinism, CLI E2E; VERDICT QUALIFIED - G-14 unchanged BLOCKED/NOT AUTHORIZED; no real-data ingestion; no workload 46; no chunk 760+; legacy checkpoint 759/18442355762 untouched
2026-09-07 17:34:57 +07:00 | | | ENGINE_VERSION = "1.1.0" # any Engine behavior change (semver) -- amended P3-DE-004
|
| | | PARSER_VERSION = "PARSER_V1.1" # any CSV/timestamp/price parsing rule change -- amended P3-DE-004
|
P3-DATA-ENGINE-002: Data Engine v1 implementation + qualification suite (QUALIFIED)
Implements the frozen P3_DATA_ENGINE_V1_SPEC (SHA 84bf0f217ffba51197112a6bbacbcc297058e04b5ca47f0459028fe33e0321e5).
Components: engine/ producer (certify, chunkmap, parse, canonical, worker,
dispatcher, aggregate, storage, journal, checkpoint, lock, evidence, manifest,
run_complete, dataset_builder, cli) + engine/verify independent verifier
(vparse, vaggregate, vinvariants, vcompare); headless CLI sniper-data; golden
corpus G01-G17; unit/property/adversarial/mutation/legacy-diff/CLI suites.
Qualification verdict: QUALIFIED (all 9 mandatory gates pass; independent
verifier accepted). Spec, governance record, and legacy checkpoint untouched.
G-14 NOT AUTHORIZED honored; no real-data processing, no pilot, no workload 46,
no chunk 760 access.
2026-09-07 14:15:25 +07:00 | | | ALGORITHM_VERSION = "ALG_V1" # serialization / chunking / aggregation semantic change
|
| | | DATASET_VERSION = "DS_V1.0.0" # human decision gate only (G-5)
|
| | |
|
P3-DATA-ENGINE-004: source-grammar correction + re-qualification (QUALIFIED) - spec 1.1.0 controlled amendment (Appendix C): G_TICKSTORY_MT5 six-col YYYYMMDD,HH:MM:SS,bid,ask,last,volume; compact timestamp; mandatory volume; last preserved via source-preservation sidecar (spec 8.6; CTS_V1 unchanged; G-4/G-5 intact); SHA effbaf2624cd46137c22a246fdabff4067052807b6b6224be464366bec4745d9 (machine-verified) - implementation: grammar registry, source_grammar config identity + fail-closed validation, compact timestamp parse, _parse_chunk_tickstory, preservation serialization + sidecars, certificate grammar_id + compact sniffing, init cert/config mismatch BLOCK, independent verifier v_parse_chunk_tickstory + preservation ids, evidence + verify_dataset preservation checks; ENGINE_VERSION 1.1.0, PARSER_V1.1 - CLI defect repair: status/resume on run-less dir exit 4 BLOCKED without traceback (P3-DE-003 NEW_ENGINE_DEFECT) + regression tests - qualification: unit, golden G01-G17, SG01-SG15 + e2e, property, adversarial, mutation 8/8, independent verifier dataset+run-complete ACCEPTED, legacy diff 0 unresolved legacy untouched, dataset-builder determinism, CLI E2E; VERDICT QUALIFIED - G-14 unchanged BLOCKED/NOT AUTHORIZED; no real-data ingestion; no workload 46; no chunk 760+; legacy checkpoint 759/18442355762 untouched
2026-09-07 17:34:57 +07:00 | | | # --- Source-grammar registry (P3-DATA-ENGINE-004, spec 8.1 amendment) --------
|
| | | # Two named, fully-specified, versioned grammar presets. Selection is NEVER
|
| | | # silent or per-row: the configured grammar must equal the certificate's
|
| | | # grammar_id at init, else the run is blocked (certificate/config mismatch).
|
| | | GRAMMAR_TICKSTORY_MT5 = "tickstory_mt5" # 6-col YYYYMMDD,HH:MM:SS,bid,ask,last,volume
|
| | | GRAMMAR_DOTTED = "dotted_v1" # YYYY.MM.DD HH:MM:SS[.mmm], 3/4 columns
|
| | | GRAMMAR_ID_TICKSTORY = "tickstory_mt5_v1"
|
| | | GRAMMAR_ID_DOTTED = "dotted_v1"
|
| | | SUPPORTED_GRAMMARS = (GRAMMAR_TICKSTORY_MT5, GRAMMAR_DOTTED)
|
| | |
|
P3-DATA-ENGINE-002: Data Engine v1 implementation + qualification suite (QUALIFIED)
Implements the frozen P3_DATA_ENGINE_V1_SPEC (SHA 84bf0f217ffba51197112a6bbacbcc297058e04b5ca47f0459028fe33e0321e5).
Components: engine/ producer (certify, chunkmap, parse, canonical, worker,
dispatcher, aggregate, storage, journal, checkpoint, lock, evidence, manifest,
run_complete, dataset_builder, cli) + engine/verify independent verifier
(vparse, vaggregate, vinvariants, vcompare); headless CLI sniper-data; golden
corpus G01-G17; unit/property/adversarial/mutation/legacy-diff/CLI suites.
Qualification verdict: QUALIFIED (all 9 mandatory gates pass; independent
verifier accepted). Spec, governance record, and legacy checkpoint untouched.
G-14 NOT AUTHORIZED honored; no real-data processing, no pilot, no workload 46,
no chunk 760 access.
2026-09-07 14:15:25 +07:00 | | | # --- Schema identities ------------------------------------------------------
|
| | | SCHEMA_TICKS = "CTS_V1"
|
| | | SCHEMA_BARS = "CBS_V1"
|
| | | CERT_SCHEMA_VERSION = "CERT_V1"
|
| | | MANIFEST_SCHEMA_VERSION = "MS_V1"
|
| | | CHECKPOINT_SCHEMA_VERSION = "CP_V1"
|
| | | RUN_COMPLETE_SCHEMA_VERSION = "RC_V1"
|
| | |
|
| | | # --- Scale constants (frozen) ------------------------------------------------
|
| | | PRICE_SCALE = 1_000_000 # micro-USD per USD unit (CTS_V1)
|
| | | PRICE_SCALE_BAR = 2_000_000 # 2 x PRICE_SCALE (mid-based OHLC, CBS_V1)
|
| | |
|
| | | # --- Chunking / workload (frozen; G-7, G-8) ----------------------------------
|
| | | CHUNK_BYTES_NOMINAL = 25_165_824 # 24 MiB
|
| | | WORKLOAD_BYTES_NOMINAL = 536_870_912 # 512 MiB
|
| | |
|
| | | # --- Timeframes (frozen; spec 11.1) -------------------------------------------
|
| | | # Only M1, M5, M15, M30, H1 are permitted in v1.
|
| | | TIMEFRAMES = {
|
| | | "M1": 60_000,
|
| | | "M5": 300_000,
|
| | | "M15": 900_000,
|
| | | "M30": 1_800_000,
|
| | | "H1": 3_600_000,
|
| | | }
|
| | | DEFAULT_TIMEFRAMES = ["M1", "M5", "M15", "M30", "H1"]
|
| | |
|
| | | # --- Lock policy (G-11) --------------------------------------------------------
|
| | | LOCK_HEARTBEAT_SEC = 60
|
| | | LOCK_STALE_SEC = 300
|
| | |
|
| | | # --- Malformed policy (spec 8.3, F02) ------------------------------------------
|
| | | MALFORMED_RATE_LIMIT = 0.001
|
| | | MALFORMED_ABS_CAP = 10_000
|
| | | DUPLICATE_TRACKER_CAP = 4_000_000 # bounded in-memory duplicate set per chunk
|
| | |
|
| | | # --- Timestamp validity window (spec 8.1) -------------------------------------
|
| | | TS_MIN_EPOCH_MS = 946684800000 # 2000-01-01T00:00:00Z in epoch ms
|
| | | TS_RANGE_FUTURE_DAYS = 7
|
| | |
|
| | | # --- Worker policy (G-2) --------------------------------------------------------
|
| | | WORKERS_REQUESTED_DEFAULT = 24
|
| | |
|
| | | # --- Determinism pins (spec 26) -------------------------------------------------
|
| | | PYTHONHASHSEED_PIN = "0"
|
| | | PYARROW_VERSION_PIN = "25.0.0" # must match the installed pyarrow
|
| | | PYTHON_MIN_VERSION = (3, 11)
|
| | |
|
| | |
|
| | | def tf_period_ms(timeframe):
|
| | | """Return the period in ms for a v1 timeframe symbol."""
|
| | | try:
|
| | | return TIMEFRAMES[timeframe]
|
| | | except KeyError:
|
| | | raise ValueError("unsupported timeframe %r (v1 allows %s)"
|
| | | % (timeframe, sorted(TIMEFRAMES)))
|