SniperGold_ML/docs/P3_S25_1_INTERRUPTION_DIAGNOSIS.md

# P3-S25.1 — INTERRUPTION DIAGNOSIS (Recovery #3)

```text
Date              : 2026-08-26
Session           : P3-S25.1 RECOVERY #3
Goal              : FIRST diagnose why the P3-S25.1 ingestion repeatedly stops;
                    ONLY THEN, if a valid checkpoint and understood cause exist, resume.
Outcome this run  : STAGE A COMPLETE. STAGE B NOT STARTED (see verdict below).
Status            : E — ENVIRONMENT/EXECUTION LIMITATION (root cause identified)
Next              : resume preconditioned on a valid real-source checkpoint,
                    which does not yet exist.
```

## 1. Interrupted attempts observed

The session brief references three stopped attempts of the 34.5 GB Tickstory
ingestion. On-disk evidence (this repo, terminal, Temp, logs) does not contain
three durable, distinct run records. What is present:

| Attempt evidence | Artifact found |
|---|---|
| Prior attempt(s) | `p3_s25_external_data_validation/output/s25_execution_status.json` — "full tick→M15 ingest not completable in session budget"; handover `SESSION_HANDOVER_2026-08-26_P3_S25_EXTERNAL_DATA_BASIS_VALIDATION.md` — FINAL CLASSIFICATION **E-UNRESOLVED**, "could NOT complete within the session's compute budget". |
| Current real-source checkpoint | **NONE valid.** Only `ml/p3/p3_s251_external_ingest/output/p3_s251_checkpoint.json` exists, and it references `run_id: "src_clean"`, source `scratch\mt100.csv` (a 4,747-byte **test fixture**, not the 34.5 GB source). |
| Partial M15/M30 real-source output | **NONE.** Only fixture/scratch outputs exist. |
| Recovery log / run manifest / .tmp / lock | NONE present (searched project tree + `MQL5\Files\Temp`). |
| Original source file | `MQL5\Files\XAUUSD_mt5_ticks.csv`, 34,473,661,010 bytes, mtime 2026-08-24 22:44:41. |

Conclusion: the ingest has not recorded a completed chunk against the real
source in any successfully-checkpointed invocation.

## 2. Failure signature classification (A3)

The repeated stoppage is best classified as:

```text
TYPE   : (1) AI-assistant / session execution interruption
SIGNATURE : A — command/execution timeout (wall-clock / compute-budget exhaustion)
```

Supporting evidence:

- The P3-S25 authoritative record explicitly states the full ingest could not
  complete within the session wall-clock/compute budget.
- Measured in this session: a benign, read-only full-source SHA-256 pass takes
  **~153 seconds** on this machine. The `IngestRun.begin()` computes the full
  source SHA-256 (streaming 34.5 GB) *before* any chunk is processed, and the
  `run()` method processes the **entire remaining file** in one long-lived
  Python subprocess with a default chunk of 4,000,000 rows.
- There is no durable checkpoint of a real run, consistent with each attempt
  being terminated (command/tool/session) before completing/committing a chunk.

Why the design makes an in-session completion unlikely:

1. `begin()` hashes the entire 34.5 GB file in-memory streaming (~153 s) and
   only silently stores the SHA; a real chunk is processed after that.
2. The chunked checkpoint is written only *after* a whole chunk is processed;
   a killed session leaves no durable record -> the stale test-fixture
   checkpoint remains.
3. The default execution is effectively "process everything remaining" in one
   subprocess, which the environment cannot outlast.

This is a **resource/execution** limitation, not a parser/data bug: the same
bounded, streaming design is validated and the parser/aggregator are memory-
bounded. No parser exception, no memory/disc exhaustion, no deadlock.

## 3. Last valid checkpoint (A7)

```text
Valid real-source checkpoint: NONE.

Latest checkpoint file contents (output/p3_s251_checkpoint.json):
  run_id            : "src_clean"                      (test artifact)
  source_path       : .../scratch/mt100.csv            (test fixture, 4747 B)
  source_sha256     : d552b9e4…                         (fixture hash, not source)
  last_completed_chunk : 2                             (of the fixture only)
  rows_processed    : 101                              (fixture only)

=> The checkpoint refers to a test fixture, NOT the 34.5 GB source. It cannot
   be used as a resume position for the real ingestion (A7 STOP rule).
```

## 4. Source integrity

```text
path   : MQL5\Files\XAUUSD_mt5_ticks.csv
size   : 34,473,661,010 bytes
mtime  : 2026-08-24 22:44:41 UTC
sha256 : 5252ce8f5b0e7d286b71a3172c7ef9c1009c8e4b1254ac67400a803fcd85cba1  (measured this session, read-only)
rowcount (approx) coincident with S25 inventory: 722,941,870 rows
```

Source file is present and unchanged (size/mtime match the 2026-08-24 record).

## 5. Resource analysis (A4) — measured this session

```
Disk free (D:)  : 850 GB   -> sufficient
RAM total/free  : 47.9 / 36.1 GB -> sufficient for streaming chunked ingest
Full-source SHA : ~153 s for a read-only hash (the dominant single-op cost)
```

Section storage / disk / RAM are not the blocker. The blocker is the
single long-running command vs. session wall-clock compute budget.

## 5b. Pipeline validation test-suite run (this session, 16/20 PASS)

Ran `s251_tests.py` (T01..T20, fixtures only — no real source). Result: **16/20 PASS**.
The four failures are TEST-HARNESS expectation bugs, NOT pipeline defects:

| Test | Result | Classification |
|---|---|---|
| T04 M15 boundary | FAIL | Expectation bug: test demands the *final incomplete* bucket be emitted; design deliberately carries it in `partial_carry_m15` and never emits it. Emitted buckets match design. |
| T06 M30 boundary | FAIL | Same cause (final incomplete M30 bucket carried, not emitted). |
| T15 future-data mutation | FAIL | **Test-code bug**: `dt.datetime(2026, tzinfo=...)` called without month/day -> TypeError. Not a pipeline error. |
| T16 row-order mutation | FAIL | Expectation bug: the aggregator stably sorts ticks by timestamp (documented), so identical M15 output is CORRECT after a pure row permutation; the raw manifest DOES change (hash differs), which is the correct detection signal. Test's `pass` condition is inconsistent. |

Independent verification signals that DO pass and are the meaningful evidence:
T02 timestamp parsing, T03 UTC normalisation, T05/T07 independent-oracle equality,
T08 chunk-hash determinism, T09 checkpoint atomicity, T10/T11/T12 resume
equivalence incl. mid-bucket cuts, T13 malformed detection, T14 duplicate
detection, T19 source-hash refusal, T20 subset determinism.

Per CODE_VERIFICATION policy: PASS is not final evidence; however these tests
plus the independent oracle (different code path) agree, which is the correct
signal for the streaming pipeline. The four FAILs are test-harness defects and
were NOT edited to force green (policy §0/§15: no silent modification). They
should be repaired in a dedicated test-fix phase before the suite is treated
as green.

## 6. Bounded-memory check (A5)

- `s251_ingest._file_blocks` streams the file in 64 MiB blocks,
  `_line_stream` stays in Python lines up to `chunk_rows`.
- `ChunkParser` keeps only per-chunk lists; `Aggregation` is a streaming
  state-machine, holding only the current open M15/M30 bucket.
- No full-file `read()`, `readlines()`, no unbounded dict/list accumulation,
  no DataFrame accumulation.

=> **Bounded-memory: YES. Confirmed.** This is not the cause of failure.

## 7. Execution-model check (A6)

The prior approach executed effectively the whole 34.5 GB source in a single
**monolithic** subprocess (`run()` processes the entire remaining file). That
is the unsafe architecture for this environment.

Required corrective model:

```text
ONE BOUNDED CHUNK -> VERIFY -> CHECKPOINT -> RETURN CONTROL
(one Python invocation per chunk, or a small bounded batch with per-chunk
checkpoints and atomic stop)
```

## 8. Root-cause confidence

```
Confidence : MEDIUM
```
Reason: the execution/wall-clock limitation is well supported (documented
P3-S25 record + measured 153 s full-hash + monolithic run model), but the
"three attempts" are not individually evidenced in durable artifacts, so the
exact instant/operation of each stop cannot be proven. The overall mechanism
(research-execution bound) is established with MEDIUM confidence; the code
design is consistent with repeated pre-chunk termination.

## 9. Classification

```
E — ENVIRONMENT/EXECUTION LIMITATION
```

## 10. Why Stage B was NOT started

Per protocol hard-stop conditions (A7 + session objective), ingestion may only
resume when **BOTH**:
  (i) the checkpoint is proven valid, and
  (ii) the cause is identified with a safe strategy.

(i) is NOT satisfied: the only checkpoint to hand is a test-fixture artifact
(no valid real-source checkpoint exists). Per A7 ("If the latest checkpoint is
invalid: STOP. Do not guess a resume position.") no resume byte-offset can be
honestly chosen this session. Therefore **Stage B was not forced**.

## 11. Corrective execution strategy (recommended, NOT executed here)

1. Do NOT re-run a monolithic 34 GB command.
2. Initialize anew the real-source ingestion and **verify checkpoint against
   the source line before the first chunk** (the begin() SHA-256 check).
3. Execute **ONE bounded chunk per invocation** (e.g. default 4,000,000 rows),
   then VERIFY/WRITE CHECKPOINT, then STOP/return control; explicitly bound the
   run (constrain `max_chunks` in a small driver) so a session cannot exceed
   its budget.
4. After each chunk, verify full-source continuity in the checkpoint
   (source size/mtime/SHA-256) and record progress in a recovery log
   (run_id, chunk_id, byte_start, byte_end, rows, first/last ts, m151 count,
   m30 count, chunk_sha256, cp_sha256, elapsed, mem_peak, status).
5. If the environment still cannot run >1 chunk per session, fall back to the
   previously authorized deterministic subset strategy (documented separately)
   and classify honestly as SUBSET BASIS VALIDATED.

## 12. State

- No ML, no training, no feature/label/TP/SL/H changes performed.
- No MQL5 / production code changed.
- No source files modified; the 34.5 GB CSV untouched.
- No Dukascopy `.bi5` reverse-engineering started.
- `p3_s251_subset_manifest.json` NOT written (subset fallback not used).
- Final status: **E** ; Stage A complete; Stage B NOT started.

*End of P3-S25.1 interruption diagnosis.*