SniperGold_ML/docs/P3_S25_1C_DYNAMIC_CHUNK_BENCHMARK.md

# P3-S25.1C — DYNAMIC CHUNK BENCHMARK

```text
Date       : 2026-08-28
Phase      : P3-S25.1C
Session    : DYNAMIC CHUNK BENCHMARK & CHUNK-SIZE INVARIANCE
Status     : COMPLETE - DISPOSABLE / ISOLATED (no real ingestion)
Classification : A - DYNAMIC CHUNK SIZE VERIFIED
```

## Purpose

Determine whether the current fixed 12 MiB chunk strategy can safely be
increased, and prove that changing chunk boundaries does NOT change the
semantic output. Computational / verification phase only.

- No chunk 9, no chunk 10, no continuation of the 34.5 GB ingestion.
- No modification of `output/s251_real_checkpoint.json` (run P3_S251_REAL_TICK_RUN_001).
- No population replay, no ML, no Dukascopy, no production MQL5 change.

## Method

A disposable engine (`chunk_benchmark/chunk_size_benchmark.py`) reuses the
SAME production parser (`s251_parse`), streaming aggregator (`s251_aggregate`),
timestamp semantics (UTC, tz 0), canonical serialisation, stable sorting,
carry representation, and SHA-256 hashing. The ONLY thing varied is
`chunk_size`. Every configuration starts from a deep copy of the SAME initial
carry (carry isolation).

Benchmark range (representative, already-processed, chunks 6-8 contiguous):

```text
byte [ 75,497,639 -> 113,246,416 )
bytes : 37,748,777  (~36.0 MiB)
```

This range captures normal tick density, M15/M30 boundary crossings, partial
M15/M30 buckets (carry), same-timestamp groups, and two internal production
chunk boundaries (88,080,562 and 100,663,500).

## Results (machine-readable: output/chunk_size_benchmark.json)

| chunk size | n_chunks | rows | M15 | M30 | elapsed s | rows/s | peak mem (bytes) |
|-----------:|---------:|-----:|----:|----:|----------:|-------:|-----------------:|
| 12 MiB     | 3        | 844772 | 3959 | 2112 | 32.17 | 26259 | 139,483,408 |
| 24 MiB     | 2        | 844772 | 3959 | 2112 | 32.34 | 26120 | 208,308,388 |
| 48 MiB     | 1        | 844772 | 3959 | 2112 | 31.65 | 26692 | 309,785,558 |
| 64 MiB     | 1        | 844772 | 3959 | 2112 | 31.69 | 26658 | 309,785,542 |

- All configs produced the SAME output hash `0ddefa47...63a8`, same M15 hash
  `a0ea2c03...adcf`, same M30 hash `4ab8da3c...552c`, same final carry hash
  `5a70c058...47a1`.
- Malformed = 0 in all configs.
- Reproducibility: 2 full passes, ALL byte-identical semantically (True).

## Interpretation

1. **Wall-clock is flat (~31-32 s).** The engine is bound by per-line
   SHA-256 canonical serialisation, not by chunk orchestration. Increasing
   chunk size does not reduce wall-clock in this environment.
2. **Memory scales with chunk size** (139 MB -> 208 MB -> 310 MB). Larger
   buffers linearly raise peak RAM.
3. **Chunk-size invariance is PROVEN** (see P3_S25_1C_CHUNK_INVARIANCE.md):
   12/24/48/64 MiB produce byte-identical semantic output to the
   single-stream reference.
4. 12 and 24 MiB were genuinely multi-chunk (3 and 2 chunks). 48/64 MiB over
   the 36 MB range run as a single chunk (== reference), so their multi-chunk
   orchestration over an extended span was not directly measured here.

## SAFE_MAX_CHUNK_SIZE recommendation

**24 MiB** is selected as SAFE_MAX_CHUNK_SIZE: it is the largest candidate that
(a) was genuinely exercised across multiple real chunk boundaries (2 chunks),
(b) is fully invariant and reproducible, (c) has repeatable success, and
(d) has bounded memory (~208 MB). 12/48/64 MiB are all stable and invariant,
but 48/64 MiB were only single-chunk tested in this window and provide no
wall-clock advantage; 12 MiB is the conservative incumbent. A 2x reduction in
chunk count is achieved at 24 MiB with no measured throughput penalty.

> If a size > 24 MiB is ever desired, a follow-on longer-span benchmark
> (range > 128 MiB) must first be run to confirm multi-chunk orchestration,
> bounded memory, and invariance for that size.

## Scope protection

- Real checkpoint `s251_real_checkpoint.json`: NOT MODIFIED.
- Run P3_S251_REAL_TICK_RUN_001 (`last_completed_chunk`, `next_chunk_id`,
  `next_byte_offset`): UNCHANGED.
- Chunk 9 / 10+ : NOT PROCESSED.
- Population replay, F1-F4, Candidate Setup, labels: NOT RUN.
- ML (train/eval/ROC/calibrate/tune/features): NOT RUN.
- Dukascopy `MQL5\Files\XAUUSD\`: NOT ACCESSED.
- Production MQL5 / contracts / historical artifacts: UNCHANGED.
- Tickstory CSV: READ ONLY.