P3-S25.1C — DYNAMIC CHUNK BENCHMARK

Date       : 2026-08-28
Phase      : P3-S25.1C
Session    : DYNAMIC CHUNK BENCHMARK & CHUNK-SIZE INVARIANCE
Status     : COMPLETE - DISPOSABLE / ISOLATED (no real ingestion)
Classification : A - DYNAMIC CHUNK SIZE VERIFIED

Purpose

Determine whether the current fixed 12 MiB chunk strategy can safely be increased, and prove that changing chunk boundaries does NOT change the semantic output. Computational / verification phase only.

No chunk 9, no chunk 10, no continuation of the 34.5 GB ingestion.
No modification of output/s251_real_checkpoint.json (run P3_S251_REAL_TICK_RUN_001).
No population replay, no ML, no Dukascopy, no production MQL5 change.

Method

A disposable engine (chunk_benchmark/chunk_size_benchmark.py) reuses the SAME production parser (s251_parse), streaming aggregator (s251_aggregate), timestamp semantics (UTC, tz 0), canonical serialisation, stable sorting, carry representation, and SHA-256 hashing. The ONLY thing varied is chunk_size. Every configuration starts from a deep copy of the SAME initial carry (carry isolation).

Benchmark range (representative, already-processed, chunks 6-8 contiguous):

byte [ 75,497,639 -> 113,246,416 )
bytes : 37,748,777  (~36.0 MiB)

This range captures normal tick density, M15/M30 boundary crossings, partial M15/M30 buckets (carry), same-timestamp groups, and two internal production chunk boundaries (88,080,562 and 100,663,500).

Results (machine-readable: output/chunk_size_benchmark.json)

chunk size	n_chunks	rows	M15	M30	elapsed s	rows/s	peak mem (bytes)
12 MiB	3	844772	3959	2112	32.17	26259	139,483,408
24 MiB	2	844772	3959	2112	32.34	26120	208,308,388
48 MiB	1	844772	3959	2112	31.65	26692	309,785,558
64 MiB	1	844772	3959	2112	31.69	26658	309,785,542

All configs produced the SAME output hash 0ddefa47...63a8, same M15 hash a0ea2c03...adcf, same M30 hash 4ab8da3c...552c, same final carry hash 5a70c058...47a1.
Malformed = 0 in all configs.
Reproducibility: 2 full passes, ALL byte-identical semantically (True).

Interpretation

Wall-clock is flat (~31-32 s). The engine is bound by per-line SHA-256 canonical serialisation, not by chunk orchestration. Increasing chunk size does not reduce wall-clock in this environment.
Memory scales with chunk size (139 MB -> 208 MB -> 310 MB). Larger buffers linearly raise peak RAM.
Chunk-size invariance is PROVEN (see P3_S25_1C_CHUNK_INVARIANCE.md): 12/24/48/64 MiB produce byte-identical semantic output to the single-stream reference.
12 and 24 MiB were genuinely multi-chunk (3 and 2 chunks). 48/64 MiB over the 36 MB range run as a single chunk (== reference), so their multi-chunk orchestration over an extended span was not directly measured here.

SAFE_MAX_CHUNK_SIZE recommendation

24 MiB is selected as SAFE_MAX_CHUNK_SIZE: it is the largest candidate that (a) was genuinely exercised across multiple real chunk boundaries (2 chunks), (b) is fully invariant and reproducible, (c) has repeatable success, and (d) has bounded memory (~208 MB). 12/48/64 MiB are all stable and invariant, but 48/64 MiB were only single-chunk tested in this window and provide no wall-clock advantage; 12 MiB is the conservative incumbent. A 2x reduction in chunk count is achieved at 24 MiB with no measured throughput penalty.

If a size > 24 MiB is ever desired, a follow-on longer-span benchmark (range > 128 MiB) must first be run to confirm multi-chunk orchestration, bounded memory, and invariance for that size.

Scope protection

Real checkpoint s251_real_checkpoint.json: NOT MODIFIED.
Run P3_S251_REAL_TICK_RUN_001 (last_completed_chunk, next_chunk_id, next_byte_offset): UNCHANGED.
Chunk 9 / 10+ : NOT PROCESSED.
Population replay, F1-F4, Candidate Setup, labels: NOT RUN.
ML (train/eval/ROC/calibrate/tune/features): NOT RUN.
Dukascopy MQL5\Files\XAUUSD\: NOT ACCESSED.
Production MQL5 / contracts / historical artifacts: UNCHANGED.
Tickstory CSV: READ ONLY.

4.2 KiB 원본 고유링크 Blame 히스토리