forked from chiki2bum2/SniperGold_ML
17 KiB
17 KiB
P3-S25.1Q — WORKER-SCALING OPTIMIZATION EXPERIMENT — FINAL REPORT
DOCUMENT ROLE : REPORT (derived summary; generated from machine
evidence under
ml/p3/p3_s251_external_ingest/workload_benchmark/
worker_scaling_q/output/*.json)
AUTHORITATIVE SOURCE: P3-S25.1Q machine evidence + canonical real checkpoint
output/s251_real_checkpoint.json (READ-ONLY)
PHASE : P3-S25.1Q - WORKER-SCALING OPTIMIZATION EXPERIMENT
CLASSIFICATION : A - OPTIMIZATION CANDIDATE IDENTIFIED
GATES PASS : correctness 14/14, boundary, state/carry,
interruption/resume, mutation 10/10,
reproducibility, checkpoint protection
STARTING GIT SHA : 1e92c291e16de78c2fab562ae7aeba49b18063c2 (branch main,
== origin/main, CLEAN, stash EMPTY, no untracked)
1. Machine CPU topology (programmatically read, never hard-coded)
CPU name : Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz (2 sockets)
sockets : 2
cores per socket : [12, 12]
physical cores : 24 (sum of sockets)
logical processors : 24 (os.cpu_count() == 24; no hyper-threading)
python : 3.14.6 (AMD64)
2. Total RAM (programmatically read)
ram_total_bytes = 51,468,738,560 (~47.93 GiB)
ram_available at gate start = 35,185,541,120 (~32.8 GiB)
ram_available at gate close = 35,148,988,416 (~32.7 GiB)
min ram available during 24-worker run (sampled) = 25,218,666,496 (~23.5 GiB)
3. Baseline worker count
BASELINE_WORKERS = 8 (already-validated configuration: 8 workers,
24 MiB atomic chunks, 512 MiB workload)
4. Candidate worker counts
8 (baseline), 12, 16, 20, 24 (exactly the required set; no intermediate
counts tested, no expansion beyond 24)
5. Disposable dataset identity
path : ml/p3/p3_s251_external_ingest/scratch/
p3_s251e_primary_fixture_24MiB.csv (gitignored scratch)
size : 2,149,580,843 bytes
schema : Tickstory XAUUSD tick schema - YYYY.MM.DD,HH:MM:SS,bid,
ask,last,volume (CFG.NCOL=6, UTC, epoch seconds) =
schema-identical to the real Tickstory source
generation : deterministic; seed 20260830; rows_written 45,735,763
workload : 512 MiB = 536,870,912 bytes, byte range [0, 536870912)
atomic semantics : 24 MiB = 25,165,824 bytes (SAFE_MAX_CHUNK_SIZE)
actual line-safe atomic chunks : 22 (each 24 MiB read extends
to the next LF; last boundary [528483217, 536870912))
immutability : ONE dataset used for EVERY candidate and BOTH runs; a
different dataset was never generated per worker count.
6. Dataset SHA
fixture sha256 : 2a2691159f1a250b537c82a623539190a59ffd8ea4089ec54245a8f8a7800be5
(matches P3-S25.1E/F recorded; verified preflight)
raw range sha256 [0,536870912) : 00c0421ac60d4851dbfe79a10cbb0f2fcdc51d978414934f0d761f40514de88f
7. Reference result (canonical single-stream, ONE per experiment)
rows_processed = 11,422,782
m15_count = 120,112
m30_count = 108,000
first_timestamp = 1,483,403,400
last_timestamp = 1,697,238,205
malformed = {"column_count": 2, "timestamp_malformed": 1,
"timestamp_invalid": 1, "non_numeric_price": 0,
"invalid_volume": 0, "bid_ask_relationship": 0,
"non_positive_price": 0, "non_monotonic_timestamp": 0}
m15_sha256 = 94714104fc23eef83e379a138ef70775c43865419c6dc8afce05b94c9388a95a
m30_sha256 = 238776d863734092c65023817f0ce53acf0d0b2e32510cd63d89806843e1a7e7
output_sha256 = cfa8c084f9bae714a4a15cb1da27e2442bd72bd11aee3f3604304a203e02ed4a
final_state_hash = 757a4d63a0f13c2757281402d5beea4278c982c2e29e36375faa30b9565994ff
output carry M15 = {"ts":1697238000,"open":1208.583,"high":1209.974,
"low":1208.583,"close":1209.351,"ticks":21,
"first_ts":1697238000,"last_ts":1697238205}
output carry M30 = null
matches P3-S25.1E recorded (known-answer) : True
reference elapsed (single stream) : 366.765 s
reference peak memory : 141,534,283 B (~135.0 MiB)
Canonical atomic chained reference (22 x 24 MiB, lean verifier):
sig == single-stream : True | boundary sequence == authoritative : True
Canonical full-digest atomic reference (22 x 24 MiB, ChunkParser digests):
all status PASS : True | == single-stream : True
raw_chunk_sha256[22] + parsed_chunk_sha256[22] recorded
parsed_chain_sha256 = ded07b41892d5e3c0e1277865c43a7d1dba0ee32e8c7f803187a8348c4ea9073
8. Correctness result per candidate (14-point contract)
Every candidate == canonical single-stream reference (EXACT, no approximation)
checks: rows | malformed counts | first ts | last ts | M15 output | M30
output | raw hash | parsed hash | row hash | output hash | final state hash
| boundary sequence | byte coverage | carry/state chain
workers=8 14/14 PASS (raw hash implied by byte coverage; parsed hash by
boundary identity + canonical full-digest equality)
workers=12 14/14 PASS
workers=16 14/14 PASS
workers=20 14/14 PASS
workers=24 14/14 PASS
9. Boundary result per candidate
no gaps / no overlaps : PASS (all candidates)
source-byte ordering : PASS (worker_id ascending == source byte order)
line-safe boundaries : PASS (authoritative 22-boundary sequence used)
boundary sequence == authoritative : PASS (all)
worker partition coverage == workload coverage [0,536870912) : PASS (all)
10. State / carry result per candidate
input_state(N) == predecessor output_state : PASS (prefix merged carry ==
canonical chunk carry at every worker boundary)
output_state(N) == input_state(N+1) : PASS (canonical 22-chunk chain
on ACTUAL byte_end; candidate boundary entry hashes == canonical chunk
hashes at every boundary)
final worker result state == reference : PASS (final_state_hash
757a4d63... for every candidate)
worker boundaries verified : 8->7 | 12->11 | 16->15 | 20->19 | 24->21
11. Interruption / resume result per candidate
(a) CANONICAL genuine resume (P3-S25.1D-R corrected; tail re-executed) at
first worker-boundary chunk (stop_after_chunk=3):
per_chunk_equal True | boundary_sequence_identical True |
aggregate_equal True | continuous_output_hash == resumed_output_hash ==
cfa8c084...e4a | continuous_final_state == resumed_final_state ==
757a4d63... -> match True
(b) PARALLEL resume at EVERY worker boundary: prefix merged carry ==
canonical chunk carry; resume byte start == actual line-safe byte end;
shuffled completion-order merge == continuous merge == reference:
workers=8 True | 12 True | 16 True | 20 True | 24 True
(c) GENUINE worker-level interruption + resume (tail re-executed):
8 workers resume at byte 75,497,593 -> equals continuous True
16 workers resume at byte 50,331,725 -> equals continuous True
continuous == interrupted + resumed : PASS (all)
The real production checkpoint was NOT interrupted (disposable only).
12. Mutation result per candidate
10/10 detected for EVERY candidate (skipped-atomic-chunk,
duplicated-atomic-chunk, wrong-byte-offset, wrong-worker-ordering,
wrong-input-carry, post-chunk-carry-as-input, output-row-deletion,
output-row-duplication, semantic-output-mutation,
boundary-sequence-mutation)
The 5 minimum required brief classes (reordered results, wrong boundary,
wrong input carry, omitted range, duplicated range) are all DETECTED.
13. run_1 vs run_2 reproducibility
semantic equality run1 == run2 (timing equality NOT required):
8 PASS | 12 PASS | 16 PASS | 20 PASS | 24 PASS
output_sha run1 == output_sha run2 == cfa8c084...e4a for all candidates
14. Wall-clock per candidate (process span, final coherent pass)
workers=8 span1 = 48.86 s | span2 = 48.15 s
workers=12 span1 = 33.73 s | span2 = 32.81 s
workers=16 span1 = 33.58 s | span2 = 33.02 s
workers=20 span1 = 33.65 s | span2 = 33.59 s
workers=24 span1 = 19.54 s | span2 = 19.25 s
total elapsed (dispatch wall): 8->53.47 s | 12->36.32 s | 16->36.33 s |
20->37.00 s | 24->21.49 s (run1)
15. Rows/sec (span basis, run1)
8 -> 233,790 | 12 -> 338,666 | 16 -> 340,177 | 20 -> 339,496 | 24 -> 584,490
rows = 11,422,782 for every candidate
16. Bytes/sec (span basis, run1)
8 -> 10,988,134 | 12 -> 15,917,298 | 16 -> 15,988,325 | 20 -> 15,956,312
| 24 -> 27,471,018
bytes = 536,870,912 for every candidate
17. Peak memory per worker (tracemalloc; bounded by 24 MiB atomic chunk)
max per-worker peak: 241,050,426 B (~229.9 MiB) for EVERY candidate
(worker memory is bounded by the 24 MiB atomic chunk, flat across N)
18. Total Python memory
total worker peak (sum of per-worker peaks):
8->1,839.0 MiB | 12->2,604.5 MiB | 16->3,524.0 MiB | 20->4,443.6 MiB
| 24->4,903.3 MiB
total worker RSS (measured, psapi):
8->945.5 MiB | 12->1,303.2 MiB | 16->1,697.6 MiB | 20->2,088.7 MiB
| 24->2,282.6 MiB
coordinator RSS during run (measured):
8->103.7 MiB | 12->246.7 MiB | 16->246.7 MiB | 20->247.1 MiB
| 24->248.2 MiB
aggregate Python RSS (workers + coordinator):
8->1,049.2 MiB | 12->1,549.9 MiB | 16->1,944.3 MiB | 20->2,335.8 MiB
| 24->2,530.8 MiB
NOTE: 1839 MiB x multipliers for the 24 MiB-chunk working set are NOT
assumed; every value is measured.
19. CPU utilization (machine-wide, sampled during worker phase)
workers=8 average 0.616 | peak 0.676
workers=12 average 0.693 | peak 0.760
workers=16 average 0.715 | peak 0.890
workers=20 average 0.754 | peak 0.998
workers=24 average 0.961 | peak 1.000
20. Coordinator overhead (merge + dispatch/spawn)
merge (run1): 8->3.41 s | 12->3.45 s | 16->3.31 s | 20->3.39 s | 24->3.27 s
dispatch/spawn: 8->0.71 s | 12->0.69 s | 16->0.77 s | 20->0.75 s | 24->1.10 s
21. Worker imbalance (max wall / mean wall, run1)
8->1.1345 | 12->1.1363 | 16->1.4846 | 20->1.7611 | 24->1.0638
(12/16/20 are quantisation-bound by 2-chunk stragglers: 22 chunks do not
divide evenly; 24 -> 22 workers x 1 chunk -> best balance)
22. Actual process count
requested=8 actual=8 non-empty partitions=8 (match)
requested=12 actual=12 non-empty partitions=12 (match)
requested=16 actual=16 non-empty partitions=16 (match)
requested=20 actual=20 non-empty partitions=20 (match)
requested=24 actual=22 non-empty partitions=22, empty partitions
(worker ids 22,23) -> DISCREPANCY RECORDED
(22 atomic chunks cannot populate 24 workers;
the executor legitimately created 22 processes)
Worker PIDs recorded per run; completion order is never used for
semantic ordering (merge is by source byte order).
23. Speedup (vs 8-worker baseline, span run1)
speedup(8) = 1.000
speedup(12) = 48.859 / 33.729 = 1.449
speedup(16) = 48.859 / 33.579 = 1.455
speedup(20) = 48.859 / 33.646 = 1.452
speedup(24) = 48.859 / 19.543 = 2.500
24. Efficiency (vs 8-worker baseline)
efficiency(8) = 1.000
efficiency(12) = speedup / (12/8) = 1.449 / 1.500 = 0.966
efficiency(16) = 1.455 / 2.000 = 0.728
efficiency(20) = 1.452 / 2.500 = 0.581
efficiency(24) = 2.500 / 3.000 = 0.833
25. MAX_SAFE_TESTED
MAX_SAFE_TESTED = 24
(24 is the largest required candidate; all gates PASS:
correctness 14/14, boundary, state/carry, interruption/resume, mutation
10/10, reproducibility, checkpoint byte-identical.)
26. PREFERRED_WORKER_COUNT
PREFERRED_WORKER_COUNT = 24
Evidence: 2.500x speedup vs the 8-worker baseline at 0.833 efficiency
is the ONLY candidate that breaks the 2-chunk-per-worker plateau
(12/16/20 all sit at ~1.45x because 22 atomic chunks quantise into
2-chunk straggler workers). 24 also has the best balance (imbalance
1.0638), the highest CPU utilisation (0.961 avg), and the lowest
coordinator merge time (3.27 s). Memory 4,903 MiB total worker peak /
2,531 MiB aggregate RSS is ~10% of the 47.93 GiB machine RAM and the
minimum available RAM stayed >= 23.5 GiB throughout.
The 'prefer fewer workers when close' rule does not bind: 12 (0.966 eff)
is the most efficient, but its 1.449x speedup is NOT close to 24's
2.500x, and 12/16/20 are mutually close (~1.45x) so the fewest of THOSE
(12) is the natural pick within the plateau cluster.
27. Resource trade-off
wall speed eff tot worker agg RSS cpu avg imb
workers=8 48.86 s 1.000 1.000 1839 MiB 1049 MiB 0.616 1.135
workers=12 33.73 s 1.449 0.966 2605 MiB 1550 MiB 0.693 1.136
workers=16 33.58 s 1.455 0.728 3524 MiB 1944 MiB 0.715 1.485
workers=20 33.65 s 1.452 0.581 4444 MiB 2336 MiB 0.754 1.761
workers=24 19.54 s 2.500 0.833 4903 MiB 2531 MiB 0.961 1.064
24 workers buys 32.3 s less wall-clock per 512 MiB workload than the
baseline for +2,664 MiB total worker peak (+1,482 MiB aggregate RSS);
8->24 = -59.9% wall-clock; 12->24 = -42.1% wall-clock for +2,298 MiB peak.
28. Failed candidates
NONE. All five candidates (8/12/16/20/24) passed every gate.
No failure artifact required preservation.
29. Checkpoint SHA before / after (real, READ-ONLY)
real_checkpoint_path :
ml/p3/p3_s251_external_ingest/output/s251_real_checkpoint.json
checkpoint_sha_before = 8c6cad1ca561308e0ba91fcdca3f8ebb95e6d79abc55edaf514d6565f6ce4010
checkpoint_sha_after = 8c6cad1ca561308e0ba91fcdca3f8ebb95e6d79abc55edaf514d6565f6ce4010
byte_identical : True
run_id : P3_S251_REAL_TICK_RUN_001 (untouched)
last_completed_chunk : 187 | next_chunk 188 | next_byte_start 4483711426 (UNCHANGED)
30. Operational-policy status
OPERATIONAL_POLICY : NOT CHANGED
SAFE_MAX_CHUNK_SIZE : 24 MiB (UNCHANGED)
SAFE_CHUNKS_PER_TURN : NOT ESTABLISHED (UNCHANGED)
production worker configuration : UNCHANGED
CURRENT_DECISION_GATE.md / CURRENT_PROJECT_STATE.md : NOT MODIFIED
production MQL5/MQH : UNCHANGED | frozen contracts : UNCHANGED
historical evidence : UNCHANGED
real checkpoint : byte-identical before/after
CHUNK 188 : NOT PROCESSED / NOT AUTHORIZED
new real Tickstory bytes : NONE read (benchmark read only the disposable
synthetic fixture [0, 536870912))
Dukascopy : NOT accessed | ML : NOT run
31. Recommendation for possible future production use
EVIDENCE ONLY - no operational promotion is made by this experiment.
If the owner later authorises a production worker-count change for the
validated ingestion architecture, the evidence supports 24 workers as the
highest-throughput safe point (2.500x vs 8, eff 0.833, all gates PASS),
with 12 workers as the most memory-efficient safe point (0.966 eff) for
environments where memory saving matters more than wall-clock.
A later owner authorization is required before any promotion; the real
SAFE_* values, the real checkpoint and the decision gate remain untouched.
32. Limitations
1. The parallel engine is the P3-S25.1F lean verifier (no per-line chunk
digest); raw/parsed hashes are therefore proven by byte-coverage +
boundary identity + the canonical full-digest atomic reference equality
(the F/G contract: lean == full-digest was proven on a 72 MiB range in
P3-S25.1E and re-asserted here on the 512 MiB canonical atomic
reference, all 22 chunks PASS).
2. All worker runs executed warm-cache: the 2.1 GiB fixture was fully
resident in the OS page cache after the reference pass; no cold-cache
runs were performed this session. Batch-to-batch timing variance
(8-worker span 52.76 s in the first batch vs 46.6-48.9 s in later
batches) is attributed to cache warmth / system noise; the decision
uses the final coherent pass (batch 3). Timing repeatability is
evidence of stability, not a correctness argument.
3. N=24 requested 24 but only 22 non-empty worker partitions exist (22
atomic chunks); the 2 empty partition ids (22, 23) are recorded and the
gate evaluation uses the actual 22-worker set. This is the legitimate
'fewer processes due to empty tasks' case; it does not weaken any gate.
4. 24 workers saturate the 24 logical CPUs (machine-wide CPU average
0.961, peak 1.000) - a near-saturation condition; further scaling above
24 physical/logical CPUs was not tested (brief: do not expand beyond
24).
5. Parsing is pure-Python and CPU-bound; results are machine/OS dependent
(2-socket Xeon Gold 5220, 24 logical, no HT).
6. The synthetic fixture is schema-identical but byte-density may differ
slightly from the real 34.5 GB Tickstory source; real-source expansion
is forbidden for this experiment.
7. Duplicated worker set mutation for N=24 appends a NEW worker id (22),
keeping the worker-identity audit meaningful for the 24->22 process
discrepancy case.
33. Final classification
CLASSIFICATION : A - OPTIMIZATION CANDIDATE IDENTIFIED
MAX_SAFE_TESTED = 24 ; PREFERRED_WORKER_COUNT = 24
Every value in this report is machine-derived (harness
p3_s251q_harness.py + evidence
workload_benchmark/worker_scaling_q/output/*.json). A was not forced:
all five candidates passed every gate; 24 was selected because it is the
only candidate that breaks the 2-chunk imbalance plateau (2.500x speedup,
0.833 efficiency, imbalance 1.064, CPU 0.961) while remaining far inside
machine memory limits. 12 workers remains the back-up preference for
memory-constrained operation (0.966 efficiency).
End of P3-S25.1Q final report (generated from machine evidence).