# P3.2.2 DE-OVERLAP & SURVIVAL REPORT — SNIPERGOLD_ML ```text Date : 2026-08-22 Session : P3.2.2 — Formal Event De-Overlap + Survival/Time-to-Event Diagnostic Status : RESEARCH / DIAGNOSTIC ONLY — no production changes Checkpoint : P3_1_P3_2_1_CHECKPOINT_SHA = dc1faa92f36ad22d4062315a5654965a08a006dc Artifacts : ml/p3/deoverlap_audit.py ; ml/p3/survival_diagnostic.py ml/p3/output/p3_2_deoverlap_h8.json / h16.json / h24.json ml/p3/output/p3_2_survival.json ``` --- ## 0. CHECKPOINT & PROVENANCE ```text P2_SOURCE_SHA : 8d330343af688e2de2b2a1d12ce19a582714709a P2.6_SOURCE_SHA : 8d330343... (train_p26=e510ef3b, build_features_p2=4680f905) P3_1_P3_2_1_CHECKPOINT : dc1faa92f36ad22d4062315a5654965a08a006dc (committed & pushed) FEATURE_CONTRACT_SHA : C44CC6F2... (canonical) DATASET_SHA : e85a08616191fb7ce4bbe79c1754228ddbcf54562034a322747501c7aabe124e symbol/tf : XAUUSD / M15 ; 2017-01-01 .. 2026-08-20 event semantics : existing FEATURE_CONTRACT v1.0 (f9/f7/f10/f11) entry semantics : close of the closed decision bar primary combo : TP=1.5 ATR / SL=0.75 ATR (candidate, 2:1 semantic) sensitivity : 1.0/1.0 (symmetric) invalidation : opposite structure break (f8 or f5 flip) — secondary only scripts : deoverlap_audit.py / survival_diagnostic.py (SHA recorded in JSON) ``` --- ## A. EVENT POPULATION ```text Family definition (existing) direction n_original E_BUY f9==1 & f7>0 (CHoCH confirms bull sweep) long 27096 E_SELL f9==1 & f7<0 (CHoCH confirms bear sweep) short 25259 E_EQH f10==1 (equal-high swept) short 877 E_EQL f11==1 (equal-low swept) long 1617 E_BUY_STRICT E_BUY & f18>=60 long 13143 E_SELL_STRICT E_SELL & f18>=60 short 9859 ``` --- ## B. CLUSTER STATISTICS (overlap window H) | H | family | n_original | clusters | lead | follow-on | retention | med_cluster | max_cluster | |---|--------|-----------:|---------:|-----:|----------:|----------:|------------:|------------:| | 8 | E_BUY | 27096 | 420 | 420 | 26676 | 1.55% | 41 | 388 | | 8 | E_SELL | 25259 | 391 | 391 | 24868 | 1.55% | 43 | 473 | | 16 | E_BUY | 27096 | 397 | 397 | 26699 | **1.47%** | 46 | 388 | | 16 | E_SELL | 25259 | 371 | 371 | 24888 | 1.47% | 46 | 473 | | 16 | E_EQH | 877 | 18 | 18 | 859 | 2.05% | 17 | 384 | | 16 | E_EQL | 1617 | 18 | 18 | 1599 | 1.11% | 42 | 504 | | 24 | E_BUY | 27096 | 373 | 373 | 26723 | 1.38% | 48 | 391 | | 24 | E_SELL | 25259 | 357 | 357 | 24902 | 1.41% | 47 | 473 | ```text De-overlap verified: next_selected - selected > H = TRUE for all families/H. Key finding: 97-99% of events are FOLLOW-ON within large clusters (median cluster 46 bars, max 473). The f9-confirm event stream is HIGHLY redundant: only ~1.5% of events are truly independent (lead). ``` --- ## C. DE-OVERLAP H=8 (1.5/0.75 ATR) | family | ALL W/L/U | LEAD W/L/U | LEAD MFE/MAE | |--------|-----------|------------|--------------| | E_BUY | 0.265/0.581/0.146 | 0.281/0.640/0.074 | 1.32 / 1.48 | | E_SELL | 0.266/0.587/0.139 | 0.289/0.637/0.059 | 1.43 / 1.49 | | E_EQH | 0.290/0.520/0.186 | 0.158/0.684/0.158 | 0.74 / 1.59 | | E_EQL | 0.231/0.584/0.176 | 0.179/0.679/0.107 | 0.98 / 1.26 | ## D. DE-OVERLAP H=16 (1.5/0.75 ATR) — PRIMARY | family | ALL W/L/U | LEAD W/L/U | LEAD MFE/MAE | |--------|-----------|------------|--------------| | E_BUY | 0.310/0.629/0.052 | 0.310/0.668/0.018 | 1.91 / 2.06 | | E_SELL | 0.310/0.633/0.048 | 0.307/0.658/0.019 | 2.14 / 2.02 | | E_EQH | 0.326/0.577/0.092 | **0.167/0.778/0.056** | 0.91 / 1.77 | | E_EQL | 0.280/0.631/0.080 | 0.278/0.667/0.056 | 1.25 / 1.88 | | E_BUY_STRICT | 0.317/0.623/0.050 | 0.304/0.659/0.031 | 1.68 / 1.73 | | E_SELL_STRICT | 0.305/0.634/0.053 | 0.325/0.633/0.039 | 1.75 / 1.94 | ## E. DE-OVERLAP H=24 (1.5/0.75 ATR) | family | ALL W/L/U | LEAD W/L/U | |--------|-----------|------------| | E_BUY | 0.322/0.640/0.029 | 0.316/0.670/0.008 | | E_SELL | 0.321/0.644/0.026 | 0.308/0.672/0.008 | | E_EQH | 0.344/0.603/0.048 | 0.167/0.833/0.000 | | E_EQL | 0.291/0.644/0.056 | 0.091/0.818/0.091 | --- ## F. OUTCOME COMPARISON — ALL vs LEAD (conclusion) ```text 1. The apparent E_EQH asymmetry (ALL: W 0.326 vs L 0.577 at H16) DISAPPEARS COMPLETELY in LEAD (W 0.167 vs L 0.778). The ALL asymmetry = artifact of repeated counting within clusters (the outcome window is shared by many events). 2. All families in LEAD: LOSS >> WIN (LOSS ~2x to 4.7x WIN). 3. De-overlap slightly raises P_LOSS for E_BUY/E_SELL (0.629->0.668) and REVEALS that the existing stream has no TP-before-SL edge. 4. Censoring drops quickly with H (H8 6-18% -> H16 2-6% -> H24 0-9%). ``` --- ## G. SURVIVAL / TIME-TO-EVENT (LEAD events, de-overlap H16, primary 1.5/0.75) | H | E_BUY TP/SL/cens | E_SELL TP/SL/cens | E_EQH TP/SL/cens | E_EQL TP/SL/cens | |---|------------------|-------------------|------------------|------------------| | 8 | 0.277/0.642/0.076 | 0.288/0.636/0.059 | 0.167/0.667/0.167 | 0.222/0.667/0.111 | | 16 | 0.310/0.668/0.018 | 0.307/0.658/0.019 | 0.167/0.778/0.056 | 0.278/0.667/0.056 | | 24 | 0.315/0.673/0.008 | 0.310/0.666/0.008 | 0.167/0.833/0.000 | 0.278/0.667/0.056 | | 48 | 0.320/0.675/0.000 | 0.313/0.671/0.000 | 0.167/0.833/0.000 | 0.333/0.667/0.000 | ```text Empirical setup lifetime (lead events): median time-to-event : 1-2 bars median TP time : 2-3 bars (E_BUY), 2 bars (E_SELL) median SL time : 1-2 bars (SL is almost always faster than TP) P(SL) asymptote ~0.67 ; P(TP) asymptote ~0.31-0.32 (E_BUY/E_SELL) ~90% of outcomes occur within 8 bars; >98% within 16 bars. Conclusion: setups "live" 1-8 bars; SL wins ~2:1. The 24-bar horizon (old label) far exceeds the setup lifetime and is dominated by late SL (not new information). ``` --- ## H. MFE / MAE CONDITIONAL (lead events, H=48, primary) | family | status | n | MFE med | MAE med | MFE/MAE ratio | |--------|--------|---:|--------:|--------:|--------------:| | E_BUY | WIN | 127 | 5.37 | 2.52 | 2.53 | | E_BUY | LOSS | 268 | 3.01 | 4.46 | 0.66 | | E_SELL | WIN | 116 | 5.15 | 1.92 | 2.83 | | E_SELL | LOSS | 249 | 3.04 | 4.66 | 0.58 | | E_EQH | WIN | 3 | 5.20 | 0.11 | 53.5 (small n) | | E_EQH | LOSS | 15 | 1.47 | 2.57 | 0.48 | | E_EQL | WIN | 6 | 2.81 | 3.43 | 1.50 | | E_EQL | LOSS | 12 | 2.25 | 3.32 | 0.72 | ```text - WIN events: MFE/MAE ratio 2.5-2.8 (favorable excursion far larger than adverse excursion) — BUT this is post-hoc (conditioned on outcome), not a structure exploitable without look-ahead. - LOSS events: median MAE 4.5-4.7 ATR within 48 bars -> after the 0.75 ATR SL is touched (bars 1-2), price keeps moving against by ~4.5 ATR on average. Consistent with a small SL + trend continuing against (late entry). - No favorable MFE/MAE asymmetry at the unconditional level: median MFE ≈ median MAE for all families (see P3.2.1 §4; confirmed in LEAD). ``` --- ## I. INVALIDATION (secondary outcome; opposite structure break f8/f5) | family | P(inv within 48 bars) | P(inv BEFORE TP/SL) | P(TP first) | P(SL first) | median t_inv | |--------|-------------------:|---------------------:|-----------:|-----------:|-------------:| | E_BUY | 0.73 | 0.003 | 0.32 | 0.66 | 24 bars | | E_SELL | 0.74 | 0.000 | 0.31 | 0.66 | 22 bars | | E_EQH | 1.00 | 0.333 | 0.00 | 0.06 | 1 bar | | E_EQL | (small n) | — | — | — | — | ```text - E_BUY/E_SELL: invalidation (structure break) almost NEVER precedes TP/SL (0.0-0.3%) because the outcome occurs within 1-3 bars while invalidation has a median of 22-24 bars. -> structure-break invalidation is NOT useful as a primary outcome for this stream; SL is far faster. - E_EQH: invalidation median 1 bar & 100% within 48 bars -> the short direction after an equal-high sweep is immediately opposed by structure; consistent with LOSS dominance. - Conclusion: for existing event units, SL (price) is the de-facto invalidator; the opposite structure break is too slow to be a primary label. ``` --- ## J. SERIAL DEPENDENCE — ALL vs LEAD (H=16, primary, outcome WIN/LOSS) | family | set | n_resolved | ac1 | P(same outcome) | mean run | max run | |--------|-----|-----------:|----:|----------------:|---------:|--------:| | E_BUY | ALL | 25451 | **0.498** | 0.778 | 4.5 | 64 | | E_BUY | LEAD | 388 | **-0.025** | 0.556 | 2.2 | 11 | | E_SELL | ALL | 23813 | **0.501** | 0.780 | 4.5 | 49 | | E_SELL | LEAD | 371 | **0.016** | 0.574 | 2.3 | 17 | | E_EQH | ALL | 792 | 0.466 | 0.753 | 4.0 | 25 | | E_EQH | LEAD | (n<50) | — | — | — | — | ```text De-overlap REMOVES outcome serial dependence: lag-1 autocorrelation 0.50 -> ~0.0 ; P(same outcome) 0.78 -> 0.56 ; mean run 4.5 -> 2.2. Independence SATISFIES the basic requirement for ML labels. Note: P_same_dir_consec = 1.0 everywhere (single direction per family) — that metric is only meaningful when families mix directions; not used for conclusions. ``` --- ## K. LABEL CONTRACT ASSESSMENT — SGML_L3_EVENT (candidate) | Criterion | Assessment | Evidence | |-----------|:---:|----------| | Causal | PASS | outcome measured from the entry bar forward, no look-ahead | | SMC aligned | PASS | unit = SMC event bar; outcome = TP-before-SL (setup validation) | | Execution realistic | PASS | entry = close of the decision bar (B ≈ A, Δ 0.008 ATR) | | **Independent** | **PARTIAL** | achieved only AFTER mandatory de-overlap (ac1 0.5->0); without de-overlap it FAILS completely | | Ambiguity | PASS | AMBIGUOUS 0.5-1.6% (low; excluded) | | Censoring | PASS | H16 2%; H8 6-8%; UNRESOLVED excluded, not forced to LOSS | | Economically interpretable | PASS | WIN/LOSS = first TP/SL; semantic 2:1 ratio | | Suitable for meta-labeling | PASS (mechanism) | binary, independent, low ambiguity/censoring | | **Unit validity** | **FAIL** | f9-confirm event stream 97-99% redundant; leads only ~1.5%; n_lead 397/371/18/18 | ```text The SGML_L3_EVENT label mechanism = technically VIABLE. THE EVENT UNIT (existing stream) = NOT VALID as an independent setup unit: too dense/redundant; after de-overlap the remaining sample is very small (~40 leads/year/family) and the outcome is LOSS-dominated (SL ~2x TP). ``` --- ## L. CANDIDATE LABEL STATUS ```text C — EVENT STREAM ITSELF INVALID (as an independent setup unit for ML labels) ``` With the combined note: ```text A — EVENT LABEL VIABLE : PARTIAL (L3 label mechanism viable; independence/ambiguity/censoring PASS) B — EVENT LABEL VIABLE WITH CENSORING : NOT REQUIRED for H16+ (2% censoring); survival-style useful for H<=8 / time-to-event C — EVENT STREAM ITSELF INVALID : STRONGLY SUPPORTED (retention 1.5%, small n_lead, LOSS-dominated after de-overlap) D — INCONCLUSIVE : no ``` --- ## M. EVIDENCE CLASSIFICATION ```text TRUSTED : cluster/de-overlap stats (verified next>H), survival incidence, serial dependence ALL vs LEAD, invalidation (deterministic, full provenance, dataset hash matches) INCONCLUSIVE : E_EQH / E_EQL lead stats (n=18) — very wide CIs; the direction is consistent (LOSS-dominated) but the size is inadequate INVALID : the E_EQH asymmetry in ALL events (P3.2.1 §5) — proven an overlap artifact, gone after de-overlap (P3.2.2 F) SUPERSEDED : none (P2.6/P3.1/P3.2.1 remain valid for their purpose) ``` --- ## N. RECOMMENDATION FOR P3.3 ```text 1. SGML_L3_EVENT as a label mechanism: KEEP as a candidate, WITH mandatory de-overlap (lead event per cluster, gap > 16) and the WIN/LOSS/UNRESOLVED/AMBIGUOUS classes (UNRESOLVED & AMBIGUOUS excluded). 2. The EXISTING EVENT UNIT (f9-confirm/EQH/EQL as event bars) is NOT suitable as a setup unit: 1.5% retention -> the de-overlapped dataset is too small and the outcome is LOSS-dominated. DO NOT retrain a model on it. 3. P3.3 must question the EVENT UNIT DEFINITION (feature/setup architecture revision): a truly discrete unit (onset/state transition, order block / breaker / FVG boundary) — outside this session's mandate, no hot-fix. 4. Horizon: 8-16 bars (setup lifetime); SL is the de-facto invalidator; structure-break invalidation is too slow -> not a primary outcome. 5. LSTM / regime / meta-label REMAIN UNTESTED until (a) an independent event unit is available and (b) the feature re-audit against the final label is done. 6. Multiple testing: 6 families x 3 H x 2 combos + survival + invalidation ≈ 80 statistics; only CONSISTENT patterns (LOSS>WIN for all families/all H after de-overlap) are used as conclusions; no cherry-picking. ``` --- ## Appendix — Artifacts ```text Script : ml/p3/deoverlap_audit.py, ml/p3/survival_diagnostic.py Output : ml/p3/output/p3_2_deoverlap_h8.json / h16.json / h24.json ml/p3/output/p3_2_survival.json Report : docs/P3_2_2_DEOVERLAP_SURVIVAL.md Checkpoint: P3_1_P3_2_1_CHECKPOINT_SHA = dc1faa92... (committed, pushed) ``` **Closing note:** P3.2.2 completed without production changes. No training, no TP/SL/horizon/filter optimization. Core findings: the L3 label mechanism is viable, but the existing event stream is not valid as an independent setup unit — research must move to a truly discrete event-unit definition before discussing models.