SniperGold_ML/docs/P3_2_2_DEOVERLAP_SURVIVAL.md

# P3.2.2 DE-OVERLAP & SURVIVAL REPORT — SNIPERGOLD_ML

```text
Date       : 2026-08-22
Session    : P3.2.2 — Formal Event De-Overlap + Survival/Time-to-Event Diagnostic
Status     : RESEARCH / DIAGNOSTIC ONLY — no production changes
Checkpoint : P3_1_P3_2_1_CHECKPOINT_SHA = dc1faa92f36ad22d4062315a5654965a08a006dc
Artifacts  : ml/p3/deoverlap_audit.py ; ml/p3/survival_diagnostic.py
             ml/p3/output/p3_2_deoverlap_h8.json / h16.json / h24.json
             ml/p3/output/p3_2_survival.json
```

---

## 0. CHECKPOINT & PROVENANCE

```text
P2_SOURCE_SHA         : 8d330343af688e2de2b2a1d12ce19a582714709a
P2.6_SOURCE_SHA       : 8d330343... (train_p26=e510ef3b, build_features_p2=4680f905)
P3_1_P3_2_1_CHECKPOINT : dc1faa92f36ad22d4062315a5654965a08a006dc (committed & pushed)
FEATURE_CONTRACT_SHA  : C44CC6F2... (canonical)
DATASET_SHA           : e85a08616191fb7ce4bbe79c1754228ddbcf54562034a322747501c7aabe124e
symbol/tf             : XAUUSD / M15 ; 2017-01-01 .. 2026-08-20
event semantics       : existing FEATURE_CONTRACT v1.0 (f9/f7/f10/f11)
entry semantics       : close of the closed decision bar
primary combo         : TP=1.5 ATR / SL=0.75 ATR (candidate, 2:1 semantic)
sensitivity           : 1.0/1.0 (symmetric)
invalidation          : opposite structure break (f8 or f5 flip) — secondary only
scripts               : deoverlap_audit.py / survival_diagnostic.py (SHA recorded in JSON)
```

---

## A. EVENT POPULATION

```text
Family        definition (existing)                     direction   n_original
E_BUY         f9==1 & f7>0  (CHoCH confirms bull sweep)  long       27096
E_SELL        f9==1 & f7<0  (CHoCH confirms bear sweep)  short      25259
E_EQH         f10==1 (equal-high swept)                  short       877
E_EQL         f11==1 (equal-low swept)                   long        1617
E_BUY_STRICT  E_BUY & f18>=60                            long       13143
E_SELL_STRICT E_SELL & f18>=60                           short       9859
```

---

## B. CLUSTER STATISTICS (overlap window H)

| H | family | n_original | clusters | lead | follow-on | retention | med_cluster | max_cluster |
|---|--------|-----------:|---------:|-----:|----------:|----------:|------------:|------------:|
| 8 | E_BUY | 27096 | 420 | 420 | 26676 | 1.55% | 41 | 388 |
| 8 | E_SELL | 25259 | 391 | 391 | 24868 | 1.55% | 43 | 473 |
| 16 | E_BUY | 27096 | 397 | 397 | 26699 | **1.47%** | 46 | 388 |
| 16 | E_SELL | 25259 | 371 | 371 | 24888 | 1.47% | 46 | 473 |
| 16 | E_EQH | 877 | 18 | 18 | 859 | 2.05% | 17 | 384 |
| 16 | E_EQL | 1617 | 18 | 18 | 1599 | 1.11% | 42 | 504 |
| 24 | E_BUY | 27096 | 373 | 373 | 26723 | 1.38% | 48 | 391 |
| 24 | E_SELL | 25259 | 357 | 357 | 24902 | 1.41% | 47 | 473 |

```text
De-overlap verified: next_selected - selected > H = TRUE for all families/H.
Key finding: 97-99% of events are FOLLOW-ON within large clusters
(median cluster 46 bars, max 473). The f9-confirm event stream is HIGHLY redundant:
only ~1.5% of events are truly independent (lead).
```

---

## C. DE-OVERLAP H=8 (1.5/0.75 ATR)

| family | ALL W/L/U | LEAD W/L/U | LEAD MFE/MAE |
|--------|-----------|------------|--------------|
| E_BUY | 0.265/0.581/0.146 | 0.281/0.640/0.074 | 1.32 / 1.48 |
| E_SELL | 0.266/0.587/0.139 | 0.289/0.637/0.059 | 1.43 / 1.49 |
| E_EQH | 0.290/0.520/0.186 | 0.158/0.684/0.158 | 0.74 / 1.59 |
| E_EQL | 0.231/0.584/0.176 | 0.179/0.679/0.107 | 0.98 / 1.26 |

## D. DE-OVERLAP H=16 (1.5/0.75 ATR) — PRIMARY

| family | ALL W/L/U | LEAD W/L/U | LEAD MFE/MAE |
|--------|-----------|------------|--------------|
| E_BUY | 0.310/0.629/0.052 | 0.310/0.668/0.018 | 1.91 / 2.06 |
| E_SELL | 0.310/0.633/0.048 | 0.307/0.658/0.019 | 2.14 / 2.02 |
| E_EQH | 0.326/0.577/0.092 | **0.167/0.778/0.056** | 0.91 / 1.77 |
| E_EQL | 0.280/0.631/0.080 | 0.278/0.667/0.056 | 1.25 / 1.88 |
| E_BUY_STRICT | 0.317/0.623/0.050 | 0.304/0.659/0.031 | 1.68 / 1.73 |
| E_SELL_STRICT | 0.305/0.634/0.053 | 0.325/0.633/0.039 | 1.75 / 1.94 |

## E. DE-OVERLAP H=24 (1.5/0.75 ATR)

| family | ALL W/L/U | LEAD W/L/U |
|--------|-----------|------------|
| E_BUY | 0.322/0.640/0.029 | 0.316/0.670/0.008 |
| E_SELL | 0.321/0.644/0.026 | 0.308/0.672/0.008 |
| E_EQH | 0.344/0.603/0.048 | 0.167/0.833/0.000 |
| E_EQL | 0.291/0.644/0.056 | 0.091/0.818/0.091 |

---

## F. OUTCOME COMPARISON — ALL vs LEAD (conclusion)

```text
1. The apparent E_EQH asymmetry (ALL: W 0.326 vs L 0.577 at H16) DISAPPEARS
   COMPLETELY in LEAD (W 0.167 vs L 0.778). The ALL asymmetry = artifact of
   repeated counting within clusters (the outcome window is shared by many events).
2. All families in LEAD: LOSS >> WIN (LOSS ~2x to 4.7x WIN).
3. De-overlap slightly raises P_LOSS for E_BUY/E_SELL (0.629->0.668) and
   REVEALS that the existing stream has no TP-before-SL edge.
4. Censoring drops quickly with H (H8 6-18% -> H16 2-6% -> H24 0-9%).
```

---

## G. SURVIVAL / TIME-TO-EVENT (LEAD events, de-overlap H16, primary 1.5/0.75)

| H | E_BUY TP/SL/cens | E_SELL TP/SL/cens | E_EQH TP/SL/cens | E_EQL TP/SL/cens |
|---|------------------|-------------------|------------------|------------------|
| 8 | 0.277/0.642/0.076 | 0.288/0.636/0.059 | 0.167/0.667/0.167 | 0.222/0.667/0.111 |
| 16 | 0.310/0.668/0.018 | 0.307/0.658/0.019 | 0.167/0.778/0.056 | 0.278/0.667/0.056 |
| 24 | 0.315/0.673/0.008 | 0.310/0.666/0.008 | 0.167/0.833/0.000 | 0.278/0.667/0.056 |
| 48 | 0.320/0.675/0.000 | 0.313/0.671/0.000 | 0.167/0.833/0.000 | 0.333/0.667/0.000 |

```text
Empirical setup lifetime (lead events):
  median time-to-event : 1-2 bars
  median TP time       : 2-3 bars (E_BUY), 2 bars (E_SELL)
  median SL time       : 1-2 bars  (SL is almost always faster than TP)
  P(SL) asymptote ~0.67 ; P(TP) asymptote ~0.31-0.32 (E_BUY/E_SELL)
  ~90% of outcomes occur within 8 bars; >98% within 16 bars.
Conclusion: setups "live" 1-8 bars; SL wins ~2:1. The 24-bar horizon (old label)
far exceeds the setup lifetime and is dominated by late SL (not new information).
```

---

## H. MFE / MAE CONDITIONAL (lead events, H=48, primary)

| family | status | n | MFE med | MAE med | MFE/MAE ratio |
|--------|--------|---:|--------:|--------:|--------------:|
| E_BUY | WIN | 127 | 5.37 | 2.52 | 2.53 |
| E_BUY | LOSS | 268 | 3.01 | 4.46 | 0.66 |
| E_SELL | WIN | 116 | 5.15 | 1.92 | 2.83 |
| E_SELL | LOSS | 249 | 3.04 | 4.66 | 0.58 |
| E_EQH | WIN | 3 | 5.20 | 0.11 | 53.5 (small n) |
| E_EQH | LOSS | 15 | 1.47 | 2.57 | 0.48 |
| E_EQL | WIN | 6 | 2.81 | 3.43 | 1.50 |
| E_EQL | LOSS | 12 | 2.25 | 3.32 | 0.72 |

```text
- WIN events: MFE/MAE ratio 2.5-2.8 (favorable excursion far larger than
  adverse excursion) — BUT this is post-hoc (conditioned on outcome), not a
  structure exploitable without look-ahead.
- LOSS events: median MAE 4.5-4.7 ATR within 48 bars -> after the 0.75 ATR SL is
  touched (bars 1-2), price keeps moving against by ~4.5 ATR on average.
  Consistent with a small SL + trend continuing against (late entry).
- No favorable MFE/MAE asymmetry at the unconditional level:
  median MFE ≈ median MAE for all families (see P3.2.1 §4; confirmed in LEAD).
```

---

## I. INVALIDATION (secondary outcome; opposite structure break f8/f5)

| family | P(inv within 48 bars) | P(inv BEFORE TP/SL) | P(TP first) | P(SL first) | median t_inv |
|--------|-------------------:|---------------------:|-----------:|-----------:|-------------:|
| E_BUY | 0.73 | 0.003 | 0.32 | 0.66 | 24 bars |
| E_SELL | 0.74 | 0.000 | 0.31 | 0.66 | 22 bars |
| E_EQH | 1.00 | 0.333 | 0.00 | 0.06 | 1 bar |
| E_EQL | (small n) | — | — | — | — |

```text
- E_BUY/E_SELL: invalidation (structure break) almost NEVER precedes
  TP/SL (0.0-0.3%) because the outcome occurs within 1-3 bars while invalidation
  has a median of 22-24 bars. -> structure-break invalidation is NOT useful as a
  primary outcome for this stream; SL is far faster.
- E_EQH: invalidation median 1 bar & 100% within 48 bars -> the short direction
  after an equal-high sweep is immediately opposed by structure; consistent with
  LOSS dominance.
- Conclusion: for existing event units, SL (price) is the de-facto invalidator;
  the opposite structure break is too slow to be a primary label.
```

---

## J. SERIAL DEPENDENCE — ALL vs LEAD (H=16, primary, outcome WIN/LOSS)

| family | set | n_resolved | ac1 | P(same outcome) | mean run | max run |
|--------|-----|-----------:|----:|----------------:|---------:|--------:|
| E_BUY | ALL | 25451 | **0.498** | 0.778 | 4.5 | 64 |
| E_BUY | LEAD | 388 | **-0.025** | 0.556 | 2.2 | 11 |
| E_SELL | ALL | 23813 | **0.501** | 0.780 | 4.5 | 49 |
| E_SELL | LEAD | 371 | **0.016** | 0.574 | 2.3 | 17 |
| E_EQH | ALL | 792 | 0.466 | 0.753 | 4.0 | 25 |
| E_EQH | LEAD | (n<50) | — | — | — | — |

```text
De-overlap REMOVES outcome serial dependence:
  lag-1 autocorrelation 0.50 -> ~0.0 ; P(same outcome) 0.78 -> 0.56 ;
  mean run 4.5 -> 2.2. Independence SATISFIES the basic requirement for ML labels.
Note: P_same_dir_consec = 1.0 everywhere (single direction per family) — that
metric is only meaningful when families mix directions; not used for conclusions.
```

---

## K. LABEL CONTRACT ASSESSMENT — SGML_L3_EVENT (candidate)

| Criterion | Assessment | Evidence |
|-----------|:---:|----------|
| Causal | PASS | outcome measured from the entry bar forward, no look-ahead |
| SMC aligned | PASS | unit = SMC event bar; outcome = TP-before-SL (setup validation) |
| Execution realistic | PASS | entry = close of the decision bar (B ≈ A, Δ 0.008 ATR) |
| **Independent** | **PARTIAL** | achieved only AFTER mandatory de-overlap (ac1 0.5->0); without de-overlap it FAILS completely |
| Ambiguity | PASS | AMBIGUOUS 0.5-1.6% (low; excluded) |
| Censoring | PASS | H16 2%; H8 6-8%; UNRESOLVED excluded, not forced to LOSS |
| Economically interpretable | PASS | WIN/LOSS = first TP/SL; semantic 2:1 ratio |
| Suitable for meta-labeling | PASS (mechanism) | binary, independent, low ambiguity/censoring |
| **Unit validity** | **FAIL** | f9-confirm event stream 97-99% redundant; leads only ~1.5%; n_lead 397/371/18/18 |

```text
The SGML_L3_EVENT label mechanism = technically VIABLE.
THE EVENT UNIT (existing stream) = NOT VALID as an independent setup unit:
too dense/redundant; after de-overlap the remaining sample is very small
(~40 leads/year/family) and the outcome is LOSS-dominated (SL ~2x TP).
```

---

## L. CANDIDATE LABEL STATUS

```text
C — EVENT STREAM ITSELF INVALID
    (as an independent setup unit for ML labels)
```

With the combined note:

```text
A — EVENT LABEL VIABLE            : PARTIAL (L3 label mechanism viable;
                                    independence/ambiguity/censoring PASS)
B — EVENT LABEL VIABLE WITH CENSORING : NOT REQUIRED for H16+ (2% censoring);
                                    survival-style useful for H<=8 / time-to-event
C — EVENT STREAM ITSELF INVALID   : STRONGLY SUPPORTED (retention 1.5%, small n_lead,
                                    LOSS-dominated after de-overlap)
D — INCONCLUSIVE                  : no
```

---

## M. EVIDENCE CLASSIFICATION

```text
TRUSTED      : cluster/de-overlap stats (verified next>H), survival incidence,
               serial dependence ALL vs LEAD, invalidation (deterministic,
               full provenance, dataset hash matches)
INCONCLUSIVE : E_EQH / E_EQL lead stats (n=18) — very wide CIs; the direction is
               consistent (LOSS-dominated) but the size is inadequate
INVALID      : the E_EQH asymmetry in ALL events (P3.2.1 §5) — proven an overlap
               artifact, gone after de-overlap (P3.2.2 F)
SUPERSEDED   : none (P2.6/P3.1/P3.2.1 remain valid for their purpose)
```

---

## N. RECOMMENDATION FOR P3.3

```text
1. SGML_L3_EVENT as a label mechanism: KEEP as a candidate,
   WITH mandatory de-overlap (lead event per cluster, gap > 16) and the
   WIN/LOSS/UNRESOLVED/AMBIGUOUS classes (UNRESOLVED & AMBIGUOUS excluded).
2. The EXISTING EVENT UNIT (f9-confirm/EQH/EQL as event bars) is NOT suitable
   as a setup unit: 1.5% retention -> the de-overlapped dataset is too small
   and the outcome is LOSS-dominated. DO NOT retrain a model on it.
3. P3.3 must question the EVENT UNIT DEFINITION (feature/setup architecture
   revision): a truly discrete unit (onset/state transition, order block /
   breaker / FVG boundary) — outside this session's mandate, no hot-fix.
4. Horizon: 8-16 bars (setup lifetime); SL is the de-facto invalidator;
   structure-break invalidation is too slow -> not a primary outcome.
5. LSTM / regime / meta-label REMAIN UNTESTED until (a) an independent event
   unit is available and (b) the feature re-audit against the final label is done.
6. Multiple testing: 6 families x 3 H x 2 combos + survival + invalidation
   ≈ 80 statistics; only CONSISTENT patterns (LOSS>WIN for all families/all H
   after de-overlap) are used as conclusions; no cherry-picking.
```

---

## Appendix — Artifacts

```text
Script  : ml/p3/deoverlap_audit.py, ml/p3/survival_diagnostic.py
Output  : ml/p3/output/p3_2_deoverlap_h8.json / h16.json / h24.json
          ml/p3/output/p3_2_survival.json
Report  : docs/P3_2_2_DEOVERLAP_SURVIVAL.md
Checkpoint: P3_1_P3_2_1_CHECKPOINT_SHA = dc1faa92... (committed, pushed)
```

**Closing note:** P3.2.2 completed without production changes. No training,
no TP/SL/horizon/filter optimization. Core findings: the L3 label mechanism is
viable, but the existing event stream is not valid as an independent setup unit —
research must move to a truly discrete event-unit definition before discussing models.