chiki2bum2/SniperGold_ML
Watch
1
0
Fork
You've already forked SniperGold_ML
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
SniperGold_ML/docs/P3_2_2_DEOVERLAP_SURVIVAL.md

13 KiB
Raw Permalink Blame History

P3.2.2 DE-OVERLAP & SURVIVAL REPORT — SNIPERGOLD_ML

Date       : 2026-08-22
Session    : P3.2.2 — Formal Event De-Overlap + Survival/Time-to-Event Diagnostic
Status     : RESEARCH / DIAGNOSTIC ONLY — no production changes
Checkpoint : P3_1_P3_2_1_CHECKPOINT_SHA = dc1faa92f36ad22d4062315a5654965a08a006dc
Artifacts  : ml/p3/deoverlap_audit.py ; ml/p3/survival_diagnostic.py
             ml/p3/output/p3_2_deoverlap_h8.json / h16.json / h24.json
             ml/p3/output/p3_2_survival.json

0. CHECKPOINT & PROVENANCE

P2_SOURCE_SHA         : 8d330343af688e2de2b2a1d12ce19a582714709a
P2.6_SOURCE_SHA       : 8d330343... (train_p26=e510ef3b, build_features_p2=4680f905)
P3_1_P3_2_1_CHECKPOINT : dc1faa92f36ad22d4062315a5654965a08a006dc (committed & pushed)
FEATURE_CONTRACT_SHA  : C44CC6F2... (canonical)
DATASET_SHA           : e85a08616191fb7ce4bbe79c1754228ddbcf54562034a322747501c7aabe124e
symbol/tf             : XAUUSD / M15 ; 2017-01-01 .. 2026-08-20
event semantics       : existing FEATURE_CONTRACT v1.0 (f9/f7/f10/f11)
entry semantics       : close of the closed decision bar
primary combo         : TP=1.5 ATR / SL=0.75 ATR (candidate, 2:1 semantic)
sensitivity           : 1.0/1.0 (symmetric)
invalidation          : opposite structure break (f8 or f5 flip) — secondary only
scripts               : deoverlap_audit.py / survival_diagnostic.py (SHA recorded in JSON)

A. EVENT POPULATION

Family        definition (existing)                     direction   n_original
E_BUY         f9==1 & f7>0  (CHoCH confirms bull sweep)  long       27096
E_SELL        f9==1 & f7<0  (CHoCH confirms bear sweep)  short      25259
E_EQH         f10==1 (equal-high swept)                  short       877
E_EQL         f11==1 (equal-low swept)                   long        1617
E_BUY_STRICT  E_BUY & f18>=60                            long       13143
E_SELL_STRICT E_SELL & f18>=60                           short       9859

B. CLUSTER STATISTICS (overlap window H)

H family n_original clusters lead follow-on retention med_cluster max_cluster
8 E_BUY 27096 420 420 26676 1.55% 41 388
8 E_SELL 25259 391 391 24868 1.55% 43 473
16 E_BUY 27096 397 397 26699 1.47% 46 388
16 E_SELL 25259 371 371 24888 1.47% 46 473
16 E_EQH 877 18 18 859 2.05% 17 384
16 E_EQL 1617 18 18 1599 1.11% 42 504
24 E_BUY 27096 373 373 26723 1.38% 48 391
24 E_SELL 25259 357 357 24902 1.41% 47 473 
De-overlap verified: next_selected - selected > H = TRUE for all families/H.
Key finding: 97-99% of events are FOLLOW-ON within large clusters
(median cluster 46 bars, max 473). The f9-confirm event stream is HIGHLY redundant:
only ~1.5% of events are truly independent (lead).

C. DE-OVERLAP H=8 (1.5/0.75 ATR)

family ALL W/L/U LEAD W/L/U LEAD MFE/MAE
E_BUY 0.265/0.581/0.146 0.281/0.640/0.074 1.32 / 1.48
E_SELL 0.266/0.587/0.139 0.289/0.637/0.059 1.43 / 1.49
E_EQH 0.290/0.520/0.186 0.158/0.684/0.158 0.74 / 1.59
E_EQL 0.231/0.584/0.176 0.179/0.679/0.107 0.98 / 1.26

D. DE-OVERLAP H=16 (1.5/0.75 ATR) — PRIMARY

family ALL W/L/U LEAD W/L/U LEAD MFE/MAE
E_BUY 0.310/0.629/0.052 0.310/0.668/0.018 1.91 / 2.06
E_SELL 0.310/0.633/0.048 0.307/0.658/0.019 2.14 / 2.02
E_EQH 0.326/0.577/0.092 0.167/0.778/0.056 0.91 / 1.77
E_EQL 0.280/0.631/0.080 0.278/0.667/0.056 1.25 / 1.88
E_BUY_STRICT 0.317/0.623/0.050 0.304/0.659/0.031 1.68 / 1.73
E_SELL_STRICT 0.305/0.634/0.053 0.325/0.633/0.039 1.75 / 1.94

E. DE-OVERLAP H=24 (1.5/0.75 ATR)

family ALL W/L/U LEAD W/L/U
E_BUY 0.322/0.640/0.029 0.316/0.670/0.008
E_SELL 0.321/0.644/0.026 0.308/0.672/0.008
E_EQH 0.344/0.603/0.048 0.167/0.833/0.000
E_EQL 0.291/0.644/0.056 0.091/0.818/0.091

F. OUTCOME COMPARISON — ALL vs LEAD (conclusion)

1. The apparent E_EQH asymmetry (ALL: W 0.326 vs L 0.577 at H16) DISAPPEARS
   COMPLETELY in LEAD (W 0.167 vs L 0.778). The ALL asymmetry = artifact of
   repeated counting within clusters (the outcome window is shared by many events).
2. All families in LEAD: LOSS >> WIN (LOSS ~2x to 4.7x WIN).
3. De-overlap slightly raises P_LOSS for E_BUY/E_SELL (0.629->0.668) and
   REVEALS that the existing stream has no TP-before-SL edge.
4. Censoring drops quickly with H (H8 6-18% -> H16 2-6% -> H24 0-9%).

G. SURVIVAL / TIME-TO-EVENT (LEAD events, de-overlap H16, primary 1.5/0.75)

H E_BUY TP/SL/cens E_SELL TP/SL/cens E_EQH TP/SL/cens E_EQL TP/SL/cens
8 0.277/0.642/0.076 0.288/0.636/0.059 0.167/0.667/0.167 0.222/0.667/0.111
16 0.310/0.668/0.018 0.307/0.658/0.019 0.167/0.778/0.056 0.278/0.667/0.056
24 0.315/0.673/0.008 0.310/0.666/0.008 0.167/0.833/0.000 0.278/0.667/0.056
48 0.320/0.675/0.000 0.313/0.671/0.000 0.167/0.833/0.000 0.333/0.667/0.000 
Empirical setup lifetime (lead events):
  median time-to-event : 1-2 bars
  median TP time       : 2-3 bars (E_BUY), 2 bars (E_SELL)
  median SL time       : 1-2 bars  (SL is almost always faster than TP)
  P(SL) asymptote ~0.67 ; P(TP) asymptote ~0.31-0.32 (E_BUY/E_SELL)
  ~90% of outcomes occur within 8 bars; >98% within 16 bars.
Conclusion: setups "live" 1-8 bars; SL wins ~2:1. The 24-bar horizon (old label)
far exceeds the setup lifetime and is dominated by late SL (not new information).

H. MFE / MAE CONDITIONAL (lead events, H=48, primary)

family status n MFE med MAE med MFE/MAE ratio
E_BUY WIN 127 5.37 2.52 2.53
E_BUY LOSS 268 3.01 4.46 0.66
E_SELL WIN 116 5.15 1.92 2.83
E_SELL LOSS 249 3.04 4.66 0.58
E_EQH WIN 3 5.20 0.11 53.5 (small n)
E_EQH LOSS 15 1.47 2.57 0.48
E_EQL WIN 6 2.81 3.43 1.50
E_EQL LOSS 12 2.25 3.32 0.72 
- WIN events: MFE/MAE ratio 2.5-2.8 (favorable excursion far larger than
  adverse excursion) — BUT this is post-hoc (conditioned on outcome), not a
  structure exploitable without look-ahead.
- LOSS events: median MAE 4.5-4.7 ATR within 48 bars -> after the 0.75 ATR SL is
  touched (bars 1-2), price keeps moving against by ~4.5 ATR on average.
  Consistent with a small SL + trend continuing against (late entry).
- No favorable MFE/MAE asymmetry at the unconditional level:
  median MFE ≈ median MAE for all families (see P3.2.1 §4; confirmed in LEAD).

I. INVALIDATION (secondary outcome; opposite structure break f8/f5)

family P(inv within 48 bars) P(inv BEFORE TP/SL) P(TP first) P(SL first) median t_inv
E_BUY 0.73 0.003 0.32 0.66 24 bars
E_SELL 0.74 0.000 0.31 0.66 22 bars
E_EQH 1.00 0.333 0.00 0.06 1 bar
E_EQL (small n) — — — — 
- E_BUY/E_SELL: invalidation (structure break) almost NEVER precedes
  TP/SL (0.0-0.3%) because the outcome occurs within 1-3 bars while invalidation
  has a median of 22-24 bars. -> structure-break invalidation is NOT useful as a
  primary outcome for this stream; SL is far faster.
- E_EQH: invalidation median 1 bar & 100% within 48 bars -> the short direction
  after an equal-high sweep is immediately opposed by structure; consistent with
  LOSS dominance.
- Conclusion: for existing event units, SL (price) is the de-facto invalidator;
  the opposite structure break is too slow to be a primary label.

J. SERIAL DEPENDENCE — ALL vs LEAD (H=16, primary, outcome WIN/LOSS)

family set n_resolved ac1 P(same outcome) mean run max run
E_BUY ALL 25451 0.498 0.778 4.5 64
E_BUY LEAD 388 -0.025 0.556 2.2 11
E_SELL ALL 23813 0.501 0.780 4.5 49
E_SELL LEAD 371 0.016 0.574 2.3 17
E_EQH ALL 792 0.466 0.753 4.0 25
E_EQH LEAD (n<50) — — — — 
De-overlap REMOVES outcome serial dependence:
  lag-1 autocorrelation 0.50 -> ~0.0 ; P(same outcome) 0.78 -> 0.56 ;
  mean run 4.5 -> 2.2. Independence SATISFIES the basic requirement for ML labels.
Note: P_same_dir_consec = 1.0 everywhere (single direction per family) — that
metric is only meaningful when families mix directions; not used for conclusions.

K. LABEL CONTRACT ASSESSMENT — SGML_L3_EVENT (candidate)

Criterion Assessment Evidence
Causal PASS outcome measured from the entry bar forward, no look-ahead
SMC aligned PASS unit = SMC event bar; outcome = TP-before-SL (setup validation)
Execution realistic PASS entry = close of the decision bar (B ≈ A, Δ 0.008 ATR)
Independent PARTIAL achieved only AFTER mandatory de-overlap (ac1 0.5->0); without de-overlap it FAILS completely
Ambiguity PASS AMBIGUOUS 0.5-1.6% (low; excluded)
Censoring PASS H16 2%; H8 6-8%; UNRESOLVED excluded, not forced to LOSS
Economically interpretable PASS WIN/LOSS = first TP/SL; semantic 2:1 ratio
Suitable for meta-labeling PASS (mechanism) binary, independent, low ambiguity/censoring
Unit validity FAIL f9-confirm event stream 97-99% redundant; leads only ~1.5%; n_lead 397/371/18/18 
The SGML_L3_EVENT label mechanism = technically VIABLE.
THE EVENT UNIT (existing stream) = NOT VALID as an independent setup unit:
too dense/redundant; after de-overlap the remaining sample is very small
(~40 leads/year/family) and the outcome is LOSS-dominated (SL ~2x TP).

L. CANDIDATE LABEL STATUS

C — EVENT STREAM ITSELF INVALID
    (as an independent setup unit for ML labels)

With the combined note:

A — EVENT LABEL VIABLE            : PARTIAL (L3 label mechanism viable;
                                    independence/ambiguity/censoring PASS)
B — EVENT LABEL VIABLE WITH CENSORING : NOT REQUIRED for H16+ (2% censoring);
                                    survival-style useful for H<=8 / time-to-event
C — EVENT STREAM ITSELF INVALID   : STRONGLY SUPPORTED (retention 1.5%, small n_lead,
                                    LOSS-dominated after de-overlap)
D — INCONCLUSIVE                  : no

M. EVIDENCE CLASSIFICATION

TRUSTED      : cluster/de-overlap stats (verified next>H), survival incidence,
               serial dependence ALL vs LEAD, invalidation (deterministic,
               full provenance, dataset hash matches)
INCONCLUSIVE : E_EQH / E_EQL lead stats (n=18) — very wide CIs; the direction is
               consistent (LOSS-dominated) but the size is inadequate
INVALID      : the E_EQH asymmetry in ALL events (P3.2.1 §5) — proven an overlap
               artifact, gone after de-overlap (P3.2.2 F)
SUPERSEDED   : none (P2.6/P3.1/P3.2.1 remain valid for their purpose)

N. RECOMMENDATION FOR P3.3

1. SGML_L3_EVENT as a label mechanism: KEEP as a candidate,
   WITH mandatory de-overlap (lead event per cluster, gap > 16) and the
   WIN/LOSS/UNRESOLVED/AMBIGUOUS classes (UNRESOLVED & AMBIGUOUS excluded).
2. The EXISTING EVENT UNIT (f9-confirm/EQH/EQL as event bars) is NOT suitable
   as a setup unit: 1.5% retention -> the de-overlapped dataset is too small
   and the outcome is LOSS-dominated. DO NOT retrain a model on it.
3. P3.3 must question the EVENT UNIT DEFINITION (feature/setup architecture
   revision): a truly discrete unit (onset/state transition, order block /
   breaker / FVG boundary) — outside this session's mandate, no hot-fix.
4. Horizon: 8-16 bars (setup lifetime); SL is the de-facto invalidator;
   structure-break invalidation is too slow -> not a primary outcome.
5. LSTM / regime / meta-label REMAIN UNTESTED until (a) an independent event
   unit is available and (b) the feature re-audit against the final label is done.
6. Multiple testing: 6 families x 3 H x 2 combos + survival + invalidation
   ≈ 80 statistics; only CONSISTENT patterns (LOSS>WIN for all families/all H
   after de-overlap) are used as conclusions; no cherry-picking.

Appendix — Artifacts

Script  : ml/p3/deoverlap_audit.py, ml/p3/survival_diagnostic.py
Output  : ml/p3/output/p3_2_deoverlap_h8.json / h16.json / h24.json
          ml/p3/output/p3_2_survival.json
Report  : docs/P3_2_2_DEOVERLAP_SURVIVAL.md
Checkpoint: P3_1_P3_2_1_CHECKPOINT_SHA = dc1faa92... (committed, pushed)

Closing note: P3.2.2 completed without production changes. No training, no TP/SL/horizon/filter optimization. Core findings: the L3 label mechanism is viable, but the existing event stream is not valid as an independent setup unit — research must move to a truly discrete event-unit definition before discussing models.