SniperGold_ML/docs/P3_S3_CHOCH_MSS_CONFORMANCE.md

# P3-S.3 CHOCH / MSS FORMAL SPECIFICATION & CODE-CONFORMANCE REPORT

```text
Date       : 2026-08-22
Session    : P3-S.3 — Formal SMC Specification + Code-Conformance Audit (CHoCH/MSS)
Scope      : f8 (choch_dir) / f9 (choch_confirms) / f18 (CHoCH confluence leg)
             + legacy signal chain (InpRequireCHoCH / InpSeqWindow)
             + Engine 2 (AF_DetectChoch) + Display (AF_BuildStructLines)
Principle  : SPECIFICATION FIRST, CODE AUDIT SECOND
Status     : NO CODE CHANGES (Phase D) — all findings DOCUMENTED
Artifacts  : docs/SMC_CHOCH_MSS_SPEC_v1.md (specification)
             ml/p3/smc_semantic/spec_tests_choch_mss.py (+ .json) (tests)
             ml/p3/smc_semantic/output/spec_tests_choch_mss_report.json
             ml/p3/smc_semantic/output/audit_choch_state.json (forensic)
             ml/p3/smc_semantic/output/parity_choch_pivot_strict_vs_equal.json
```

---

## 0. CHECKPOINT & PROVENANCE (session-start verification)

```text
Forge HEAD          : 595b4c9a622703b05228bd563b4979fef81a2ebf (P3-S.2)  VERIFIED
Research checkpoint : c7a2b3f5e6acbf8deb748315dcbcc200046d0b84           VERIFIED
Branch              : main                                                VERIFIED
Remote              : forge.mql5.io/chiki2bum2/SniperGold_ML.git         VERIFIED
Working tree        : CLEAN at start                                      VERIFIED
P3-S.2 artifacts    : spec/conformance/tests/report/handover INTACT        VERIFIED
HUMAN VERIFICATION  : CANCELLED (human annotation is NOT a validation gate)  VERIFIED
```

---

## A. SOURCE MATERIAL (Phase A — inventory & reconciliation)

| # | Source | Location | Role |
|---|--------|----------|------|
| 1 | FEATURE_CONTRACT v1.0 (f8/f9/f18) | docs/FEATURE_CONTRACT.md | Feature contract: f8 "last CHoCH direction (internal structure)"; f9 "CHoCH confirms sweep"; f18 +15 when chochDir==swpDir |
| 2 | DESIGN.md (Algo Forge) | docs/DESIGN.md | Design intent: "CHoCH/MSS" element of agent N (structure); E (confirmation); synonym labels |
| 3 | DESAIN_MTF_v45.md | Shared Projects\SniperGold_ML | v4.5 per-TF structure (BOS/CHoCH) + cache; ML features stay M15; AUC decisions only for f10/f11 |
| 4 | v4.3/v4.4/v4.5 ProcessStructure | Indicators\Downloads\SniperGold_SMC_ProPlus_v4_3/4_4/4_5.mq5 | Legacy CHoCH semantics: internal break + InpConfluenceFilter + InpSeqWindow signal gate |
| 5 | EA ProcessStructure (current) | Experts\AlgoForge_Backtest_Baseline.mq5:294-340 | f8/f9 runtime implementation (v4.4 copy) |
| 6 | build_features_p2.py | ml/parity/build_features_p2.py | Training side (build_structure_fast = equal-allowed pivots, parity-verified) |
| 7 | train_model.py build_structure | ml/train_model.py:161-229 | Streaming ProcessStructure replica (STRICT pivots: equal NOT allowed) |
| 8 | smc_semantic_common.py | ml/p3/smc_semantic/ | P3 forensic infra (uses TM.build_structure -> strict pivots) |
| 9 | AF_DetectChoch / AF_DetectBos | Include\AlgoForge\AF_Engine2_Agents.mqh:238-256 | Second CHoCH definition (Engine 2: fractal 2/2, no gate, per-TF) |
| 10 | AF_BuildStructLines | Include\AlgoForge\AF_Engine2_Display.mqh:137-192 | Third CHoCH definition (display: fractal 2/2, CHoCH/BOS labels) |
| 11 | P3-S.0 f7 forensic | docs/P3_S_F7_EVENT_LIFECYCLE_FORENSIC.md | State/event bug pattern + lifecycle fix (CHoCH comparison) |
| 12 | P3-S.2 spec/conformance | docs/SMC_LIQUIDITY_SWEEP_SPEC_v1.md, docs/P3_S2_... | Protocol, BUG-P3S2-001 (f9/f18 stale sweep), next gate CHoCH/MSS |
| 13 | Golden dataset doc | docs/P3_SMC_SEMANTIC_GOLDEN_DATASET.md | SMC concept list (Liquidity Sweep | CHoCH/MSS | BOS | FVG | OB); event/state findings |
| 14 | audit_choch_state.json (NEW) | ml/p3/smc_semantic/output/ | Measured CHoCH state behavior (this session) |
| 15 | parity_choch_pivot_strict_vs_equal.json (NEW) | ml/p3/smc_semantic/output/ | Strict vs equal pivot divergence (this session) |

**Three-layer semantic reconciliation:**

```text
Intended semantics  : CHoCH = internal structure reversal (close break of the last
                      internal swing + opposite prior trend + non-opposing swing
                      gate); CHoCH/MSS = synonym labels (DESIGN.md); Sweep->CHoCH->
                      OB/FVG chain within InpSeqWindow (40); a separate MSS
                      (break+displacement) NOT defined.
Legacy semantics    : trigger identical; the signal gate applies freshness
                      (curBar-chochBar)<=40; ML features f8/f9/f18 use a PERSISTENT
                      state without expiry (last CHoCH).
Current semantics   : trigger identical; f8 = last CHoCH (persistent, per contract);
                      f9/f18 = raw-state consumption WITHOUT expiry (stale 34.7%/58%);
                      Engine 2 AF_DetectChoch = second definition (fractal 2/2,
                      no gate); display = third definition (fractal 2/2).
```

---

## B. PROJECT SEMANTIC SPECIFICATION (Phase B)

Full document: **docs/SMC_CHOCH_MSS_SPEC_v1.md** (PROJECT SEMANTIC SPECIFICATION — not "universal SMC ground truth").

Spec decision summary:

```text
S-1  Structure source  : internal swing pivot fractal len=5 (equal allowed)
                         for f8/f9; swing len=50 only as the gate; Engine 2 fractal 2/2.
S-2  Swing definition  : fractal 5/5, confirmation p+5, window [max(95,t-604),t-5],
                         equal allowed, no height threshold.
S-3  CHoCH             : bullish: close[b] > last internal pivot high, prior
                         internal trend < 0, SwingTrendAt(b) >= 0.
                         bearish: close[b] < last internal pivot low, prior
                         internal trend > 0, SwingTrendAt(b) <= 0.
S-4  MSS               : SAME AS CHOCH (project synonym label); no separate
                         primitive; the MSS+displacement definition NOT adopted.
S-5  Direction         : +1 bullish (reversal from down), -1 bearish, 0 none.
S-6  Break semantics   : CLOSE-CONFIRMED (wick-only insufficient; body not
                         required; displacement not required).
S-7  Precondition      : prior internal trend != 0; non-opposing swing gate;
                         NO explicit HH/HL/LH/LL sequence.
S-8  Temporal          : closed bar t; break b <= t; pivot p+5 <= b; no future.
S-9  Event vs State    : EVENT (detection) + STATE (f8 last dir) + VALIDITY WINDOW
                         40 for chain consumers (f9/f18/signal gate).
S-10 Lifecycle         : NO_CHOCH / ONSET / ACTIVE / EXPIRED / SUPERSEDED;
                         INVALIDATED & CONSUMED not modeled.
S-11 Timeframe         : f8/f9 = M15 only; Engine 2 per-TF; canonical project =
                         SPECIFICATION AMBIGUOUS (MTF).
S-12 Sweep relation    : detection independent; the consumption chain (f9/gate)
                         uses ordering + freshness.
```

---

## C. FORMAL DEFINITIONS (Phase B — extract)

```text
CHoCH_event(internal structure S, bar t) ->
  exists internal pivot P=(p, L, isHigh) with p in [max(95,t-604), t-5],
  p+5 <= b <= t, and P is the last pivot of that side before b, such that:

  Bullish CHoCH (dir=+1):
    isHigh(P) AND close[b] > L AND priorInternalTrend(b) < 0
    AND SwingTrendAt(b) >= 0
  Bearish CHoCH (dir=-1):
    !isHigh(P) AND close[b] < L AND priorInternalTrend(b) > 0
    AND SwingTrendAt(b) <= 0

  priorInternalTrend(b) = direction of the last internal break < b (from the
  structure engine; 0 if no break yet) — the first window break = BOS, not CHoCH.

  One structure transition = one event (bar b). The f8 state = direction of the
  LAST event; chain consumers (f9/f18) are valid only while (t - b) <= SEQ_WINDOW (40).
```

---

## D. CURRENT CODE BEHAVIOR (Phase C — code audit)

### D.1 Structure process (EA / v4.4 / v4.5 — identical)

```mql5
// ProcessStructure(len=5, internal=true) — EA:294-340
for i in [begin, total):              // begin = max(100, total-600)
  p = i-len
  if p>=len:
     if IsPivotHigh(high,p,len): upTarget=high[p]; upBar=p;      // fractal 5/5, equal allowed
     if IsPivotLow (low, p,len): dnTarget=low[p];  dnBar=p;
  // bullish break
  if upBar>=0 && close[i]>upTarget:
     choch = (trend<0)
     allow = SwingTrendAt(i)>=0                      // InpConfluenceFilter=true
     trend = 1
     if internal: g_inTrend=1; g_inLastBreak=i;
        if allow && choch: g_chochDir=1; g_chochBar=i;    // f8 event
  // bearish break
  if dnBar>=0 && close[i]<dnTarget:
     choch = (trend>0)
     allow = SwingTrendAt(i)<=0
     trend = -1
     if internal: g_inTrend=-1; g_inLastBreak=i;
        if allow && choch: g_chochDir=-1; g_chochBar=i;
```

Emission (ComputeMLFeatures EA:487,500-501,522):
```mql5
chochOK  = (g_chochDir!=0 && g_chochBar>=g_swpBar && g_chochDir==g_swpDir); // f9, NO expiry
g_feat[8]= g_chochDir;                                                      // f8, last state
g_feat[9]= chochOK?1.0:0.0;
if(g_chochDir!=0 && g_chochDir==g_swpDir) cb+=15;                           // f18, NO expiry
```

### D.2 Measured behavior (real XAUUSD M15 2017-2026, n=197.032)

```text
CHoCH onsets (new events)          : 2.197
active bars (f8 != 0)              : 196.817 (99.89%)
repetition ratio                   : 89.6x   (f7 = 69.5x)
state run median / max             : 318 / 2.187 bars
state age (r - chochBar)           : median 49, p99 242, max 418
state returns to 0 after event 1   : NEVER (same as f7)

f9 chochOK TRUE total              : 51.105 bars
  - fresh (CHoCH age <= 40)        : 33.391
  - STALE (age > 40)               : 17.714  (34.7% of f9 TRUE)
f18 +15 leg (chochDir==swpDir)     : 97.826 bars
  - STALE (age > 40)               : 56.761  (58.0% of the leg)
```

Runtime-training pivot divergence (parity diagnostic):
```text
strict pivots (train_model.build_structure, equal NOT allowed) vs
equal-allowed (build_features_p2.build_structure_fast == EA):
  choch_dir mismatch : 512 bars (0.26%)
  choch_bar mismatch : 4.722 bars (2.40%)
  onsets             : 2.165 (strict) vs 2.197 (equal)
```

---

## E. CONFORMANCE MATRIX (Phase C)

Verdicts: CONFORMING / PARTIALLY CONFORMING / NON-CONFORMING / UNVERIFIABLE /
SPECIFICATION AMBIGUOUS.

| # | Requirement | Specification | Code (Function/Line) | Actual Behavior | Verdict |
|---|-------------|--------------|----------------------|-----------------|---------|
| E-1 | Structure source | Internal pivot len=5, equal allowed (S-1) | `ProcessStructure(INTERNAL_LEN=5,internal=true)` EA:294-340; `AddInternalPivot` EA:181-184 | g_ip = fractal 5/5; swing 50 only as gate (SwingTrendAt) | **CONFORMING** |
| E-2 | Swing definition | fractal 5/5, equal allowed, confirm p+5, window [max(95,t-604),t-5] (S-2) | `IsPivotHigh/Low` EA:157-169; `begin=max(100,t-599)` | equal highs/lows do NOT cancel; confirm p+5; window matches | **CONFORMING** |
| E-3 | CHoCH trigger | close > last internal pivot high (bull) / < last pivot low (bear) (S-3) | EA:316-324 (bull), 326-335 (bear) | `close[i]>upTarget` / `close[i]<dnTarget`; target = last pivot | **CONFORMING** |
| E-4 | Break semantics | CLOSE-CONFIRMED (S-6) | EA:316,326 | close (not wick) vs level; wick-only insufficient | **CONFORMING** |
| E-5 | Direction | +1 bullish / -1 bearish (S-5) | EA:322,332 (`g_chochDir=1/-1`) | matches the reversal definition | **CONFORMING** |
| E-6 | Precondition | prior internal trend != 0 (S-7) | `choch=(trend<0)/(trend>0)` EA:318,328 | first break (trend=0) = BOS, not CHoCH | **CONFORMING** |
| E-7 | Confluence gate | non-opposing SwingTrendAt (S-7/A-4) | `allow = SwingTrendAt(i)>=0/<=0` EA:319,329 | WEAK gate (>=0/<=0) vs the "align" comment (>=1/<=-1); weak/strong intent A-4 | **PARTIALLY CONFORMING** (weak-gate ambiguity) |
| E-8 | Closed-bar / no future | all inputs <= t (S-8) | Engine-1 closed-bar; ComputeAll; CH-T14 | no future visibility | **CONFORMING** |
| E-9 | Detector lifecycle | ONE event per transition (S-9/S-10) | ProcessStructure: g_chochDir set only at the break bar; full recompute per bar | detector = "last event" state; onsets 2.197 (no duplicate emission per se — state, not a stream) | **CONFORMING** (detector) |
| E-10 | f8 emission | last CHoCH direction, persistent per contract (S-9) | `g_feat[8]=g_chochDir` EA:500 | last-event state; age p99 242 bars — MATCHES the contract | **CONFORMING** |
| E-11 | f9/f18 lifecycle | VALIDITY WINDOW 40 for chain consumers (S-9/S-10) | f9 EA:487; f18 EA:522 — NO expiry | stale 34.7% (f9) / 58.0% (f18 leg) — a stale event stays confirm/+15 | **NON-CONFORMING** (BUG-P3S3-001) |
| E-12 | Timeframe | M15 only for f8/f9 (S-11) | EA M15 slot; training M15 | M15 explicit; HTF only bias | **CONFORMING** |
| E-13 | Runtime-training parity | equal semantics runtime==training (S-1/S-2) | build_features_p2 `_pivots_vec`/`build_structure_fast` == EA (P2.5 parity 1e-9); train_model.build_structure & smc_semantic_common = STRICT | parity feed matches; train_model/P3-infra diverge (0.26% choch_dir; 2.40% choch_bar) | **PARTIALLY CONFORMING** (BUG-P3S3-002) |
| E-14 | MSS distinct concept | NO MSS primitive (S-4) | no MSS code anywhere (grep 0 matches) | no code claims a separate MSS | **CONFORMING** (no claim) / label AMBIGUOUS (A-1) |
| E-15 | Engine 2 AF_DetectChoch | second definition: fractal 2/2, no gate, per-TF (S-1, spec §4.2) | AF_Engine2_Agents.mqh:238-247 (BuildSwing 196-230, TrendFromSwing 232-248); N agent 444-449; E agent 605-608 | close break vs highs[0]/lows[0] with the 4-swing trend; NO gate/displacement; per-slot TF | **CONFORMING** to the Engine-2 definition; AMBIGUOUS at the project level (two CHoCH definitions, A-2) |
| E-16 | Same-bar precedence (bull & bear break) | A-3: SPECIFICATION AMBIGUOUS | EA:316 & 326 both ifs (not else-if) | bullish executes first; unreachable with consistent data (analogous to P3-S.2 E-11) | **SPECIFICATION AMBIGUOUS** (unreachable) |

---

## F. CONFIRMED BUGS (Phase D — DOCUMENT ONLY, no code changes)

```text
BUG-P3S3-001
  SEVERITY     : MEDIUM
  SOURCE       : f9 (chochOK) & f18 (confluence leg) — EA:487, 522 (v4.4:951,986;
                 v4.5:920,938); training build_features_p2 choch_ok
  SPEC REQ     : S-9/S-10 — event validity (SEQ_WINDOW=40) applies to chain
                 consumers (f9/f18/signal gate); the detector is already correct
  ACTUAL CODE  : f9/f18 use raw g_chochDir/g_chochBar, WITHOUT
                 (t - g_chochBar) <= 40
  IMPACT       : an EXPIRED CHoCH event stays "confirmed" in f9 (34.7% of f9-TRUE
                 bars stale, measured 17.714 bars) and keeps +15 in f18 (58.0% of
                 the leg stale, 56.761 bars). The semantic chain is inconsistent:
                 the v4.4 signal gate applies freshness, the ML features do not.
                 Same bug class as BUG-P3S2-001 (sweep side) — P3-S.0 only fixed f7,
                 f9/f18 not yet.
  PROPOSED FIX : apply (t - g_chochBar) <= SEQ_WINDOW at f9/f18 OR refactor the
                 CHoCH event into an onset object with a single validity lifetime
                 (one lifecycle source of truth for sweep & choch).

BUG-P3S3-002
  SEVERITY     : LOW
  SOURCE       : train_model.build_structure (STRICT pivots: equal NOT allowed)
                 vs EA/parity feed (equal allowed)
  SPEC REQ     : S-2 — equal-level handling consistent runtime-training
  ACTUAL CODE  : MQL5 IsPivotHigh: `if(h[idx-k]>v) return false` (equal OK);
                 Python _pivots_vec: `h >= np.roll(...)` (equal OK);
                 Python train_model.build_structure: `if h[p]<=h[p-k] ...` (equal FAIL)
                 -> build_structure_fast & EA match; train_model diverges.
  IMPACT       : choch_dir mismatch 512 bars (0.26%); choch_bar 4.722 bars (2.40%);
                 onsets 2.165 vs 2.197 (XAUUSD M15 2017-2026). smc_semantic_common
                 (P3 forensic infra) uses train_model -> forensic results differ
                 slightly from runtime on equal-level bars.
  PROPOSED FIX : align the pivot predicate (use _pivots_vec/build_structure_fast on
                 all training & forensic paths), or document the tolerance.

BUG-P3S3-003
  SEVERITY     : INFO (documentation)
  SOURCE       : InpConfluenceFilter comment "Internal must align with swing trend"
                 vs the implementation SwingTrendAt(i)>=0/<=0
  SPEC REQ     : S-7/A-4 — the confluence gate must be explicit (weak vs strong)
  ACTUAL CODE  : gate = "swing NON-opposing" (>=0/<=0), NOT "aligned" (>=1/<=-1)
  IMPACT       : CHoCH is still recorded when the swing is flat (trend 0); the input
                 comment is misleading; spec A-4 flags AMBIGUOUS
  PROPOSED FIX : revise the comment/documentation; adjudicate weak vs strong gate
```

---

## G. SEMANTIC AMBIGUITIES (Phase F — consistency)

```text
G-1  MSS vs CHOCH (TERMINOLOGY)
     The project uses "CHoCH/MSS" as synonym labels (DESIGN.md, P3 concept list).
     No separate MSS primitive (0 code matches). If "MSS" means break+displacement
     (common SMC definition), the project does NOT define it.
     Status: SPECIFICATION AMBIGUOUS (label) — but no code to audit.

G-2  TWO CHOCH DEFINITIONS IN ONE PROJECT
     f8/f9 (fractal 5/5 + swing gate, M15) vs Engine-2 AF_DetectChoch (fractal 2/2,
     no gate, per-TF H4/M30/M15/M3) vs Display (fractal 2/2, chart TF). All three
     agree on direction (reversal) but differ in parameters & preconditions.
     For ML features, the f8/f9 definition is authoritative.

G-3  WEAK CONFLUENCE GATE
     SwingTrendAt(b)>=0/<=0 (non-opposing) vs "align" (aligned). A-4.

G-4  CANONICAL TIMEFRAME
     f8/f9 M15 explicit; the HTF-structure role on LTF events not established.
     Same as sweep G-3 (P3-S.2).

G-5  EVENT vs STATE IN CONSUMERS
     f8 = state (per contract); f9/f18 should be event+window but use the state
     (BUG-P3S3-001). The lifecycle consistency is not yet complete in the project.
```

---

## H. SEMANTIC RELATIONSHIP WITH LIQUIDITY SWEEP (Phase C)

```text
Detection : CHoCH f8 STANDS ALONE (no sweep required) — CONFORMING S-12.
Consumption: f9 = chain (chochBar>=swpBar & same direction) — CONFORMING for
             ordering; freshness NOT applied (BUG-P3S3-001) — NON-CONFORMING.
Gate      : v4.4/v4.5 signal gate uses InpRequireSweep+InpRequireCHoCH with the
            40-bar window — CONFORMING to the legacy signal semantics.
Engine 2  : E agent ZONE + CONFIRMATION (sweep OR choch) — consistent with DESIGN.md.

NO detection dependency; f7 NOT changed to satisfy CHoCH (meets §17 instructions).
```

---

## I. RUNTIME/TRAINING PARITY (Phase C — diagnostic, not a retrain)

```text
f8/f9 (choch_dir/choch_confirms):
  parity feed build_features_p2 (build_structure_fast, equal allowed) vs EA:
  PARITY VERIFIED IN P2.5 (eps 1e-9, f0-f5/f7-f12/f18).
  THE DEPLOYED-MODEL PATH (P2.6) = build_features_p2 -> runtime MATCHES.
train_model.build_structure (old path/P3 infra): STRICT pivots -> diverges
  0.26% choch_dir / 2.40% choch_bar (BUG-P3S3-002) — documented.
NO retraining; the parity results are diagnostic only.
```

---

## J. VERDICT

```text
PARTIALLY CONFORMING

  Detection semantics (E-1..E-10, E-12)  : CONFORMING
  Confluence gate (E-7)                   : PARTIALLY CONFORMING (A-4)
  Event-consumer lifecycle (E-11)         : NON-CONFORMING (BUG-P3S3-001)
  Runtime-training pivots (E-13)          : PARTIALLY CONFORMING (BUG-P3S3-002)
  MSS distinct concept (E-14)             : NO CLAIM (synonym label)
  Engine 2 / Display (E-15)               : CONFORMING to its definition; project AMBIGUOUS
  Overall implementation                  : PARTIALLY CONFORMING

Verdict = conformance to the PROJECT SEMANTIC SPECIFICATION, NOT profitability/edge.
DO NOT replace with PASS just because the synthetic tests pass.
```

---

## K. NEXT GATE

```text
After the CHoCH/MSS verdict (per brief §22):
  FVG -> OB -> Displacement -> MTF Alignment -> Candidate Setup.
DO NOT return to ML; DO NOT build a meta-label; DO NOT create new event labels.
```

---

## L. CHECKPOINT & PROVENANCE (P3-S.3)

```text
Forge HEAD (session start)   : 595b4c9a (P3-S.2) — VERIFIED
P3-S.2 spec (immutable)      : docs/SMC_LIQUIDITY_SWEEP_SPEC_v1.md
FEATURE_CONTRACT             : C44CC6F2... (P2-era) / 7b908b12... (committed)
symbol/tf                    : XAUUSD / M15 (2017-01-01 .. 2026-08-20)
Human verification           : CANCELLED
No production change         : AF_Engine2_Agents.mqh / f8 / f9 / f18 / ProcessStructure
                               / v4.4 / v4.5 / Engine 2 / Display / ML : NOT touched
```