Warrior_EA/Meta_Labeling_Design.md

# Meta-Labeling: the NN predicts trade quality, not market direction

**Status: DESIGN — agreed 2026-08-12, implementation staged below.**

## Why this target

Per-bar direction of one instrument from its own H1 history is measured dead three
detector-independent ways (MI noise floor, lag profile 0–20, post-backprop-fix retrain at
chance vs a positive control that finds volatility at 4x its null). No topology change can
extract information that is not in the input-target pair. Every credible "NN works on
markets" result changes the **target** or the **data** instead. This design changes the
target; cross-sectional pooled training (change the data) is the follow-up lever, and a
quantile head can ride along on the retrain either forces.

The new question the NN answers: **given that a specific setup just fired, will THIS
instance reach its target before its stop, at the EA's own geometry, net of cost?**
That is a well-posed supervised problem where per-bar direction never was:

- the label is the realised outcome of an actual trade shape, not a synthetic class;
- the base rate is the setup's win rate (~40–65%), not a 1% tail;
- part of the answer is PROVEN predictable (range at 4x null; spread/vol/session state),
  because "does the target get hit first" is partly a volatility-geometry question;
- every labeled example is a moment someone would actually consider trading.

## Architecture

```
classic patterns (26 x Buy/Sell)  ->  candidate: (side, pattern, geometry)      [WHEN]
                                          |
excursion head (proven)           ->  SL/TP geometry for the candidate          [SHAPE]
                                          |
META-HEAD (this design)           ->  P(win | candidate, market state)          [WHETHER]
                                          |
vote = f(P - break-even)          ->  existing threshold / MM / risk plumbing   [HOW MUCH]
```

- **Primaries**: the existing classic signals. They are NOT assumed to carry edge — they
  define when a candidate exists. SQX remains the user's separate rule factory; nothing
  here parses or reverse-engineers SQX. More primaries (e.g. the user's Wyckoff blocks)
  are added only after the meta-head demonstrates skill on the current 52.
- **One net for all patterns**, pattern identity as input features — shares representation
  across patterns, and event-rare patterns borrow strength from state-common ones.
- **Integration point**: the per-side pattern slots added in `652bf81`
  (`GetActivePatternLong/Short` + `LastNetVote`) are exactly the hook: the meta-signal
  reads which candidates fired this bar and emits a gating vote for that side. The AI
  filter's global direction vote (closed verdict) is what it replaces.
- **Unchanged**: excursion head, expectancy stop, risk budget, the DB as objective logger
  with its decision layer (the DB keeps ranking primaries; the meta-head is a second,
  richer consumer of the same "did this instance pay" question).

## Training

- **Sample**: bars where at least one pattern fired, found by sweeping the classic ladders
  over training history (the ladders are already index-based; a historical sweep is cheap).
  One training row per (bar, fired pattern, side).
- **Label**: triple-barrier outcome for the proposed side at the EA's own SL/TP geometry —
  the existing labeler (`b4a704d` line), evaluated from the fire bar. Win=1 / loss-or-
  timeout=0. Same horizon rules as current training.
- **Features**: the existing `BuildFeatureWindow()` output (unchanged front end), plus a
  setup descriptor appended at the head: pattern one-hot (26), side, the filter's own
  netVote, SL and TP distances in ATR, spread/ATR at fire time.
- **Head**: binary logistic with the existing logit prior-correction (`f2ec1ed`), trained
  with the existing optimizer/batch-norm/mini-batch stack — all of that machinery is
  target-agnostic.
- **Checkpoint selection**: the existing operating-point objective, coverage x
  (precision − break-even) on the purged calibration slice (`2189316`, `983a6a3`) — it was
  built for exactly this shape of question.
- **Deploy gate**: family-wise, as always — the final deploy decision is best-of-N over
  eras and must clear the null of the maximum (`1df3054` lesson). No exceptions because
  the idea is likeable.

## The honest risk, stated up front

If the primaries carry zero conditional structure, the meta-head degenerates into a
volatility/cost/session timer. That is the FLOOR, not failure — "skip candidates whose
geometry can't pay in this vol/spread state" is real risk control — but it is bounded and
must not be sold as entry edge. The experiment's upside case is that sparse conditional
structure exists at setup moments that the dense per-bar tests could not see (they measure
marginal per-bar information; a signal alive on 3% of bars is invisible to them by
construction). The gate decides which case we are in; nothing else does.

## Staged implementation (each stage compiles and is testable alone)

1. **S1 — corpus from the signal DB** (implemented; supersedes the original "training-time
   ladder sweep"): the per-side journaling built in `652bf81`/`195be20` means the DB already
   IS the candidate stream — every instance the live ladders fire, both sides, uncensored,
   with netVote and entry price. Reading it (`Expert\AIBase\MetaCorpus.mqh`) instead of
   re-implementing 26 ladder conditions in training code eliminates the silent-divergence
   trap outright: the corpus is by construction identical to live behaviour. Costs accepted:
   corpus coverage = whatever backtest populated the DB (a corpus build is a tester run with
   `UseDatabaseRanking=true` and the new `DB_MaxRowsPerTable` input raised, e.g. 20000, so a
   15–20 year run isn't pruned), and sampling is one candidate per fire-stretch (the correct
   dedup for training anyway). The `VerboseMode` OnInit report prints corpus volume per
   family, closed fraction, and the measured GMT→server bar-offset match table that S2's
   label plumbing pins to.
2. **S2 — meta-head + label plumbing** (implemented 2026-08-13): `Signals\SignalMETA.mqh`
   (`AIType = AI_META`) + meta seams through the existing training machinery. What shipped,
   including the four deviations from the sketch above and why:
   - **2-output softmax head, not a separate logistic path**: a 2-class softmax+CE is
     mathematically a logistic/BCE head, so the `total==3` host-side gradient in
     `AI\Impl\NetForward.mqh` was generalized to `2||3` (both backprop paths; zero backend
     changes) instead of growing a parallel loss.
   - **Descriptor at the INPUT, not "at the head"**: CNet has no concat layer. On the S2
     MLP front end input-append is equivalent up to one linear layer. 31 features:
     26-slot pattern one-hot, side, tanh(netVote/20), SL/TP in ATR, spread/ATR at fire.
     Conv/LSTM meta variants would need whole-pseudo-bar padding — out of S2 scope.
   - **No prior correction**: the logit adjustment is 3-class-shaped and exists for a ~6%
     tail; the meta base rate is ~40% and the operating-point fit carries calibration.
   - **Trained on the live chart like every other model** (not "offline in the tester" —
     the tester is inference-only by architecture). The corpus is read at attach from the
     LARGEST signal DB on disk (config-fingerprint-independent — the S1 procedural traps
     came from exactly that coupling), resolved to server bars per era with the GMT offset
     MEASURED PER ROW against `entryPrice` vs bar open (DST-immune; histogram logged), and
     pre-2017 daily-backfill rows are dropped by a window-span regime filter.
   - **The whole selection/deploy stack runs unchanged** via a counter mapping (win→Buy,
     loss→Sell): precision = wins among traded candidates, chance = base win rate, buy/sell
     recall = sensitivity/specificity (kills always/never-call collapses), operating point =
     coverage x (win − geometric break-even) on the purged calibration band, deploy gate =
     family-wise as always. Era-end `META` log line reports cov x (p − BE) vs the base-rate
     null with its standard error.
   - Labels are the side-conditional triple-barrier win caches (never the DB's S&R outcome);
     the fingerprint gains a conditional `|TGT:META1` token and meta models live under
     `State\META\` with the 2-output filename slot, so no collision with direction models.
   S2 casts no votes: live inference/online-learning entries are guarded off for the meta
   target until S3.
3. **S3 — inference integration**: meta-signal consumes fired candidates via the per-side
   hooks and votes; classic weights stay as a fallback layer; VerboseMode panel exposure.
4. **S4 — retrain + family-wise gate + (only if passed) deploy.**

Cross-sectional pooling (one model over SP500 + the 4 tick + 5 SQX-decoded instruments,
scale-free features already in place) is the next lever after S4, whatever S4's verdict.
Snapshot/seed ensembling rides along from S2 (average the last K checkpoint snapshots
instead of trusting one).

## S2 verdict on SP500 H1, and the pre-registered H4 experiment (2026-08-13)

**Measured over 350+370 eras:** the head carries real, persistent ranking skill — the
first out-of-sample signal since the pivot — but 0 eras cleared `cov x (p − BE)`. The
decomposition line localized it: the lift is LONG-ONLY (+1.8–2pp; the model's short picks
do *worse* than the short base rate), strongest on the MA family (67–70% traded win,
nominally above the 67.5% break-even but <1σ on ~350 trades and best-of-32 cells — not
family-wise evidence). The clinching arithmetic: edge x width = 0.02 x 4.74 ATR ≈ 0.095
ATR/trade against a measured spread of 0.099 ATR/trade. **The signal is real and being
consumed exactly by cost.**

**H4 experiment (pre-registered before any H4 data exists):** on SP500 H4 the ATR roughly
doubles while the spread is unchanged, so the cost drag (BE − base gap) roughly halves
(~1.3pp) while the lift — if it transfers — stays ~+2pp. Same pipeline end to end: S1
corpus backtest on H4, AI_META chart on H4. Hypotheses, stated in advance:
- H1 (primary): `cov x (p − BE)` turns positive at ≥25% coverage, clearing the 2σ edge
  floor — deployability decided by the existing gate, nothing relaxed.
- H2 (secondary): the lift remains long-only; SHORT traded win stays at/below its base.
- H3 (risk): the lift decays with timeframe as fast as the cost does (the tick-flow
  failure shape), leaving the net at zero again. If H3 is what the data says, the
  single-instrument well is closed at every accessible cost point and cross-sectional
  pooling is the only remaining lever.
The known thinning: ~1/4 the bars, so ~11k usable candidates — acceptable for this head.
The corpus loader now REQUIRES a symbol+timeframe-matching DB, so the H1 corpus cannot
leak into the H4 run.

**H4 RESULT (2026-08-13, 999 eras, 13,436 candidates): H3.** The cost gap halved exactly
as computed (BE 64.1% vs base 63.0% = 1.1pp at the derived 3.06/1.71 geometry) — and the
lift did not come with it: max cov x (p − BE) = +0.07 (1 era of 999, pre-2σ), skill at
selective thresholds +0.47pp mean (noise), and the head's ranking on H4 was slightly
INVERTED out of sample (its skipped candidates won 67% vs 62% for its trades), so the
expectancy fitter correctly pinned the threshold at 0/100% coverage. The lift shrank
faster than the cost — the tick-flow failure shape, now measured at the setup-conditional
level too. **The single-instrument SP500 well is closed at both accessible cost points,
with pre-registered hypotheses.** Remaining levers, in expected-value order: (1) candidate
streams that carry positive base edge (the user's own SQX strategies' signals as a
journaled candidate family — the meta head is an amplifier, and it finally would have
something to amplify; their trade-list CSVs measure this before any build), (2)
cross-sectional pooling (the last pure-NN lever; honest prior lowered by the twice-
measured signal≈cost equilibrium), (3) S3 as a variance-reducing risk filter at zero net
edge.