Warrior_EA/research/test_temporal.py

"""
DOES ACROSS-TIME STRUCTURE CARRY INFORMATION THE MARGINAL MI CANNOT SEE?

This is the one hypothesis the shipped diagnostics explicitly do NOT cover. The EA's own report
says it: "that measure is marginal (one feature at a time) and per-bar, whereas the network sees
20 bars jointly. A floor-level reading rules out a simple per-feature edge; it cannot rule out one
that only exists in combination or across time."

Everything measured so far is marginal and per-bar - MI headline (p=0.4975), the lag profile
(nothing at any lag 0-20), the geometry scan (p=0.39). The instrument for across-time structure is
the sequence model, and until 2026-08-07 (`bfc1da9`) it read its window BACKWARDS: the LSTM's
output came from the OLDEST bar in the window, with the bar being predicted attenuated ~80x. So
every LSTM/HYBRID result to date is confounded and the hypothesis has never actually been tested.

WHY THIS SCRIPT INSTEAD OF RETRAINING THE LSTM. A topology comparison in MT5 confounds the thing
we care about (can order be exploited?) with learning rate, initialisation, capacity, class
weighting, batch-norm state and a single OOS slice - and costs hours per run on a CPU-only box.
The question is about the DATA, so ask the data directly.

THE EXPERIMENT. Three configurations, identical model, identical CV, identical null:

  A  entry bar only      X = F[i]                      joint across FEATURES, one bar
  B  window, in order    X = F[i-W+1..i] flattened     joint across features AND time
  C  window, order dead  same values, lag order shuffled INDEPENDENTLY PER SAMPLE

  B vs A  -> does history help at all?
  B vs C  -> does temporal POSITION carry information?   <-- the actual hypothesis

C is the point of the design. A fixed shuffle would be worthless: any model can relearn a
permuted column layout, so a single global permutation tests nothing. Shuffling per sample
destroys the correspondence between column position and lag while preserving the exact multiset of
values in the window - the time-order analogue of the shuffled-label null used everywhere else here.

METRIC. Directional precision on the bars the model calls Buy/Sell, against the zero-skill
benchmark max(P(Buy), P(Sell)) - deliberately the same metric and benchmark the EA's own
deployability gate uses (see EDGE_MIN_SIGMAS), so a result here is directly comparable to an era
line rather than needing translation.

TWO NULLS, because one of them lies. The analytic binomial p assumes independent calls; barrier
labels are massively autocorrelated (adjacent labels share all but one bar of their outcome
window), so that p is optimistic and is reported only for comparability with the era lines. The
BLOCK-PERMUTATION p (block = the barrier horizon, Phipson & Smyth) is the one to believe. Where
they disagree, the block null is right - this is the same correction test_volume.py and
test_spread.py already apply.

CV. Purged, embargoed walk-forward (Lopez de Prado ch. 7): train strictly before each test fold,
with an H-bar embargo on both sides so no training label's outcome window overlaps the test window.
Without the embargo the barrier horizon leaks the answer across the split and every configuration
scores well - which is the mistake that produced +0.53 R on an "honest" chronological holdout once
already.

PREREQUISITE: the rates CSV. Build the EA with WARRIOR_EXPORT_FEATURES defined and attach it to a
chart once; ExportRawRates() writes
    <Common>/Files/Warrior_EA/Research/<SYM>_<TF>_rates.csv
That build returns from OnTick() before the trading path, so it cannot place an order.

USAGE
    python test_temporal.py                        # SP500 H1, the shipped geometry
    python test_temporal.py --sym EURUSD --tf 60
    python test_temporal.py --fast                 # fewer permutations, smaller model
"""
import argparse
import sys

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier

from kit import load_rates, atr, barrier_vec, features

BUY, SELL, NEU = 0, 1, 2


def build_windows(X, W):
    """Flatten a W-bar lookback into one row per bar, OLDEST COLUMN BLOCK FIRST.

    Column block k holds bar (i - W + 1 + k), so block W-1 is bar i itself - the same
    chronological order BuildFeatureWindow() now feeds the network, deliberately, so that
    'column position' here means the same thing as 'timestep' there.

    Rows before W-1 cannot be formed and are marked invalid rather than zero-filled: a zero-filled
    early window is a fabricated observation, and there are 38k real ones.
    """
    n, f = X.shape
    out = np.zeros((n, W * f), dtype=np.float32)
    for k in range(W):
        lag = W - 1 - k                      # k=0 -> oldest, k=W-1 -> current bar
        out[lag:, k * f:(k + 1) * f] = X[:n - lag] if lag else X
    ok = np.zeros(n, bool)
    ok[W - 1:] = True
    return out, ok


def shuffle_lags(Xw, W, f, rng):
    """Per-sample permutation of the W lag blocks. Feature identity WITHIN a block is preserved -
    only the block's position, i.e. its lag, is randomised. So this destroys 'when' while leaving
    'what' completely intact, which is exactly the contrast B vs C is supposed to isolate.
    """
    out = np.empty_like(Xw)
    view = Xw.reshape(len(Xw), W, f)
    dst = out.reshape(len(Xw), W, f)
    for i in range(len(Xw)):
        dst[i] = view[i, rng.permutation(W)]
    return out


def purged_folds(n, k, embargo):
    """Contiguous test blocks, training restricted to bars more than `embargo` away on either side."""
    edges = np.linspace(0, n, k + 1, dtype=int)
    for j in range(k):
        lo, hi = edges[j], edges[j + 1]
        test = np.zeros(n, bool)
        test[lo:hi] = True
        train = np.ones(n, bool)
        train[max(0, lo - embargo):min(n, hi + embargo)] = False
        if train.sum() < 500 or test.sum() < 200:
            continue
        yield train, test


def oof_calls(X, y, folds, model_kw, seed):
    """Out-of-fold predictions. Returns (predicted, truth) over every scored bar."""
    preds, truths = [], []
    for train, test in folds:
        clf = HistGradientBoostingClassifier(random_state=seed, **model_kw)
        clf.fit(X[train], y[train])
        preds.append(clf.predict(X[test]))
        truths.append(y[test])
    if not preds:
        return np.array([], dtype=np.int8), np.array([], dtype=np.int8)
    return np.concatenate(preds), np.concatenate(truths)


def dir_precision(pred, truth):
    """Precision on the bars a direction was called, plus coverage. Mirrors the EA's oosDirCalls /
    oosDirHits, so `edge` below is the same quantity its era line prints."""
    called = (pred == BUY) | (pred == SELL)
    n_calls = int(called.sum())
    if n_calls == 0:
        return 0.0, 0, 0.0
    hits = int((pred[called] == truth[called]).sum())
    return 100.0 * hits / n_calls, n_calls, 100.0 * n_calls / len(pred)


def zero_skill_pct(truth):
    """What an information-free model scores by always calling the more common direction: that
    class's share of ALL bars. Same benchmark as chancePrecPct in Training.mqh - NOT the Buy+Sell
    share, which is a different (and once badly wrong) thing."""
    return 100.0 * max((truth == BUY).mean(), (truth == SELL).mean())


def block_perm_labels(y, horizon, rng):
    """Shuffle contiguous blocks of labels, preserving within-block order. Block = the barrier
    horizon, because adjacent labels share almost their whole outcome window and a free shuffle
    produces a null far too tight."""
    n = len(y)
    blk = max(int(horizon), 1)
    starts = np.arange(0, n, blk)
    order = rng.permutation(len(starts))
    idx = np.concatenate([np.arange(starts[o], min(starts[o] + blk, n)) for o in order])[:n]
    return y[idx]


def normal_upper(z):
    """A&S 26.2.17, same approximation as NormalUpperTail() in Training.mqh so the two agree."""
    from math import exp, sqrt, pi
    if z < 0:
        return 1.0 - normal_upper(-z)
    p = 0.2316419
    b = (0.319381530, -0.356563782, 1.781477937, -1.821255978, 1.330274429)
    t = 1.0 / (1.0 + p * z)
    pdf = exp(-0.5 * z * z) / sqrt(2 * pi)
    poly = t * (b[0] + t * (b[1] + t * (b[2] + t * (b[3] + t * b[4]))))
    return max(0.0, min(1.0, pdf * poly))


def run(sym, tf, sl_m, tp_m, H, W, k_folds, nperm, model_kw, seed=0):
    t, o, h, l, c, v, spr = load_rates(sym, tf)
    a = atr(h, l, c, 14)
    tick = np.nanmin(np.abs(np.diff(np.unique(np.round(c, 8)))))
    sp = np.nanmedian(spr) * tick
    if not np.isfinite(sp):
        sp = 0.0
    lab, valid = barrier_vec(h, l, c, np.where(a > 0, a, np.nan), sl_m, tp_m, H, sp)

    X, names, _ = features(t, o, h, l, c, v)
    f = X.shape[1]
    Xw, wok = build_windows(X, W)

    m = valid & np.isfinite(a) & (a > 0) & wok
    m[:200] = False
    m[-(H + 2):] = False
    y = lab[m].astype(np.int8)
    n = int(m.sum())
    if n < 2000:
        print(f"!! only {n} usable bars - not enough to run this. Export more history.")
        return

    chance = zero_skill_pct(y)
    print(f"\n=== {sym} tf{tf}  SL{sl_m}:TP{tp_m} H={H}  W={W} bars x {f} features  n={n} ===")
    print(f"    labels Buy {100*(y==BUY).mean():.1f}%  Sell {100*(y==SELL).mean():.1f}%  "
          f"Neutral {100*(y==NEU).mean():.1f}%   zero-skill precision {chance:.2f}%")
    print(f"    spread charged {sp:.5f} ({np.nanmedian(spr):.0f} points)   "
          f"purged CV {k_folds} folds, embargo {H} bars each side")

    folds = list(purged_folds(n, k_folds, H))
    if not folds:
        print("!! no usable folds")
        return

    rng = np.random.default_rng(seed)
    configs = {
        "A entry bar only  ": X[m],
        "B window, in order": Xw[m],
        "C window, order    dead": shuffle_lags(Xw[m], W, f, rng),
    }

    print(f"\n{'config':<26}{'precision':>11}{'edge':>9}{'calls':>9}{'cover':>8}"
          f"{'z':>7}{'p_anl':>8}{'p_block':>9}")
    results = {}
    for name, Xc in configs.items():
        pred, truth = oof_calls(Xc, y, folds, model_kw, seed)
        prec, ncalls, cover = dir_precision(pred, truth)
        edge = prec - chance
        p0 = chance / 100.0
        se = 100.0 * np.sqrt(p0 * (1 - p0) / max(ncalls, 1))
        z = edge / se if se > 0 else 0.0
        p_anl = normal_upper(z)

        # Block-permutation null: relabel, re-run the whole CV, count how often the null reaches
        # this edge. Expensive but it is the only p worth believing on autocorrelated labels.
        worse = 0
        for b in range(nperm):
            yp = block_perm_labels(y, H, np.random.default_rng(seed + 1000 + b))
            pp, tt = oof_calls(Xc, yp, folds, model_kw, seed)
            pn, nc, _ = dir_precision(pp, tt)
            if nc > 0 and (pn - zero_skill_pct(tt)) >= edge:
                worse += 1
            print(f"    {name.strip()}: null {b+1}/{nperm}", end="\r", flush=True)
        p_blk = (1 + worse) / (nperm + 1)     # Phipson & Smyth
        results[name] = (prec, edge, ncalls, cover, z, p_anl, p_blk)
        print(f"{name:<26}{prec:>10.2f}%{edge:>+8.2f}{ncalls:>9d}{cover:>7.1f}%"
              f"{z:>7.2f}{p_anl:>8.4f}{p_blk:>9.3f}")

    a_edge = results["A entry bar only  "][1]
    b_edge = results["B window, in order"][1]
    c_edge = results["C window, order    dead"][1]
    b_pblk = results["B window, in order"][6]

    print("\n--- verdict ---")
    print(f"  history helps at all      B - A = {b_edge - a_edge:+.2f}pp")
    print(f"  temporal ORDER is used    B - C = {b_edge - c_edge:+.2f}pp   <-- the hypothesis")
    if b_pblk > 0.05:
        print(f"\n  B does not clear its own block-permutation null (p={b_pblk:.3f}). Whatever B - C"
              f"\n  shows, there is no edge here to attribute to anything - the across-time hypothesis"
              f"\n  is CLOSED on this instrument and geometry, and the sequence topologies are not"
              f"\n  being held back by their architecture.")
    elif b_edge - c_edge > 0 and b_pblk <= 0.05:
        print(f"\n  B clears its null (p={b_pblk:.3f}) AND beats the order-destroyed control by"
              f"\n  {b_edge - c_edge:+.2f}pp. That is across-time structure, and it is the first thing in this"
              f"\n  project a sequence model could exploit that a per-bar measure cannot. Re-run on a"
              f"\n  second instrument before believing it.")
    else:
        print(f"\n  B clears its null (p={b_pblk:.3f}) but does NOT beat the order-destroyed control."
              f"\n  Any edge here is joint-across-FEATURES, not across-time - so it belongs in a wider"
              f"\n  per-bar feature vector, not in a recurrent topology.")
    print("\n  Reminder: this is ONE instrument/geometry. Do not read a single positive as an edge -"
          "\n  that is the best-of-N error this codebase has now found in four places.")


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--sym", default="SP500")
    ap.add_argument("--tf", type=int, default=16385)   # PERIOD_H1 as MT5 writes it
    ap.add_argument("--sl", type=float, default=3.33)  # the DERIVED geometry from the 08-07 run
    ap.add_argument("--tp", type=float, default=6.66)
    ap.add_argument("--horizon", type=int, default=256)
    ap.add_argument("--window", type=int, default=20)  # HistoryBars
    ap.add_argument("--folds", type=int, default=5)
    ap.add_argument("--nperm", type=int, default=25)
    ap.add_argument("--fast", action="store_true")
    args = ap.parse_args()

    model_kw = dict(max_iter=200, learning_rate=0.06, max_leaf_nodes=31, early_stopping=False)
    nperm = args.nperm
    if args.fast:
        model_kw.update(max_iter=60, max_leaf_nodes=15)
        nperm = 10

    try:
        run(args.sym, args.tf, args.sl, args.tp, args.horizon, args.window,
            args.folds, nperm, model_kw)
    except OSError as e:
        print(f"!! could not read the rates CSV: {e}")
        print("   Build the EA with WARRIOR_EXPORT_FEATURES and attach it to a chart once to write")
        print("   <Common>/Files/Warrior_EA/Research/<SYM>_<TF>_rates.csv, then re-run.")
        sys.exit(1)


if __name__ == "__main__":
    main()