"""Retail's own side looked positive once the fill was fixed. Is that the pattern or the drift?

Retrial 1 killed the fade and inverted it: on USDJPY, XAUUSD and SP500 the fade loses 0.05
to 0.14 R because retail's trend-following setups are on the right side of instruments that
rose for the whole sample. Several cells show retail's own expR positive - SP500 H1 engulf
+0.113, XAUUSD H1 engulf +0.095 - and a POSITIVE base would be better than the near-zero one
step 2 is hunting for.

Before any of that is believed it has to beat the control that makes the boring explanation
explicit:

    CONTROL = random bars drawn from the SAME trend-filter state, same direction, same entry
              mechanics (a stop through the bar's extreme), same stop rule, same n.

Everything is held fixed except the candlestick itself. If retail's edge is really "buy the
break of any bar's high while the 20-MA is rising, on something that went up", the control
matches it and the pattern is worth nothing. That is the same shape of control that killed
"stops are a farmable magnet".

Reported on the NON-OVERLAPPING subset, and the difference carries its own standard error
rather than being eyeballed from two columns.
"""
import numpy as np, sys
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
import fills, book, retrial

SYMS = ('EURUSD', 'USDJPY', 'XAUUSD', 'SP500')


def diff_t(a, b):
    """Welch t for the difference of two independent means."""
    se = np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
    return (a.mean() - b.mean()) / max(se, 1e-12)


if __name__ == '__main__':
    syms = [s for s in sys.argv[1:] if s in SYMS] or list(SYMS)
    print("=== RETRIAL 2: is retail's trend side a PATTERN, or just drift? ===")
    print("    control = same trend state, same direction, same entry, random bar.")
    print("    'excess' is pattern minus control on independent trades - the real claim.\n")
    print(f"  {'sym':>7}{'tf':>5}{'setup':>8}{'dir':>4}{'k':>3}"
          f"{'n':>6}{'pattern':>9}{'t':>6}{'ctrl n':>7}{'control':>9}{'t':>6}"
          f"{'EXCESS':>9}{'t':>6}")
    tally = []
    for sym in syms:
        bk = fills.Book(sym)
        for tf in ('M15', 'H1'):
            f = book.frame(sym, tf, bk)
            for k in (1.0, 2.0):
                pat = retrial.retail_arm(sym, tf, k=k, bk=bk, f=f)
                ctl = retrial.retail_arm(sym, tf, k=k, control=True, bk=bk, f=f)
                cmap = {(nm, d): o for nm, d, o in ctl}
                for nm, d, o in pat:
                    c = cmap.get((nm, d))
                    if c is None:
                        continue
                    A = o['R'][o['indep']]; B = c['R'][c['indep']]
                    ex = A.mean() - B.mean()
                    t = diff_t(A, B)
                    tally.append(ex)
                    print(f"  {sym:>7}{tf:>5}{nm:>8}{d:>+4}{k:>3.0f}"
                          f"{len(A):>6}{A.mean():>+9.4f}{book.tstat(A):>+6.2f}"
                          f"{len(B):>7}{B.mean():>+9.4f}{book.tstat(B):>+6.2f}"
                          f"{ex:>+9.4f}{t:>+6.2f}")
    ex = np.array(tally)
    print(f"\n  {len(ex)} cells   mean excess {ex.mean():+.4f}   "
          f"positive {int((ex>0).sum())}/{len(ex)}")
    print("  A pattern with real content clears its control in most cells and by a margin")
    print("  that does not shrink when the sample is made independent.")