05c6a5c484
Everything tested before this asked ONE question - can recent price or flow
predict the next bar's direction - and answered no four ways. These are different
families, each with a published prior rather than a hunch.
1 TIME-SERIES MOMENTUM (Moskowitz/Ooi/Pedersen). 34 configurations across 4
symbols x D1/H4 x 6 lookbacks. Nothing. The one rule that looks strong -
XAUUSD H4 250-bar, p=0.0038, t+3.30, +10.32%/yr - returns essentially exactly
buy-and-hold's +10.36%. It is not timing gold, it is being long gold. Hence the
vs-B&H column: on a drifting asset a rule that is merely long most of the time
looks skilful and is not.
2 SEASONALITY. The first thing in this project to survive a properly controlled
test: 5 of 8 clear a family-wise max-statistic bar, two at p=0.0002. Split-half
kills two of them (USDJPY dow-6 n=116 and SP500 hour-0 n=533 are thin
off-session buckets). Two HOLD with near-identical halves:
XAUUSD hour 1 +2.29 bp (t+6.44) / +2.43 bp (t+5.53)
EURUSD hour 13 -1.47 bp (t-6.80) / -0.64 bp (t-3.56)
Gold's hour 1 alone carries more than half the +4.22 bp/day drift.
And it is still not tradeable. Widening the window to amortise the 4.92 bp
round trip: the best of 144 windows (hour 1, 8h) nets +0.14 bp/day, t +0.23,
and splits +1.29 / -1.00 - the sign flips between halves. Every other window is
negative. Real, stable, well measured, and about 2x too small to cross its own
spread. Same shape as the flow result.
3 OVERNIGHT/INTRADAY - folded into the hour analysis above.
4 VOLATILITY-MANAGED DRIFT (Moreira/Muir) - the one needing no directional edge.
Does NOT reproduce here: flat on gold (-0.01), and it HURTS SP500 (0.71 -> 0.41
D1, 0.77 -> 0.54 H4). Honest negative against a strong prior.
What survives all of it is drift, which is large and significant while every
timing rule is noise: XAUUSD +10.24%/yr (t 2.88), SP500 +12.25%/yr (t 2.84),
against USDJPY +1.16%/yr (t 0.59) and EURUSD ~0.
resample.py gains D1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
79 lines
3.9 KiB
Python
79 lines
3.9 KiB
Python
"""Compose M5 microstructure bars into higher timeframes. Exact, no re-decode.
|
|
|
|
Rebuilding from ticks would cost another full 37-minute pass per timeframe. It is
|
|
unnecessary: every column this pipeline produces composes exactly across sub-bars.
|
|
|
|
open first sub-bar high max low min close last sub-bar
|
|
ticks, upticks, downticks, bid_up/dn, ask_up/dn, rvol, volume sum
|
|
spread_max, gap_max max
|
|
spread_mean, gap_mean TICK-WEIGHTED mean, never mean-of-means
|
|
|
|
rvol composes exactly too, which is worth stating because it is the one that looks like it
|
|
should not: it is the sum of squared per-tick mid returns, and the return spanning a bar
|
|
boundary is already attributed to the later bar (prev_mid carries across batches in
|
|
_scan), so no squared term is dropped or double-counted at a seam.
|
|
|
|
WHY: at M5 the spread is 0.099 ATR on EURUSD and costs 13pp of win rate, which is what
|
|
made the real ~1pp flow effect untradeable. ATR grows roughly as sqrt(time) while the
|
|
spread does not, so spread/ATR should fall ~3.5x by H1 and ~7x by H4. That predicts the
|
|
cost falls to ~4pp and ~2pp. Whether the EFFECT survives the same dilation is the actual
|
|
question - a seconds-to-minutes phenomenon has no reason to.
|
|
"""
|
|
import numpy as np, sys, os
|
|
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
|
|
from ticks_to_bars import COLUMNS, _I, SUM_IDX, MAX_IDX, MEAN_IDX
|
|
|
|
BARS = 'c:/Users/admin/Documents/Workspaces/Market Data/bars/'
|
|
|
|
|
|
def resample(arr, step_ms):
|
|
"""M5 bars -> `step_ms` bars. Input must be time-sorted and duplicate-free."""
|
|
t = arr[:, _I['time']]
|
|
bucket = (t // step_ms) * step_ms
|
|
starts = np.concatenate(([0], np.flatnonzero(np.diff(bucket)) + 1))
|
|
ends = np.concatenate((starts[1:] - 1, [len(t) - 1]))
|
|
|
|
out = np.zeros((len(starts), len(COLUMNS)))
|
|
out[:, _I['time']] = bucket[starts]
|
|
out[:, _I['open']] = arr[starts, _I['open']]
|
|
out[:, _I['close']] = arr[ends, _I['close']]
|
|
out[:, _I['high']] = np.maximum.reduceat(arr[:, _I['high']], starts)
|
|
out[:, _I['low']] = np.minimum.reduceat(arr[:, _I['low']], starts)
|
|
for i in SUM_IDX:
|
|
if i in MEAN_IDX:
|
|
continue
|
|
out[:, i] = np.add.reduceat(arr[:, i], starts)
|
|
for i in MAX_IDX:
|
|
out[:, i] = np.maximum.reduceat(arr[:, i], starts)
|
|
#--- means: re-weight by the sub-bar tick counts before summing, then divide by the
|
|
#--- total. A plain mean of the M5 means would weight a 3-tick bar like a 900-tick one.
|
|
n = arr[:, _I['ticks']]
|
|
tot = np.maximum(out[:, _I['ticks']], 1.0)
|
|
for i in MEAN_IDX:
|
|
out[:, i] = np.add.reduceat(arr[:, i] * n, starts) / tot
|
|
return out
|
|
|
|
|
|
if __name__ == '__main__':
|
|
TFS = {'M15': 15 * 60 * 1000, 'H1': 60 * 60 * 1000, 'H4': 4 * 60 * 60 * 1000,
|
|
'D1': 24 * 60 * 60 * 1000}
|
|
syms = [a for a in sys.argv[1:] if not a.startswith('-')] or \
|
|
['EURUSD', 'USDJPY', 'XAUUSD', 'SP500']
|
|
for sym in syms:
|
|
src = f"{BARS}{sym}_M5_ticks.npz"
|
|
if not os.path.exists(src):
|
|
print(f"{sym}: no M5 bars"); continue
|
|
z = np.load(src, allow_pickle=True)
|
|
a = z['bars']
|
|
print(f"\n{sym}: {len(a):,} M5 bars")
|
|
for name, step in TFS.items():
|
|
out = resample(a, step)
|
|
#--- conservation checks: aggregation must not create or destroy ticks, and the
|
|
#--- extremes must survive. Cheap, and catches a mis-set index instantly.
|
|
assert abs(out[:, _I['ticks']].sum() - a[:, _I['ticks']].sum()) < 1
|
|
assert abs(out[:, _I['high']].max() - a[:, _I['high']].max()) < 1e-9
|
|
assert abs(out[:, _I['low']].min() - a[:, _I['low']].min()) < 1e-9
|
|
assert (np.diff(out[:, _I['time']]) > 0).all()
|
|
dst = f"{BARS}{sym}_{name}_ticks.npz"
|
|
np.savez_compressed(dst, bars=out, columns=np.array(COLUMNS))
|
|
print(f" {name:>3}: {len(out):>9,} bars ticks conserved -> {os.path.basename(dst)}")
|