resample.py composes M5 bars into M15/H1/H4 exactly - every column this pipeline
produces is composable (sums sum, maxes max, OHLC nests, means re-weight by tick
count), so this costs seconds instead of another 37-minute decode per timeframe.
Asserts tick conservation and extreme preservation on every output.
Motivation: ATR grows ~sqrt(time) while the spread does not, so spread/ATR should
fall with timeframe and make a small edge affordable. It does, monotonically, and
the measurement is clean (EURUSD, 1:1 barriers):
M5 spread 0.099 ATR random wins 36.8% cost 13.2pp
M15 0.057 39.7% 10.3pp
H1 0.029 43.6% 6.4pp
H4 0.015 47.6% 2.4pp
But the signal decays at the same rate. Rows clearing the family-wise bar:
M5 many, z to -10.1
M15 many, z to -5.8
H1 2 of 9, one POSITIVE and one negative - the shape of noise, not signal
H4 none
So the effect lives where the cost is fatal and is gone where the cost is
affordable. They never cross.
Also added --cheap=Q, which trades only the lowest-Q quantile of spread/ATR. This
is the one honest use of an unsigned feature: it cannot point a direction but it
can decline to trade, and both terms are known before entry. It does cut cost
(EURUSD M15 10.3 -> 7.6pp, USDJPY 12.7 -> 6.4pp) and the effect does not survive
there either - nothing clears the bar.
test_flow.py gained --tf= and keeps the z-score window at ~1 day on every
timeframe rather than a fixed 288 bars.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1.93 BILLION ticks -> 5.5M M5 bars (EURUSD/USDJPY/XAUUSD 2003-2026, SP500
2011-2026). Sequential non-overlapping trades, triple barriers, per-bar spread,
direction-permutation null with a family-wise max-statistic bar.
Order flow is genuinely ANTI-predictive at M5 - price mildly reverses the prior
bar's flow. Same sign on all four instruments, clearing the family-wise bar on
three:
USDJPY z -10.10 -1.73pp vs null
EURUSD z -7.95 -1.41pp
XAUUSD z -5.34 -0.68pp
SP500 z -2.60 -0.87pp (does not clear; half the sample)
Agrees with the -0.0151 next-bar correlation (vs +0.4961 same-bar, which is the
contemporaneous Cont/Kukanov/Stoikov effect and is not edge).
And the cost dwarfs it. Random entry at 1 ATR barriers after spread:
EURUSD spread 0.099 ATR -> wins 36.8% (13.2pp below the costless 50%)
USDJPY 0.154 34.9% (15.1pp)
SP500 0.292 26.5% (23.5pp)
XAUUSD 0.450 20.0% (30.0pp)
Cost rises monotonically with spread/ATR, which is an internal consistency check
on the apparatus. A ~1pp effect against 13-30pp of cost is 10-100x short.
Reversing does not rescue it - expR_rev is negative in every row of every
geometry. Widening the barriers does not either: at 4-8 ATR nothing clears the
bar (max |z| 2.51 vs 2.97). The effect lives exactly where the spread is fatal
and vanishes where the spread would be affordable, which is what a
seconds-to-minutes phenomenon predicts.
Adds --narrow/--wide geometry sets so both regimes are reproducible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first run reported -23pp edges at -75 sigma, which is not a market effect -
it is the tell this project has been burned by before (a lookahead, or here a
wrong reference, inflates whatever sign it lands on). The give-away was in the
output itself: a family-wise 5% bar of |z| > 71.67 where a centred null over 16
tests should sit near 2.5. Random entry was losing almost as badly as the signal.
Cause: z and "edge pp" were measured against be = sl/(sl+tp), the break-even of a
COSTLESS coin. These barriers charge the spread and book a loss when a single bar
spans both levels, so random entry at 1 ATR on M5 wins ~36.8%, not 50%. The table
was reporting the fixed cost of trading as if it were signal.
Now every row shows the empirical null win rate, the gap against it, and z
standardised by the null's own spread. Family-wise bar drops to 2.95 and the
result becomes legible: order flow is genuinely ANTI-predictive at M5, about 1pp
below random at z -5 to -8, clearing the bar in 12 of 16 tests and reproducing
across three geometries and two independent signal families. It agrees with the
-0.0151 next-bar correlation.
It is also untradeable, which the table now says out loud: the spread is 0.099 ATR
and costs 13pp of win rate against a 1pp effect. Reversing does not rescue it -
expR_rev is reported per row and stays negative everywhere. Added a footer stating
that beating the null is necessary but NOT sufficient; only exp R > 0 makes money.
Also:
- permutation null was allocating a single (nperm x nT) array, ~2 GB at these
trade counts. Now batched.
- null permutes the OBSERVED directions instead of flipping a fair coin, so a
directionally skewed rule on a trending instrument cannot pass on drift alone.
- timeouts reported separately rather than silently booked as stop-outs.
- calibration falls back to a midpoint sample when the tick history predates the
MT5 reference series (XAUUSD ticks start 2003-05-05, its H1 export 2004-06-11,
so the head sample overlapped by nothing). 2M ticks, because the sample must
span >=50 reference HOURS - 200k ticks of modern gold is nine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SQX decoder is a per-record Python loop and cannot be vectorised - record
LENGTH depends on the config nibbles, so record k+1's offset is unknowable
without parsing record k. It therefore saturated exactly one core: 10% CPU on a
12-core box, ~3h for the four files.
But the format is randomly seekable. Every BLOCK_LENGTH records SQX restates all
four fields as absolute int64s, so byte ranges beginning at block headers decode
with no shared history. split_offsets() cuts a file on those boundaries and
decode_iter() gained start/stop. EURUSD: 55 min -> 9.4 min, 94% CPU.
find_block() will not trust a bare MAGIC match: 0x00..0x0e is a byte run that
delta payloads produce by coincidence, so a candidate is accepted only when the
next header downstream carries the next sequential block index.
Verified equal, not assumed equal: the same 315MB span decoded serially and in 6
chunks gives identical tick counts (31,056,000), identical bar counts (206,316)
and identical OHLC. The only divergence is the documented seam artifact - the
first tick of a chunk has no predecessor so its delta counts as zero, bounded at
workers-1 ticks in 513M (~2e-8).
Two fixes this shook out:
- The feed is not perfectly time-ordered. EURUSD carries 2 backward steps in
513,494,303 ticks, both under an hour, both in 2003-2006. Bucketing is by
absolute timestamp so every tick still lands in its true bar; the symptom is a
bucket emitted twice out of order. finalise() now stable-sorts before the
duplicate merge. The ordering assert is kept but keyed to MAGNITUDE, since a
real chunking bug displaces a large fraction of rows and feed noise displaces a
handful - only one of those is safe to continue past.
- Chunk workers return undivided sums; means are divided once globally. Dividing
per chunk would weight a straddling bar's mean-of-means wrong.
test_flow.py: charge the PER-BAR spread instead of a single median across
2003-2026 - FX spreads narrowed by roughly an order of magnitude over that span,
so one median charges modern cost to the 2000s and vice versa. Timeouts are now
reported separately rather than silently booked as stop-outs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ready ahead of the bar build so the analysis runs the moment EURUSD lands.
Tests the two SIGNED features the tick pipeline can produce - tick-rule imbalance and
event-count OFI - as entry triggers at 1:1/12, 1:2/24 and 2:3/48 bar geometries. Everything
else the pipeline computes is unsigned and cannot point a trade however well it measures.
Same discipline as every other test here: signal on bar i, entry at the OPEN of i+1,
barriers scanned forward only, sequential NON-OVERLAPPING trades so each is independent
(skipping that is what produced a fake +2.66pp at 2.9 sigma earlier in this project),
break-even == chance by the gambler's-ruin identity, spread charged inside the barrier, and
a sign-flip null taking the max over the whole family for the family-wise bar.
The prior is written into the docstring before any result exists: OFI is established as a
CONTEMPORANEOUS explainer whose predictive power decays within seconds, and it reproduced
that here at +0.56 against the same-bar return. So the expectation is that it explains the
bar it is measured in and says nothing about the next. What is actually being tested is the
gap between that literature - equities, sub-second, size-weighted book data - and this
setting: retail FX CFD feed, 5-minute bars, event counts without sizes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
