The first run reported -23pp edges at -75 sigma, which is not a market effect -
it is the tell this project has been burned by before (a lookahead, or here a
wrong reference, inflates whatever sign it lands on). The give-away was in the
output itself: a family-wise 5% bar of |z| > 71.67 where a centred null over 16
tests should sit near 2.5. Random entry was losing almost as badly as the signal.
Cause: z and "edge pp" were measured against be = sl/(sl+tp), the break-even of a
COSTLESS coin. These barriers charge the spread and book a loss when a single bar
spans both levels, so random entry at 1 ATR on M5 wins ~36.8%, not 50%. The table
was reporting the fixed cost of trading as if it were signal.
Now every row shows the empirical null win rate, the gap against it, and z
standardised by the null's own spread. Family-wise bar drops to 2.95 and the
result becomes legible: order flow is genuinely ANTI-predictive at M5, about 1pp
below random at z -5 to -8, clearing the bar in 12 of 16 tests and reproducing
across three geometries and two independent signal families. It agrees with the
-0.0151 next-bar correlation.
It is also untradeable, which the table now says out loud: the spread is 0.099 ATR
and costs 13pp of win rate against a 1pp effect. Reversing does not rescue it -
expR_rev is reported per row and stays negative everywhere. Added a footer stating
that beating the null is necessary but NOT sufficient; only exp R > 0 makes money.
Also:
- permutation null was allocating a single (nperm x nT) array, ~2 GB at these
trade counts. Now batched.
- null permutes the OBSERVED directions instead of flipping a fair coin, so a
directionally skewed rule on a trending instrument cannot pass on drift alone.
- timeouts reported separately rather than silently booked as stop-outs.
- calibration falls back to a midpoint sample when the tick history predates the
MT5 reference series (XAUUSD ticks start 2003-05-05, its H1 export 2004-06-11,
so the head sample overlapped by nothing). 2M ticks, because the sample must
span >=50 reference HOURS - 200k ticks of modern gold is nine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SQX decoder is a per-record Python loop and cannot be vectorised - record
LENGTH depends on the config nibbles, so record k+1's offset is unknowable
without parsing record k. It therefore saturated exactly one core: 10% CPU on a
12-core box, ~3h for the four files.
But the format is randomly seekable. Every BLOCK_LENGTH records SQX restates all
four fields as absolute int64s, so byte ranges beginning at block headers decode
with no shared history. split_offsets() cuts a file on those boundaries and
decode_iter() gained start/stop. EURUSD: 55 min -> 9.4 min, 94% CPU.
find_block() will not trust a bare MAGIC match: 0x00..0x0e is a byte run that
delta payloads produce by coincidence, so a candidate is accepted only when the
next header downstream carries the next sequential block index.
Verified equal, not assumed equal: the same 315MB span decoded serially and in 6
chunks gives identical tick counts (31,056,000), identical bar counts (206,316)
and identical OHLC. The only divergence is the documented seam artifact - the
first tick of a chunk has no predecessor so its delta counts as zero, bounded at
workers-1 ticks in 513M (~2e-8).
Two fixes this shook out:
- The feed is not perfectly time-ordered. EURUSD carries 2 backward steps in
513,494,303 ticks, both under an hour, both in 2003-2006. Bucketing is by
absolute timestamp so every tick still lands in its true bar; the symptom is a
bucket emitted twice out of order. finalise() now stable-sorts before the
duplicate merge. The ordering assert is kept but keyed to MAGNITUDE, since a
real chunking bug displaces a large fraction of rows and feed noise displaces a
handful - only one of those is safe to continue past.
- Chunk workers return undivided sums; means are divided once globally. Dividing
per chunk would weight a straddling bar's mean-of-means wrong.
test_flow.py: charge the PER-BAR spread instead of a single median across
2003-2026 - FX spreads narrowed by roughly an order of magnitude over that span,
so one median charges modern cost to the 2000s and vice versa. Timeouts are now
reported separately rather than silently booked as stop-outs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces bidmoves/askmoves with bid_up, bid_dn, ask_up, ask_dn. A bid ticking UP and an ask
ticking DOWN both mean buy-side pressure, and a counter that only records "the bid changed"
cannot tell them apart - it throws away the direction, which is the only part that could
ever point a trade. This is order-flow imbalance in the Cont/Kukanov/Stoikov sense, in its
event-count form; the feed carries no sizes so it cannot be size-weighted.
Caught before the 3-hour build rather than after, which was the point of smoke-testing on a
bounded sample first.
Verified against the previous column set on the same 3M ticks: 21,971 bars, ticks/bar 76,
up 38, dn 38, spread 0.000126, rvol 4.800e-07, gaps 3.95/31.4 - all identical - and
bid_up+bid_dn reproduces the old bidmoves count of 74 exactly, as it must.
The orientation check that matters: OFI correlates +0.56 with the SAME-bar return. That is
the contemporaneous signature the literature reports, and it is also the cheapest guard
against the failure mode that would otherwise pass silently - a sign flip would read -0.56
and every downstream test would then be fitting the negative of the intended feature.
Expectations set in the docstring rather than discovered later: OFI is well established as
a contemporaneous EXPLAINER of price change and its predictive power decays within seconds.
At M5 with multi-hour horizons the prior should be that it explains the bar it is measured
in, not the next one. Measuring it anyway is the point - but a +0.56 contemporaneous
correlation is not evidence of an edge and must not be reported as one.
Merge/mean bookkeeping is now index-driven off COLUMNS instead of positional, so adding a
feature cannot silently mis-merge a bar that straddles a batch boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sqx.decode_iter() turns the decoder into a generator, and ticks_to_bars.py reduces a
symbol to bars in one bounded-memory pass. Necessary rather than tidy: EURUSD is ~458M
ticks, which is ~15 GB held as arrays, so nothing downstream can take the raw stream.
23 years collapses to ~2.4M M5 bars that every test can load instantly.
Aggregation is vectorised with reduceat rather than looping per tick. The only real
complexity is that a bar can straddle a batch boundary, so the last partial bar of each
batch is carried and merged into the first of the next; per-tick deltas are likewise seeded
from the previous batch's final tick, so the first tick of a batch is not silently treated
as having no predecessor. Verified against the per-tick implementation it replaces: 21,971
bars either way, and every reported median identical to the digit (ticks/bar 76, up 38, dn
38, bidmoves 74, askmoves 74, spread 0.000126/0.000250, rvol 4.800e-07, gaps 3.95/31.4).
Throughput 132k ticks/s, at which point the decoder itself is the bottleneck and the
aggregation costs ~12%.
Features are chosen by what the feed can honestly support. It carries (time, bid, ask,
volume) and no trade direction - SQX's record has one volume field and MT5's
TICK_FLAG_BUY/SELL are empty on FX - so true signed order flow does not exist here and is
not synthesised under a flattering name. What is available:
tick rule up/down mid-price changes; the standard Lee-Ready fallback
quote asymmetry bid updates vs ask updates - which side is being repriced harder
arrival rate inter-tick gaps, mean and max; urgency rather than size
realised variance sum of squared mid returns, a far better volatility estimate than
the bar range and only obtainable from ticks
spread mean and max within the bar
Of these only the tick rule and quote asymmetry can point a direction; the rest are
unsigned, like every feature that has measured above noise in this project so far.
Bars are stamped by the OPEN of their interval and built only from ticks inside it, so no
bar's features depend on a tick after it closes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AnimateDread