The SQX decoder is a per-record Python loop and cannot be vectorised - record
LENGTH depends on the config nibbles, so record k+1's offset is unknowable
without parsing record k. It therefore saturated exactly one core: 10% CPU on a
12-core box, ~3h for the four files.
But the format is randomly seekable. Every BLOCK_LENGTH records SQX restates all
four fields as absolute int64s, so byte ranges beginning at block headers decode
with no shared history. split_offsets() cuts a file on those boundaries and
decode_iter() gained start/stop. EURUSD: 55 min -> 9.4 min, 94% CPU.
find_block() will not trust a bare MAGIC match: 0x00..0x0e is a byte run that
delta payloads produce by coincidence, so a candidate is accepted only when the
next header downstream carries the next sequential block index.
Verified equal, not assumed equal: the same 315MB span decoded serially and in 6
chunks gives identical tick counts (31,056,000), identical bar counts (206,316)
and identical OHLC. The only divergence is the documented seam artifact - the
first tick of a chunk has no predecessor so its delta counts as zero, bounded at
workers-1 ticks in 513M (~2e-8).
Two fixes this shook out:
- The feed is not perfectly time-ordered. EURUSD carries 2 backward steps in
513,494,303 ticks, both under an hour, both in 2003-2006. Bucketing is by
absolute timestamp so every tick still lands in its true bar; the symptom is a
bucket emitted twice out of order. finalise() now stable-sorts before the
duplicate merge. The ordering assert is kept but keyed to MAGNITUDE, since a
real chunking bug displaces a large fraction of rows and feed noise displaces a
handful - only one of those is safe to continue past.
- Chunk workers return undivided sums; means are divided once globally. Dividing
per chunk would weight a straddling bar's mean-of-means wrong.
test_flow.py: charge the PER-BAR spread instead of a single median across
2003-2026 - FX spreads narrowed by roughly an order of magnitude over that span,
so one median charges modern cost to the 2000s and vice versa. Timeouts are now
reported separately rather than silently booked as stop-outs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sqx.decode_iter() turns the decoder into a generator, and ticks_to_bars.py reduces a
symbol to bars in one bounded-memory pass. Necessary rather than tidy: EURUSD is ~458M
ticks, which is ~15 GB held as arrays, so nothing downstream can take the raw stream.
23 years collapses to ~2.4M M5 bars that every test can load instantly.
Aggregation is vectorised with reduceat rather than looping per tick. The only real
complexity is that a bar can straddle a batch boundary, so the last partial bar of each
batch is carried and merged into the first of the next; per-tick deltas are likewise seeded
from the previous batch's final tick, so the first tick of a batch is not silently treated
as having no predecessor. Verified against the per-tick implementation it replaces: 21,971
bars either way, and every reported median identical to the digit (ticks/bar 76, up 38, dn
38, bidmoves 74, askmoves 74, spread 0.000126/0.000250, rvol 4.800e-07, gaps 3.95/31.4).
Throughput 132k ticks/s, at which point the decoder itself is the bottleneck and the
aggregation costs ~12%.
Features are chosen by what the feed can honestly support. It carries (time, bid, ask,
volume) and no trade direction - SQX's record has one volume field and MT5's
TICK_FLAG_BUY/SELL are empty on FX - so true signed order flow does not exist here and is
not synthesised under a flattering name. What is available:
tick rule up/down mid-price changes; the standard Lee-Ready fallback
quote asymmetry bid updates vs ask updates - which side is being repriced harder
arrival rate inter-tick gaps, mean and max; urgency rather than size
realised variance sum of squared mid returns, a far better volatility estimate than
the bar range and only obtainable from ticks
spread mean and max within the bar
Of these only the tick rule and quote asymmetry can point a direction; the rest are
unsigned, like every feature that has measured above noise in this project so far.
Bars are stamped by the OPEN of their interval and built only from ticks inside it, so no
bar's features depend on a tick after it closes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads SQX tick format 4.2 in pure Python. Derived from SQX's own writer, disassembled out
of internal/libs/SQDataLib.jar (TickDataWriter, NewDataFormat{,Writter}) with the javap
bundled in the install - so this follows the format as specified rather than as guessed.
Why, when Scripts/ExportTicks.mq5 pulls the same four fields from MT5: DEPTH. The broker's
MT5 tick history covers a few years; this file starts 2011-09-19. Sample size has been the
binding constraint on every question in this project - the H1/128-bar setup yields ~300
independent trades in 18 years, enough to resolve only a +6pp edge - so 15 years of ticks
is worth a decoder.
Format: four writeUTF strings, ten zero bytes, one more writeUTF, then records of
(time, ASK, BID, volume) - ask before bid, and the writer swaps them when bid>ask so ask is
always the larger. Every BLOCK_LENGTH=1000 records: MAGIC (15 bytes 0x00..0x0e) + int32
block index + config + four raw int64s. In between, deltas against the previous record.
Config is two bytes = four nibbles laid out high-first, nibble = (logicType << 2) |
dataType, where dataType 0..3 selects a 1/2/4/8-byte payload by magnitude and logicType
supplies the sign (MINUS/PLUS carry unsigned magnitudes; ASIS is a plain signed read).
The scale is the one thing NOT in the file. SQX keeps `decimals` in external metadata, and
1216010000 is equally plausible at 10^3, 10^5 or 10^6 - nothing in the bytes distinguishes
them. Guessing would be precisely the silent, plausible-looking error this project keeps
getting caught by: a 100x price scale error crashes nothing, it just quietly rescales every
ATR-normalised feature downstream. So calibrate_decimals() matches against a known
reference series instead. Against the MT5 SP500 H1 export the answer is not marginal:
decimals=5 median rel.err 9.001630
decimals=6 median rel.err 0.004078 <-
decimals=7 median rel.err 0.899984
Verified on 4M ticks: strictly monotonic timestamps, zero negative spreads, price range
1118.03-2048.38 over 2011-09 to 2014-11 (correct for SP500), median spread 0.43 (matches
this broker's H1 record). The 0.41% residual is the expected artefact of comparing a tick
ask against the nearest H1 bar close.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AnimateDread