mql5-tick-compression/docs/FORMAT.md

TKZ format, version 1

TKZ stores the tick stream (time_msc, bid, ask) losslessly: decoding yields exactly the same 64-bit integers and the same IEEE-754 bit patterns that were encoded, in the same order, including duplicates, ticks that go back in time and arbitrary double values (NaN, infinities, -0.0, denormals, non-normalised prices).

The reference implementation is MQL5/Include/Tkz/TkzCodec.mqh; where this document and the code differ, the code is normative. All integers are little-endian.

1. Archive file (.tkz)

offset	size	field
0	4	magic "TKZF"
4	2	file format version = 1
6	2	header size = 64 (chunks start here)
8	32	symbol name, UTF-8, zero padded (informational)
40	4	symbol digits as int32, informational (-1 = unknown)
44	20	reserved, zero

The header is followed by chunks stored back to back. A reader builds its index by walking the chunk headers (each one states its payload size), so no separate index is needed and appending chunks to an existing file is trivial.

2. Chunk

A chunk is a self-contained block of ticks: it is decoded without any other chunk, so chunks can be read in any order (random access by day/week/month).

offset	size	field
0	4	magic "TKZ1"
4	1	format version = 1
5	1	model version = 1
6	1	table_bits: log2 of the counter table size (16..26)
7	1	digits d (0..12): prices are integers in units of 10^-d
8	1	modulus M (1..16): decimal sub-grid modulus (1 = off)
9	1	flags: bit 0 = chunk contains escaped (irregular) prices
10	2	reserved, zero
12	4	tick count N
16	4	payload size in bytes
20	4	checksum of the decoded ticks (see 2.4)
24	8	t_base: time_msc of the first tick
32	8	time_msc of the last tick
40	8	time step gt >= 1
48	8	price step g >= 1
56	8	B_base: first regular bid as an integer (units of 10^-d)
64	8	A_base: first regular ask as an integer
72	8	minimum time_msc in the chunk
80	8	maximum time_msc in the chunk
88	payload	arithmetic-coded tick stream

Decoders must reject unknown format or model versions.

2.1 Prices as integers

• digits is the smallest d for which every price p satisfies bits((double)round(p * 10^d) / 10^d) == bits(p) with |p * 10^d| <= 4e15. If no d in 0..12 makes all prices exact, the d with the fewest failures is used. Prices that fail this test are irregular: they are stored as raw 64-bit patterns (escapes) and do not move the integer price state.
• 10^d is computed by repeated multiplication by 10.0, so it is exact.
• Integer bid states start at B_base (ask: A_base). The price step g is the GCD of all differences between consecutive regular values of both sides (1 if none). The model works on X = (B - (B_base mod g)) / g (floored mod), which is exact, so tick sizes like 0.25 or 5 points cost nothing.
• The decoder reconstructs a regular price as (double)(X * g + off) / 10^d, which is the correctly rounded value and therefore bit-identical.

2.2 Time

gt is the GCD of all |t[i] - t[i-1]| (1 if all are equal), e.g. 1000 for second-resolution history. Each tick codes dt = (t[i] - t[i-1]) / gt (the first tick uses t_base, so its dt is 0). Negative and huge values are legal and go through the raw escape described below.

2.3 Decimal sub-grid

Many feeds quote on a coarser grid with a constant markup, e.g. XAUUSD with 3 digits where almost every price ends in 5. If the entropy of X mod 10 over the chunk is below 2 bits (and N >= 256), M = 10 is used: every price state is split into q = floor(X / M) and residue r = X mod M, and moves are coded as a new residue plus a move of q. Otherwise M = 1 (no split).

2.4 Checksum

h = 0; for every tick, with T = time_msc, B/A = bit patterns of bid/ask: h = H(h ^ lo32(T), hi32(T)), then the same for B and A, where H is the 32-bit mixing function H2 of the reference code. The decoder verifies it, so corruption and implementation mismatches are detected.

3. Tick coding

Every tick is coded as a sequence of binary decisions, each predicted by the context-mixing model (section 4) and arithmetic coded. Decisions are grouped in segments of 1-4 bits that form a small binary tree (node 1 = root); one hash-table block of 16 counters serves one segment.

1. Time delta (dt, magnitude group 0): bit length k of dt as a 4-bit value min(k, 15); 15 escapes to a second nibble min(k - 15, 15); 15 again escapes to the raw 64-bit value as 16 nibbles (negative values always take this path). Then the k - 1 bits below the leading one, MSB first, in nibbles; the first three nibbles use the full prefix as context, later ones its low 12 bits.
2. Tick type: tree of bid changed, ask changed and, if both changed, both moved by the same amount.
3. Escapes (only if flag bit 0): for each side that did not change, a flag; a flagged side is followed by its raw 64-bit pattern (16 nibbles).
4. Bid move (if changed): with M > 1 the new residue (4 bits); then the coarse move dq: when the residue changed, dq == 0 is possible and a nonzero decision precedes the sign; the magnitude |dq| - 1 is coded like the time delta (group 1).
5. Ask move (if changed and not equal to the bid move): new residue (M > 1); if the bid did not change, the coarse ask move (group 2), otherwise the change of the coarse spread dqa - dqb (group 3), which is much more predictable than the ask move itself.
6. State update: last two dt values and bit lengths, last three tick types, last two coarse moves per side, last spread change, EMAs of the dt bit length (rates 1/8 and 1/64), EMA of the bid move size, EMA of the coarse spread.

4. Model

• Counters: 32-bit p22 << 10 | n (22-bit probability, 10-bit count), update p += (target - p) * 2 / (2n + 3), n saturates at 255.
• Hash table: 2^table_bits counters in blocks of 16; slot 0 holds a 32-bit check, slots 1-15 the tree nodes of a segment. Block lookup is 2-way associative; on a miss the block whose node 1 has the smaller count is replaced.
• Contexts: 4-10 per field, built from the coding state (previous dt and its bit length, EMAs, previous tick types, last moves of both sides, the current dt bucket, spread relative to its EMA and in absolute units, residues, hour of the previous tick, ...). See CodeTick() for the exact hashes.
• Mixing: inputs are stretch(p) of each context plus a bias. Mixer 1 selects its weight set by the decision (class * 16 + node, 5376 sets), mixer 2 by a hash of the decision and a field-specific context (65536 sets), a final 2-input mixer per decision class combines both. Learning rates: 8/16384 for the first layer, 6/16384 for the final mixer.
• APM / SSE: two interpolating APMs (33 buckets over the stretch domain, rate 1/128), one per decision, one per hashed decision context. The final probability is (2 * p_mix + p_apm1 + p_apm2) / 4, clamped to [1, 4095].
• Arithmetic coder: 32-bit carry-less binary coder, xmid = x1 + ((x2 - x1) * p) >> 12 (64-bit product); on bit 1 x2 = xmid, else x1 = xmid + 1; leading equal bytes are shifted out. The encoder flushes the 4 bytes of x1; the decoder reads zeros past the end.

All model arithmetic is integer-only, so every platform produces the same bytes. The self-test (TkzSelfTest) checks this against a golden hash.