mql5-tick-compression/docs/FORMAT.md

# TKZ format, version 1

TKZ stores the tick stream `(time_msc, bid, ask)` losslessly: decoding yields
exactly the same 64-bit integers and the same IEEE-754 bit patterns that were
encoded, in the same order, including duplicates, ticks that go back in time
and arbitrary double values (NaN, infinities, `-0.0`, denormals,
non-normalised prices).

The reference implementation is `lib/TkzCodec.mqh`; where this
document and the code differ, the code is normative. All integers are
little-endian.

## 1. Archive file (`.tkz`)

| offset | size | field |
|---:|---:|---|
| 0 | 4 | magic `"TKZF"` |
| 4 | 2 | file format version = 1 |
| 6 | 2 | header size = 64 (chunks start here) |
| 8 | 32 | symbol name, UTF-8, zero padded (informational) |
| 40 | 4 | symbol digits as int32, informational (`-1` = unknown) |
| 44 | 20 | reserved, zero |

The header is followed by chunks stored back to back. A reader builds its
index by walking the chunk headers (each one states its payload size), so no
separate index is needed and appending chunks to an existing file is trivial.

## 2. Chunk

A chunk is a self-contained block of ticks: it is decoded without any other
chunk, so chunks can be read in any order (random access by day/week/month).

| offset | size | field |
|---:|---:|---|
| 0 | 4 | magic `"TKZ1"` |
| 4 | 1 | format version = 1 |
| 5 | 1 | model version = 1 |
| 6 | 1 | `table_bits`: log2 of the counter table size (16..26) |
| 7 | 1 | `digits` d (0..12): prices are integers in units of 10^-d |
| 8 | 1 | `modulus` M (1..16): decimal sub-grid modulus (1 = off) |
| 9 | 1 | flags: bit 0 = chunk contains escaped (irregular) prices |
| 10 | 2 | reserved, zero |
| 12 | 4 | tick count N |
| 16 | 4 | payload size in bytes |
| 20 | 4 | checksum of the decoded ticks (see 2.4) |
| 24 | 8 | `t_base`: time_msc of the first tick |
| 32 | 8 | time_msc of the last tick |
| 40 | 8 | time step `gt` >= 1 |
| 48 | 8 | price step `g` >= 1 |
| 56 | 8 | `B_base`: first regular bid as an integer (units of 10^-d) |
| 64 | 8 | `A_base`: first regular ask as an integer |
| 72 | 8 | minimum time_msc in the chunk |
| 80 | 8 | maximum time_msc in the chunk |
| 88 | payload | arithmetic-coded tick stream |

Decoders must reject unknown format or model versions.

### 2.1 Prices as integers

* `digits` is the smallest d for which every price p satisfies
  `bits((double)round(p * 10^d) / 10^d) == bits(p)` with `|p * 10^d| <= 4e15`.
  If no d in 0..12 makes all prices exact, the d with the fewest failures is
  used. Prices that fail this test are **irregular**: they are stored as raw
  64-bit patterns (escapes) and do not move the integer price state.
* `10^d` is computed by repeated multiplication by 10.0, so it is exact.
* Integer bid states start at `B_base` (ask: `A_base`). The price step `g` is the
  GCD of all differences between consecutive regular values of both sides
  (1 if none). The model works on `X = (B - (B_base mod g)) / g` (floored mod),
  which is exact, so tick sizes like 0.25 or 5 points cost nothing.
* The decoder reconstructs a regular price as `(double)(X * g + off) / 10^d`,
  which is the correctly rounded value and therefore bit-identical.

### 2.2 Time

`gt` is the GCD of all `|t[i] - t[i-1]|` (1 if all are equal), e.g. 1000 for
second-resolution history. Each tick codes `dt = (t[i] - t[i-1]) / gt`
(the first tick uses `t_base`, so its dt is 0). Negative and huge values are
legal and go through the raw escape described below.

### 2.3 Decimal sub-grid

Many feeds quote on a coarser grid with a constant markup, e.g. XAUUSD with 3
digits where almost every price ends in 5. If the entropy of `X mod 10` over
the chunk is below 2 bits (and N >= 256), M = 10 is used: every price state is
split into `q = floor(X / M)` and residue `r = X mod M`, and moves are coded as
a new residue plus a move of `q`. Otherwise M = 1 (no split).

### 2.4 Checksum

`h = 0`; for every tick, with `T` = time_msc, `B`/`A` = bit patterns of bid/ask:
`h = H(h ^ lo32(T), hi32(T))`, then the same for `B` and `A`, where `H` is the
32-bit mixing function `H2` of the reference code. The decoder verifies it, so
corruption and implementation mismatches are detected.

## 3. Tick coding

Every tick is coded as a sequence of binary decisions, each predicted by the
context-mixing model (section 4) and arithmetic coded. Decisions are grouped
in *segments* of 1-4 bits that form a small binary tree (node 1 = root); one
hash-table block of 16 counters serves one segment.

1. **Time delta** (`dt`, magnitude group 0):
   bit length `k` of `dt` as a 4-bit value `min(k, 15)`; 15 escapes to a
   second nibble `min(k - 15, 15)`; 15 again escapes to the raw 64-bit value
   as 16 nibbles (negative values always take this path). Then the `k - 1`
   bits below the leading one, MSB first, in nibbles; the first three nibbles
   use the full prefix as context, later ones its low 12 bits.
2. **Tick type**: tree of `bid changed`, `ask changed` and, if both changed,
   `both moved by the same amount`.
3. **Escapes** (only if flag bit 0): for each side that did not change, a flag;
   a flagged side is followed by its raw 64-bit pattern (16 nibbles).
4. **Bid move** (if changed): with M > 1 the new residue (4 bits); then the
   coarse move `dq`: when the residue changed, `dq == 0` is possible and a
   `nonzero` decision precedes the sign; the magnitude `|dq| - 1` is coded like
   the time delta (group 1).
5. **Ask move** (if changed and not equal to the bid move): new residue
   (M > 1); if the bid did not change, the coarse ask move (group 2), otherwise
   the change of the coarse spread `dqa - dqb` (group 3), which is much more
   predictable than the ask move itself.
6. **State update**: last two dt values and bit lengths, last three tick
   types, last two coarse moves per side, last spread change, EMAs of the dt
   bit length (rates 1/8 and 1/64), EMA of the bid move size, EMA of the
   coarse spread.

## 4. Model

* **Counters**: 32-bit `p22 << 10 | n` (22-bit probability, 10-bit count),
  update `p += (target - p) * 2 / (2n + 3)`, `n` saturates at 255.
* **Hash table**: `2^table_bits` counters in blocks of 16; slot 0 holds a
  32-bit check, slots 1-15 the tree nodes of a segment. Block lookup is 2-way
  associative; on a miss the block whose node 1 has the smaller count is
  replaced.
* **Contexts**: 4-10 per field, built from the coding state (previous dt and
  its bit length, EMAs, previous tick types, last moves of both sides, the
  current dt bucket, spread relative to its EMA and in absolute units,
  residues, hour of the previous tick, ...). See `CodeTick()` for the exact
  hashes.
* **Mixing**: inputs are `stretch(p)` of each context plus a bias. Mixer 1
  selects its weight set by the decision (class * 16 + node, 5376 sets),
  mixer 2 by a hash of the decision and a field-specific context (65536 sets),
  a final 2-input mixer per decision class combines both. Learning rates:
  8/16384 for the first layer, 6/16384 for the final mixer.
* **APM / SSE**: two interpolating APMs (33 buckets over the stretch domain,
  rate 1/128), one per decision, one per hashed decision context. The final
  probability is `(2 * p_mix + p_apm1 + p_apm2) / 4`, clamped to [1, 4095].
* **Arithmetic coder**: 32-bit carry-less binary coder,
  `xmid = x1 + ((x2 - x1) * p) >> 12` (64-bit product); on bit 1 `x2 = xmid`,
  else `x1 = xmid + 1`; leading equal bytes are shifted out. The encoder
  flushes the 4 bytes of `x1`; the decoder reads zeros past the end.

All model arithmetic is integer-only, so every platform produces the same
bytes. The self-test (`TkzSelfTest`) checks this against a golden hash.