mql5-tick-compression/README.md

# TKZ: tick history compression for MetaTrader 5

TKZ is an MQL5 library and a set of scripts that store MetaTrader 5 tick
history, meaning `time_msc`, `bid` and `ask`, in as few bytes as possible.
Compression is lossless and the round trip is bit-exact.

It uses a context-mixing compressor built for the structure of quote streams:

* integer price mapping with automatic digits, tick size and decimal sub-grid
  detection;
* a binary decomposition of time deltas and price moves;
* about ten adaptive contexts per decision;
* a two-layer logistic mixer, two SSE stages and an arithmetic coder.

On real tick data it takes **1.37–2.2 bytes per tick**, 11–17× smaller than
24-byte `(time_msc, bid, ask)` records and 27–44× smaller than `MqlTick`. That
is 16–26 % smaller than `xz -9e` applied to already delta-encoded data.

## Results

Bytes per tick, whole file as one chunk. The baselines are the best of
`xz -9e` and `zstd -19 --long`, run on raw records and on delta-encoded
columns (`tests/data/baselines.py`).

| Data set | Ticks | Raw records | xz, raw | xz, deltas + varint columns | **TKZ** | vs best baseline |
|---|---:|---:|---:|---:|---:|---:|
| Dukascopy EURUSD, Nov 2023 | 1 938 713 | 24 | 3.077 | 1.743 | **1.460** | −16 % |
| Dukascopy XAUUSD, Sep 2023 | 2 391 711 | 24 | 2.956 | 1.687 | **1.369** | −19 % |
| Broker XAUUSD feed, May 2026 | 687 853 | 24 | 3.908 | 1.905 | **1.445** | −24 % |
| US500 CFD (Tickstory), Feb 2025 | 199 851 | 24 | 4.931 | 2.993 | **2.201** | −26 % |

In forex, about 70 % of the output is the tick timestamps. Bid and ask
together cost about 3.3 bits per tick on EURUSD.

Effect of the chunk size (all four sets combined):

| Ticks per chunk | 20 000 | 100 000 (≈ a day of EURUSD) | 500 000 (≈ a week) | whole month |
|---|---:|---:|---:|---:|
| bytes/tick | 1.485 | 1.459 | 1.448 | 1.445 |

Speed of the C++ build of the same code (bounds-checked arrays, one core):
encoding and decoding are symmetric at about 250 000 ticks/s. MQL5 is usually
somewhat slower than optimised C++. `TkzSelfTest` and `TkzExport` print the real
speed in your terminal. Memory: about 25 MB of model tables per codec instance
(a writer with verification uses two), plus about 40 bytes per tick of the
chunk being encoded.

## Installation

Clone the repository into the `MQL5/Shared Projects` folder of the terminal
data folder (MetaEditor: *File → Open Data Folder*):

```sh
cd "<data folder>/MQL5/Shared Projects"
git clone https://forge.mql5.io/boyvlad/mql5-tick-compression.git
```

An unpacked download of the repository in the same place works too. Then
compile the scripts in `scripts/` in MetaEditor. MetaEditor puts the compiled
programs into `MQL5/Scripts/Shared Projects`, so they appear in the terminal
Navigator under *Scripts → Shared Projects*. No `.mqproj` file is needed, and
all includes are relative, so the name of the folder does not matter.

```
lib/TkzCodec.mqh         codec: one chunk <-> ticks
lib/TkzArchive.mqh       .tkz files: CTkzWriter, CTkzReader
lib/TkzSelfTest.mqh      self-test used by the script and the C++ tests
scripts/TkzSelfTest.mq5  run this first
scripts/TkzExport.mq5    history -> .tkz
scripts/TkzVerify.mq5    .tkz -> compare with history, measure decoding
scripts/TkzToCustom.mq5  .tkz -> custom symbol (charts, Strategy Tester)
```

## Scripts

1. **TkzSelfTest**: synthetic edge cases (NaN/Inf/-0.0, negative time steps,
   huge gaps, tick grids, negative prices, corruption), a byte-exact *golden*
   comparison with the reference implementation, a round trip of the latest
   ticks of the chart symbol, and archive I/O. Every line must say `PASS`.
2. **TkzExport**: inputs are the symbol, the date range, the chunk period
   (day/week/month), the tick selection (`COPY_TICKS_ALL` or
   `COPY_TICKS_INFO`), the file name, the common folder and the verification
   switch. Ticks are fetched day by day with `CopyTicksRange`. Each chunk is
   compressed and, with verification on, decoded again and compared before it
   is written. The script prints the size, bytes/tick, the compression ratios
   and the speed.
3. **TkzVerify**: decodes every chunk, compares it with `CopyTicksRange` over
   the same time span and reports the decoding speed.
4. **TkzToCustom**: creates a custom symbol (by default `<symbol>.tkz`,
   properties copied from the original symbol) and loads the archive with
   `CustomTicksReplace`.

The files are written to `MQL5/Files` (or the common `Files` folder).

## API

The library is included with a path relative to the including file, for
example from a program at the root of another project in `MQL5/Shared Projects`:

```mql5
#include "../mql5-tick-compression/lib/TkzArchive.mqh"

// --- one chunk in memory
CTkzCodec codec;
uchar     blob[];
int size = codec.EncodeTicks(ticks, 0, ArraySize(ticks), blob);  // appends a chunk
MqlTick   restored[];
int n = codec.DecodeTicks(blob, 0, restored);                   // -1 on error
// also: Encode()/Decode() with parallel arrays long time_msc[], double bid[], ask[]

// --- archive files
CTkzWriter w;
w.SetVerify(true);                       // optional: decode and compare each chunk
w.Open("EURUSD.tkz", "EURUSD", 5, false); // append=true adds chunks to an existing file
w.WriteTicks(ticks, 0, ArraySize(ticks)); // one call = one chunk
w.Close();

CTkzReader r;
r.Open("EURUSD.tkz", false);
r.ReadRange(from_msc, to_msc, ticks);     // decodes only the chunks that overlap
r.ReadChunk(i, ticks);  r.ReadAll(ticks); r.GetChunkInfo(i, info);
```

Decoded `MqlTick`s have `time`, `time_msc`, `bid` and `ask` set, plus
`TICK_FLAG_BID`/`TICK_FLAG_ASK` where the value changed. `last` and volumes are
not stored and are zero.

## Format

[docs/FORMAT.md](docs/FORMAT.md) describes the file and chunk layout,
preprocessing, binarisation and model. In short:

* every chunk is independent: an 88-byte header plus the payload, with a
  checksum of the decoded ticks;
* the archive is a 64-byte file header followed by chunks;
* all model arithmetic is integer-only, so archives are portable between
  machines and builds.

## Testing without MetaTrader

The MQL5 sources are translated to C++ by `tests/mql2cpp.py` and compiled
against `tests/mql5_shim.h`. The shim is a small emulation of the MQL5
runtime: arrays are bounds-checked, memory from `ArrayResize` is poisoned,
and terminal functions run over a tick file. This is the exact code that runs
in the terminal:

* `test_codec`: synthetic edge cases, bit-exact round trips of real data,
  determinism, archives (append, range queries), 3000 fuzzed chunks
  (rejected or decoded identically, never a crash), and the `TkzSelfTest.mqh`
  suite;
* `test_scripts`: runs `TkzExport` (day/week/month chunks, checked against
  independent calendar periods), `TkzVerify`, `TkzToCustom` and `TkzSelfTest`
  end to end on an emulated terminal.

```sh
tests/data/fetch_samples.sh            # optional: download the benchmark samples (~36 MB)
CXX="c++" tests/run_tests.sh tests/build/ticks/*.ticks
```

Any C++17 compiler works (`CXX="zig c++"` also works).

The translator resolves every `#include` the way MetaEditor does in
`Shared Projects`: the path must be quoted and point to an existing file
relative to the including file. A broken include path therefore fails here,
not only in MetaEditor.

The golden check of `TkzSelfTest` encodes three deterministic chunks and
compares the bytes with this reference build. The chunks include negative
prices, crossed quotes, escapes, a decimal sub-grid, and time and price
steps. A `PASS` in MetaTrader means the terminal produces exactly the same
archives.