TKZ: tick history compression for MetaTrader 5

TKZ is an MQL5 library and a set of scripts that store MetaTrader 5 tick history, meaning time_msc, bid and ask, in as few bytes as possible. Compression is lossless and the round trip is bit-exact.

It uses a context-mixing compressor built for the structure of quote streams:

• integer price mapping with automatic digits, tick size and decimal sub-grid detection;
• a binary decomposition of time deltas and price moves;
• about ten adaptive contexts per decision;
• a two-layer logistic mixer, two SSE stages and an arithmetic coder.

On real tick data it takes 1.37–2.2 bytes per tick, 11–17× smaller than 24-byte (time_msc, bid, ask) records and 27–44× smaller than MqlTick. That is 16–26 % smaller than xz -9e applied to already delta-encoded data.

Results

Bytes per tick, whole file as one chunk. The baselines are the best of xz -9e and zstd -19 --long, run on raw records and on delta-encoded columns (tests/data/baselines.py).

Data set	Ticks	Raw records	xz, raw	xz, deltas + varint columns	TKZ	vs best baseline
Dukascopy EURUSD, Nov 2023	1 938 713	24	3.077	1.743	1.460	−16 %
Dukascopy XAUUSD, Sep 2023	2 391 711	24	2.956	1.687	1.369	−19 %
Broker XAUUSD feed, May 2026	687 853	24	3.908	1.905	1.445	−24 %
US500 CFD (Tickstory), Feb 2025	199 851	24	4.931	2.993	2.201	−26 %

In forex, about 70 % of the output is the tick timestamps. Bid and ask together cost about 3.3 bits per tick on EURUSD.

Effect of the chunk size (all four sets combined):

Ticks per chunk	20 000	100 000 (≈ a day of EURUSD)	500 000 (≈ a week)	whole month
bytes/tick	1.485	1.459	1.448	1.445

Speed of the C++ build of the same code (bounds-checked arrays, one core): encoding and decoding are symmetric at about 250 000 ticks/s. MQL5 is usually somewhat slower than optimised C++. TkzSelfTest and TkzExport print the real speed in your terminal. Memory: about 25 MB of model tables per codec instance (a writer with verification uses two), plus about 40 bytes per tick of the chunk being encoded.

Installation

Copy the MQL5 folder of this repository into the terminal data folder (MetaEditor: File → Open Data Folder), then compile the scripts in MQL5/Scripts/Tkz/ in MetaEditor.

MQL5/Include/Tkz/TkzCodec.mqh     codec: one chunk <-> ticks
MQL5/Include/Tkz/TkzArchive.mqh   .tkz files: CTkzWriter, CTkzReader
MQL5/Include/Tkz/TkzSelfTest.mqh  self-test used by the script and the C++ tests
MQL5/Scripts/Tkz/TkzSelfTest.mq5  run this first
MQL5/Scripts/Tkz/TkzExport.mq5    history -> .tkz
MQL5/Scripts/Tkz/TkzVerify.mq5    .tkz -> compare with history, measure decoding
MQL5/Scripts/Tkz/TkzToCustom.mq5  .tkz -> custom symbol (charts, Strategy Tester)

Scripts

1. TkzSelfTest: synthetic edge cases (NaN/Inf/-0.0, negative time steps, huge gaps, tick grids, negative prices, corruption), a byte-exact golden comparison with the reference implementation, a round trip of the latest ticks of the chart symbol, and archive I/O. Every line must say PASS.
2. TkzExport: inputs are the symbol, the date range, the chunk period (day/week/month), the tick selection (COPY_TICKS_ALL or COPY_TICKS_INFO), the file name, the common folder and the verification switch. Ticks are fetched day by day with CopyTicksRange. Each chunk is compressed and, with verification on, decoded again and compared before it is written. The script prints the size, bytes/tick, the compression ratios and the speed.
3. TkzVerify: decodes every chunk, compares it with CopyTicksRange over the same time span and reports the decoding speed.
4. TkzToCustom: creates a custom symbol (by default <symbol>.tkz, properties copied from the original symbol) and loads the archive with CustomTicksReplace.

The files are written to MQL5/Files (or the common Files folder).

API

#include <Tkz/TkzArchive.mqh>

// --- one chunk in memory
CTkzCodec codec;
uchar     blob[];
int size = codec.EncodeTicks(ticks, 0, ArraySize(ticks), blob);  // appends a chunk
MqlTick   restored[];
int n = codec.DecodeTicks(blob, 0, restored);                   // -1 on error
// also: Encode()/Decode() with parallel arrays long time_msc[], double bid[], ask[]

// --- archive files
CTkzWriter w;
w.SetVerify(true);                       // optional: decode and compare each chunk
w.Open("EURUSD.tkz", "EURUSD", 5, false); // append=true adds chunks to an existing file
w.WriteTicks(ticks, 0, ArraySize(ticks)); // one call = one chunk
w.Close();

CTkzReader r;
r.Open("EURUSD.tkz", false);
r.ReadRange(from_msc, to_msc, ticks);     // decodes only the chunks that overlap
r.ReadChunk(i, ticks);  r.ReadAll(ticks); r.GetChunkInfo(i, info);

Decoded MqlTicks have time, time_msc, bid and ask set, plus TICK_FLAG_BID/TICK_FLAG_ASK where the value changed. last and volumes are not stored and are zero.

Format

docs/FORMAT.md describes the file and chunk layout, preprocessing, binarisation and model. In short:

• every chunk is independent: an 88-byte header plus the payload, with a checksum of the decoded ticks;
• the archive is a 64-byte file header followed by chunks;
• all model arithmetic is integer-only, so archives are portable between machines and builds.

Testing without MetaTrader

The MQL5 sources are translated to C++ by tests/mql2cpp.py and compiled against tests/mql5_shim.h. The shim is a small emulation of the MQL5 runtime: arrays are bounds-checked, memory from ArrayResize is poisoned, and terminal functions run over a tick file. This is the exact code that runs in the terminal:

• test_codec: synthetic edge cases, bit-exact round trips of real data, determinism, archives (append, range queries), 3000 fuzzed chunks (rejected or decoded identically, never a crash), and the TkzSelfTest.mqh suite;
• test_scripts: runs TkzExport (day/week/month chunks, checked against independent calendar periods), TkzVerify, TkzToCustom and TkzSelfTest end to end on an emulated terminal.

tests/data/fetch_samples.sh            # optional: download the benchmark samples (~36 MB)
CXX="c++" tests/run_tests.sh tests/build/ticks/*.ticks

Any C++17 compiler works (CXX="zig c++" also works).

The golden check of TkzSelfTest encodes three deterministic chunks and compares the bytes with this reference build. The chunks include negative prices, crossed quotes, escapes, a decimal sub-grid, and time and price steps. A PASS in MetaTrader means the terminal produces exactly the same archives.