| # TKZ: tick history compression for MetaTrader 5
|
|
|
| TKZ is an MQL5 library and a set of scripts that store MetaTrader 5 tick
|
| history, meaning `time_msc`, `bid` and `ask`, in as few bytes as possible.
|
| Compression is lossless and the round trip is bit-exact.
|
|
|
| It uses a context-mixing compressor built for the structure of quote streams:
|
|
|
| * integer price mapping with automatic digits, tick size and decimal sub-grid
|
| detection;
|
| * a binary decomposition of time deltas and price moves;
|
| * about ten adaptive contexts per decision;
|
| * a two-layer logistic mixer, two SSE stages and an arithmetic coder.
|
|
|
| On real tick data it takes **1.37–2.2 bytes per tick**, 11–17× smaller than
|
| 24-byte `(time_msc, bid, ask)` records and 27–44× smaller than `MqlTick`. That
|
| is 16–26 % smaller than `xz -9e` applied to already delta-encoded data.
|
|
|
| ## Results
|
|
|
| Bytes per tick, whole file as one chunk. The baselines are the best of
|
| `xz -9e` and `zstd -19 --long`, run on raw records and on delta-encoded
|
| columns (`tests/data/baselines.py`).
|
|
|
| | Data set | Ticks | Raw records | xz, raw | xz, deltas + varint columns | **TKZ** | vs best baseline |
|
| |---|---:|---:|---:|---:|---:|---:|
|
| | Dukascopy EURUSD, Nov 2023 | 1 938 713 | 24 | 3.077 | 1.743 | **1.460** | −16 % |
|
| | Dukascopy XAUUSD, Sep 2023 | 2 391 711 | 24 | 2.956 | 1.687 | **1.369** | −19 % |
|
| | Broker XAUUSD feed, May 2026 | 687 853 | 24 | 3.908 | 1.905 | **1.445** | −24 % |
|
| | US500 CFD (Tickstory), Feb 2025 | 199 851 | 24 | 4.931 | 2.993 | **2.201** | −26 % |
|
|
|
| In forex, about 70 % of the output is the tick timestamps. Bid and ask
|
| together cost about 3.3 bits per tick on EURUSD.
|
|
|
| Effect of the chunk size (all four sets combined):
|
|
|
| | Ticks per chunk | 20 000 | 100 000 (≈ a day of EURUSD) | 500 000 (≈ a week) | whole month |
|
| |---|---:|---:|---:|---:|
|
| | bytes/tick | 1.485 | 1.459 | 1.448 | 1.445 |
|
|
|
| Speed of the C++ build of the same code (bounds-checked arrays, one core):
|
| encoding and decoding are symmetric at about 250 000 ticks/s. MQL5 is usually
|
| somewhat slower than optimised C++. `TkzSelfTest` and `TkzExport` print the real
|
| speed in your terminal. Memory: about 25 MB of model tables per codec instance
|
| (a writer with verification uses two), plus about 40 bytes per tick of the
|
| chunk being encoded.
|
|
|
| ## Installation
|
|
|
| Copy the `MQL5` folder of this repository into the terminal data folder
|
| (MetaEditor: *File → Open Data Folder*), then compile the scripts in
|
| `MQL5/Scripts/Tkz/` in MetaEditor.
|
|
|
| ```
|
| MQL5/Include/Tkz/TkzCodec.mqh codec: one chunk <-> ticks
|
| MQL5/Include/Tkz/TkzArchive.mqh .tkz files: CTkzWriter, CTkzReader
|
| MQL5/Include/Tkz/TkzSelfTest.mqh self-test used by the script and the C++ tests
|
| MQL5/Scripts/Tkz/TkzSelfTest.mq5 run this first
|
| MQL5/Scripts/Tkz/TkzExport.mq5 history -> .tkz
|
| MQL5/Scripts/Tkz/TkzVerify.mq5 .tkz -> compare with history, measure decoding
|
| MQL5/Scripts/Tkz/TkzToCustom.mq5 .tkz -> custom symbol (charts, Strategy Tester)
|
| ```
|
|
|
| ## Scripts
|
|
|
| 1. **TkzSelfTest**: synthetic edge cases (NaN/Inf/-0.0, negative time steps,
|
| huge gaps, tick grids, negative prices, corruption), a byte-exact *golden*
|
| comparison with the reference implementation, a round trip of the latest
|
| ticks of the chart symbol, and archive I/O. Every line must say `PASS`.
|
| 2. **TkzExport**: inputs are the symbol, the date range, the chunk period
|
| (day/week/month), the tick selection (`COPY_TICKS_ALL` or
|
| `COPY_TICKS_INFO`), the file name, the common folder and the verification
|
| switch. Ticks are fetched day by day with `CopyTicksRange`. Each chunk is
|
| compressed and, with verification on, decoded again and compared before it
|
| is written. The script prints the size, bytes/tick, the compression ratios
|
| and the speed.
|
| 3. **TkzVerify**: decodes every chunk, compares it with `CopyTicksRange` over
|
| the same time span and reports the decoding speed.
|
| 4. **TkzToCustom**: creates a custom symbol (by default `<symbol>.tkz`,
|
| properties copied from the original symbol) and loads the archive with
|
| `CustomTicksReplace`.
|
|
|
| The files are written to `MQL5/Files` (or the common `Files` folder).
|
|
|
| ## API
|
|
|
| ```mql5
|
| #include <Tkz/TkzArchive.mqh>
|
|
|
| // --- one chunk in memory
|
| CTkzCodec codec;
|
| uchar blob[];
|
| int size = codec.EncodeTicks(ticks, 0, ArraySize(ticks), blob); // appends a chunk
|
| MqlTick restored[];
|
| int n = codec.DecodeTicks(blob, 0, restored); // -1 on error
|
| // also: Encode()/Decode() with parallel arrays long time_msc[], double bid[], ask[]
|
|
|
| // --- archive files
|
| CTkzWriter w;
|
| w.SetVerify(true); // optional: decode and compare each chunk
|
| w.Open("EURUSD.tkz", "EURUSD", 5, false); // append=true adds chunks to an existing file
|
| w.WriteTicks(ticks, 0, ArraySize(ticks)); // one call = one chunk
|
| w.Close();
|
|
|
| CTkzReader r;
|
| r.Open("EURUSD.tkz", false);
|
| r.ReadRange(from_msc, to_msc, ticks); // decodes only the chunks that overlap
|
| r.ReadChunk(i, ticks); r.ReadAll(ticks); r.GetChunkInfo(i, info);
|
| ```
|
|
|
| Decoded `MqlTick`s have `time`, `time_msc`, `bid` and `ask` set, plus
|
| `TICK_FLAG_BID`/`TICK_FLAG_ASK` where the value changed. `last` and volumes are
|
| not stored and are zero.
|
|
|
| ## Format
|
|
|
| [docs/FORMAT.md](docs/FORMAT.md) describes the file and chunk layout,
|
| preprocessing, binarisation and model. In short:
|
|
|
| * every chunk is independent: an 88-byte header plus the payload, with a
|
| checksum of the decoded ticks;
|
| * the archive is a 64-byte file header followed by chunks;
|
| * all model arithmetic is integer-only, so archives are portable between
|
| machines and builds.
|
|
|
| ## Testing without MetaTrader
|
|
|
| The MQL5 sources are translated to C++ by `tests/mql2cpp.py` and compiled
|
| against `tests/mql5_shim.h`. The shim is a small emulation of the MQL5
|
| runtime: arrays are bounds-checked, memory from `ArrayResize` is poisoned,
|
| and terminal functions run over a tick file. This is the exact code that runs
|
| in the terminal:
|
|
|
| * `test_codec`: synthetic edge cases, bit-exact round trips of real data,
|
| determinism, archives (append, range queries), 3000 fuzzed chunks
|
| (rejected or decoded identically, never a crash), and the `TkzSelfTest.mqh`
|
| suite;
|
| * `test_scripts`: runs `TkzExport` (day/week/month chunks, checked against
|
| independent calendar periods), `TkzVerify`, `TkzToCustom` and `TkzSelfTest`
|
| end to end on an emulated terminal.
|
|
|
| ```sh
|
| tests/data/fetch_samples.sh # optional: download the benchmark samples (~36 MB)
|
| CXX="c++" tests/run_tests.sh tests/build/ticks/*.ticks
|
| ```
|
|
|
| Any C++17 compiler works (`CXX="zig c++"` also works).
|
|
|
| The golden check of `TkzSelfTest` encodes three deterministic chunks and
|
| compares the bytes with this reference build. The chunks include negative
|
| prices, crossed quotes, escapes, a decimal sub-grid, and time and price
|
| steps. A `PASS` in MetaTrader means the terminal produces exactly the same
|
| archives.
|
