A high-performance, memory-free JSON parser for MQL5, based on a flat tape architecture.<br/> Single-pass iterative state machine (as is, without extra loops, "pure" o(n)): no recursion, no dynamic memory fragmentation, no overhead from function calls.
2026-06-03 09:20:57 -05:00
</p>
---
## Main Features
2026-10-18 16:22:57 -05:00
- **Tape-based zero-alloc model**: the entire JSON is parsed into a single contiguous `long[]` array (numbers, booleans and nulls are materialized eagerly; only keys and strings are stored lazily as offset + length)
- **Single flat loop**: one `switch` over a token enum, no helper function calls during parsing, strictly O(n), with branch-less padding sentinels to keep the hot path as short as possible
- **Two parsing engines**: a pure-MQL5 engine (`Parse()`, no external dependencies) and an optional SIMD engine that runs through a companion C++ DLL (`ParseSimd()`, AVX2, see [SIMD via DLL](#simd-via-dll) below) — same tape output either way, pick whichever fits your deployment
2026-10-18 16:22:57 -05:00
- **Strict validation**: numbers are fully validated (leading zeros, exponent and decimal form), `true`/`false`/`null` are checked completely (not just skipped), strings/keys must be closed, and grammar order is enforced by the state machine. Detailed guarantees in [SPEC.md](./SPEC.md)
- **Safe number handling**: full 64-bit unsigned support (`ulong`, no precision loss on large ids), and any integer or real the parser can't represent exactly is kept as a string (`ToString()`) so you can use a big-number library instead of silently losing digits
- **JIT perfect-hash key lookup**: object access starts as a linear scan; once an object (with enough keys) has been accessed enough times, a minimal perfect hash table is built for it on the fly (powered by [PerfectHashByLeo](https://forge.mql5.io/nique_372/PerfectHashByLeo)), promoting that object's lookups to O(1). Nothing is hashed upfront during parsing, small objects/arrays never trigger it, and the thresholds are configurable via defines (`TSN_NODEBASEREAD_MIN_JIT_ATTEMPS_TO_PHASH`, `TSN_NODEBASEREAD_MIN_JIT_VALUES_TO_HASH`) or disabled entirely with `Options()`
2026-06-03 09:20:57 -05:00
- **Handle-based navigation**: `CJsonNode` is a lightweight struct (pointer + two ints), copying is free
- **Iterator support**: `CJsonIteratorObj` and `CJsonIteratorArray` for clean traversal
2026-10-18 16:22:57 -05:00
- **On-demand unescape**: keys and strings are unescaped only when you read them (`node["\n"]` works, emojis and `\uXXXX` surrogates included); invalid escapes are reported as errors instead of being silently ignored
- **Mutable DOM API**: `CDomNodeBase` builds a real, editable node tree from the tape (`ToDom`) — add/remove keys, change a value's type in place, slice arrays Python-style (`PySlicing`), serialize back to JSON (`CJsonDomSerializer`), and load/dump native MQL5 structs directly (`Load<T>()` / `GetAs<T>()`)
2026-10-18 16:22:57 -05:00
- **SAX streaming parser**: `CJsonParserStream` + `ITsnJsonParserStream` feed the tape incrementally in fixed-size chunks (e.g. 64 bytes at a time) and fire `OnStartObj`/`OnKey`/`OnInteger`/... callbacks as it goes — no need to hold the whole document, or the whole tape, in memory at once. Numbers too big to convert safely are delivered raw through `OnBigInteger`/`OnBigDecimal`
- **JSON Builder**: `CJsonBuilder`/`CJsonBuilderStr` for constructing valid JSON (or an escaped JSON string literal) directly, without an intermediate DOM
2026-10-18 16:22:57 -05:00
- **JSON Minify**: whitespace stripping for existing documents
- **JSON Pointer (RFC 6901)**: `Query("/a/0/b~1c")`-style lookups on both `CJsonNode` (tape) and `CDomNodeBase` (DOM), precompiled once with `BSFLPointerCompiler`
- **JSONPath-like query VM**: a small register-based virtual machine (`CSLDomQueryVm`) compiles an assembly-like query language (`INS_ACCESO_KEY`, `INS_COMPARE_*`, `INS_ITER_ASSING`, custom `INS_CALL`-able functions, ...) to filter/traverse a DOM tree — think `$.libros[?(@.valor > 100)]` compiled down to a tiny bytecode loop
- **Two input formats, one API**: `AssingFile` auto-detects by extension — `.json` (plain text, needs `Parse()`) and `.jsontc` (precompiled binary tape, ready to use immediately, no parsing step)
2026-06-20 17:39:08 -05:00
- **Compiled tape cache (`.jsontc`)**: `SaveCompiled` persists the tape, the source JSON (or a reference to its file), and every perfect-hash table already built — so a reload via `AssingFile` skips parsing entirely and keeps any JIT-hashed nodes already O(1)
2026-10-18 16:22:57 -05:00
- **Modular packages**: include only what you need — `JsonNode.mqh` (parser + node + DOM), `Json.mqh` (+ serializer, SAX, builders, pointer, minify) or `JsonFull.mqh` (+ JsonPath VM). See [Src/README.md](./Src/README.md)
- **100+ unit tests** in `Wf/` covering node access, the builder, DOM mutation, `.jsontc` round-trips and malformed-input handling — runnable as a standalone script inside MetaTrader
`SaveCompiled` writes the parsed tape (plus the source JSON and every perfect-hash table already built) to a binary `.jsontc` file. Reloading it via `AssingFile` skips parsing entirely:
```mql5
//--- Compile once
TSN::CJsonParser parser;
parser.Assing(my_json_string);
2026-07-19 15:56:36 -05:00
parser.CorrectPadding();
2026-06-20 17:39:08 -05:00
parser.Parse();
parser.GetRoot()["symbol"]; // (optional) warm up perfect-hash tables before saving
parser.SaveCompiled("data.jsontc", false);
//--- Reload later — no Parse()/ParseAssembly() needed, tape and hash tables are restored as-is
TSN::CJsonParser parser2;
parser2.AssingFile("data.jsontc", false);
TSN::CJsonNode root = parser2.GetRoot();
```
There's also an overload that stores a reference to an external JSON file instead of embedding it:
For MQL5, `Parse()` is fast but limited to what a script/EA can do in pure MQL5. `ParseSimd()` offloads phase 1 (finding every structural character: `{`, `}`, `[`, `]`, `:`, `,`, string bounds) to a small companion C++ DLL that uses AVX2 (SWAR bit-tricks for backslash/quote detection, `pclmulqdq` for the in-string prefix-xor, `popcnt`/`tzcnt` to walk set bits) — phase 2 (building the tape) still runs the same flat state machine. Same `long[]` tape as `Parse()`; only the structural scan changes.
Requires the `tsndep` package, available on [PyPI](https://pypi.org/project/tsndep). It automatically downloads and installs all declared dependencies.