260 lines
8.5 KiB
Markdown
260 lines
8.5 KiB
Markdown
<p align="center">
|
|
<img src="https://img.shields.io/badge/Language-MQL5-1B6CA8?style=flat-square"/>
|
|
<img src="https://img.shields.io/badge/Platform-MetaTrader%205-0D1B2A?style=flat-square"/>
|
|
<img src="https://img.shields.io/badge/Author-nique__372-C9D6DF?style=flat-square&logoColor=white"/>
|
|
<img src="https://img.shields.io/badge/MQL5.com-nique__372-1B6CA8?style=flat-square"/>
|
|
</p>
|
|
|
|
<p align="center">
|
|
A high-performance, memory-free JSON parser for MQL5, based on a flat tape architecture.<br/> Single-pass iterative state machine (as is, without extra loops, "pure" o(n)): no recursion, no dynamic memory fragmentation, no overhead from function calls.
|
|
</p>
|
|
|
|
---
|
|
|
|
## Main Features
|
|
|
|
- **Tape-based zero-alloc model**: the entire JSON is parsed into a single contiguous `long[]` array
|
|
- **Single flat loop**: one `switch` over a token enum, no helper function calls during parsing, strictly O(n)
|
|
- **JIT perfect-hash key lookup**: object access starts as linear probing (`StrEquals`); once a node passes `TSN_JSON_JIT_MIN_REF_TO_HASING` accesses, a minimal perfect hash table is built for it on the fly (`k = n`, no slack), promoting that node's lookups to O(1). No FNV hashing is computed upfront during parsing.
|
|
- **Handle-based navigation**: `CJsonNode` is a lightweight struct (pointer + two ints), copying is free
|
|
- **In-place mutation**: `SetInt`, `SetDbl`, `SetBool` modify the tape directly without re-parsing
|
|
- **Iterator support**: `CJsonIteratorObj` and `CJsonIteratorArray` for clean traversal
|
|
- **Three input formats, one API**: `AssingFile` auto-detects by extension — `.json` (plain text, needs `Parse()`), `.jsonasm` (assembly-like text dump of the tape, needs `ParseAssembly()`), `.jsontc` (precompiled binary tape, ready to use immediately, no parsing step)
|
|
- **Compiled tape cache (`.jsontc`)**: `SaveCompiled` persists the tape, the source JSON (or a reference to its file), and every perfect-hash table already built — so a reload via `AssingFile` skips parsing entirely and keeps any JIT-hashed nodes already O(1)
|
|
|
|
### Parse and navigate
|
|
|
|
```mql5
|
|
#include "Src\\JsonNode.mqh"
|
|
|
|
TSN::CJsonParser parser;
|
|
parser.Assing("{\"symbol\":\"EURUSD\",\"bid\":1.2345,\"ask\":1.2347,\"active\":true}");
|
|
parser.CorrectPadingForSwar();
|
|
parser.Parse();
|
|
|
|
TSN::CJsonNode root = parser.GetRoot();
|
|
string symbol = root["symbol"].ToString();
|
|
double bid = root["bid"].ToDouble(0.0);
|
|
bool active = root["active"].ToBool(false);
|
|
```
|
|
|
|
### Build json as json string or valid json with CJsonBuilder
|
|
|
|
```mql5
|
|
TSN::CJsonBuilderStr build;
|
|
build.PutChar('"');
|
|
build.Obj();
|
|
build.Key("Valor").ValS("└");
|
|
build.Key("asas").ValS("\r\n\v\t aaaaa");
|
|
uchar data[4] = {'h', 'o', 'l', 'a'};
|
|
build.KeyWV("valora").ValU(data);
|
|
build.EndObj();
|
|
build.PutChar('"');
|
|
Print(build.Build());
|
|
/*
|
|
"{\"Valor\":\"\\u2514\",\"asas\":\"\\r\\n\\v\\t aaaaa\",\"valora\":\"hola\"}"
|
|
*/
|
|
```
|
|
|
|
### Iterate an object
|
|
|
|
```mql5
|
|
TSN::CJsonIteratorObj it = root.BeginObj();
|
|
while(it.IsValid())
|
|
{
|
|
PrintFormat("%s : %s", it.Key(), it.Val().ToString());
|
|
it.Next();
|
|
}
|
|
```
|
|
|
|
### Iterate an array
|
|
|
|
```mql5
|
|
TSN::CJsonNode arr = root["prices"];
|
|
TSN::CJsonIteratorArray it = arr.BeginArr();
|
|
while(it.IsValid())
|
|
{
|
|
Print(it.Val().ToDouble(0.0));
|
|
it.Next();
|
|
}
|
|
```
|
|
|
|
### Parse from file
|
|
|
|
```mql5
|
|
TSN::CJsonParser parser;
|
|
parser.AssingFile("data.json", false);
|
|
parser.CorrectPadingForSwar();
|
|
parser.Parse();
|
|
```
|
|
|
|
### Parse a `.jsonasm` file (text dump of the tape)
|
|
|
|
`AssingFile` detects the `.jsonasm` extension and loads the raw text — you still need to call `ParseAssembly()` (instead of `Parse()`) to turn it into a tape:
|
|
|
|
```mql5
|
|
TSN::CJsonParser parser;
|
|
parser.AssingFile("data.jsonasm", false);
|
|
parser.CorrectPadingForSwar();
|
|
parser.ParseAssembly();
|
|
|
|
TSN::CJsonNode root = parser.GetRoot();
|
|
```
|
|
|
|
You can also go the other way and dump an already-parsed tape into `.jsonasm` text (useful for debugging/inspecting the tape):
|
|
|
|
```mql5
|
|
uchar asm_text[];
|
|
parser.ToJsonAsm(asm_text);
|
|
Print(CharArrayToString(asm_text));
|
|
```
|
|
|
|
### Compile and reload a `.jsontc` tape cache
|
|
|
|
`SaveCompiled` writes the parsed tape (plus the source JSON and every perfect-hash table already built) to a binary `.jsontc` file. Reloading it via `AssingFile` skips parsing entirely:
|
|
|
|
```mql5
|
|
//--- Compile once
|
|
TSN::CJsonParser parser;
|
|
parser.Assing(my_json_string);
|
|
parser.CorrectPadingForSwar();
|
|
parser.Parse();
|
|
parser.GetRoot()["symbol"]; // (optional) warm up perfect-hash tables before saving
|
|
parser.SaveCompiled("data.jsontc", false);
|
|
|
|
//--- Reload later — no Parse()/ParseAssembly() needed, tape and hash tables are restored as-is
|
|
TSN::CJsonParser parser2;
|
|
parser2.AssingFile("data.jsontc", false);
|
|
TSN::CJsonNode root = parser2.GetRoot();
|
|
```
|
|
|
|
There's also an overload that stores a reference to an external JSON file instead of embedding it:
|
|
|
|
```mql5
|
|
parser.SaveCompiled("data.jsontc", false, "data.json", false);
|
|
```
|
|
|
|
---
|
|
|
|
## Performance
|
|
|
|
Benchmark: `twitter.json` (616.7 KB), 1000 iterations. Same file, Iterations, and Machine (Laptop)
|
|
|
|
> Note: Here we measure how quickly the JSON is parsed.<br>
|
|
> All benchmarks are located in: Test\\Ben\\
|
|
|
|
| Parser | Language | Time (ms, total / 1000 iter) |
|
|
|--------|----------|-------------------------------|
|
|
| **simdjson::dom::parser** (reused parser) | C++ | **381** |
|
|
| **simdjson.Parser (reused parser)** | Python | **475** |
|
|
| **JsonParserByLeo** ASM (One copy of array - JSONASM File R460kb~) | MQL5 | **652-653** Stable |
|
|
| **JsonParserByLeo** (One copy of array) | MQL5 | **662 - 667** |
|
|
| fast_json v3.4 (One copy of array) | MQL5 | 1229 - 1300 |
|
|
| YamlParserByLeo (One copy of array) | MQL5 | 1300-1304 |
|
|
| Claude Opus 4.8 (Effort=Max) (Best Ai now) (Extended reasoning activated) Generating code (Fast json lib) (One copy of array) | MQL5 | 1312-1313 Stable |
|
|
| orjson | Python | 1863 |
|
|
| ding9736/MQL5-JsonLib (One copy of string) (no dom building only tape parsing) | MQL5 | 2145-2163 |
|
|
| ryml (Pure parse time) | C++ | 2846.55 |
|
|
| MQL5 Lite (goose) Metaeditor AI Model | MQL5 | 3242-3250 |
|
|
| simdjson.Parser + as_dict | Python | 4309 |
|
|
| simdjson.loads | Python | 4955 |
|
|
| ujson | Python | 5322 |
|
|
| json (stdlib) | Python | 5860 |
|
|
| JAson (One copy of array) | MQL5 | ~(19070-19220) |
|
|
| ToyJson3 (One copy of string) Only tokenization | MQL5 | ~(24600) Stable |
|
|
| CJsonNode (One copy of array) | MQL5 | ~(76720-80470) |
|
|
|
|
|
|
|
|
## Performance in acceses
|
|
Benchmark: `test.json`, 1000 iterations. Same file, Iterations, and Machine (Laptop)
|
|
|
|
> Note: Here we measure how fast access to a node: `root["key..."]`.<br>
|
|
> All benchmarks are located in: Test\\All\\
|
|
|
|
| Parser | Language | Time (microseconds, total / 1000 iter) |
|
|
|--------|----------|-------------------------------|
|
|
| **JsonParserByLeo** | MQL5 | **161-163** |
|
|
| fast_json v3.4 | MQL5 | 223 Stable |
|
|
| Claude Opus 4.8 (Effort=Max) (Best Ai now) (Extended reasoning activated) Generating code (Fast json lib) (One copy of array) | MQL5 | 595-605 |
|
|
|
|
|
|
## Performance notes
|
|
- Machine: (Laptop Lenovo ~2017, Windows 10 build 19045, Intel Core i5-8250U @ 1.60GHz, AVX2, GMT-5).
|
|
- MQL5 run: MetaTrader 5 x64 build 5836
|
|
- Python/C++ runs: Same machine (py = 3.10.9) and C++ (CXX23~)
|
|
|
|
|
|
---
|
|
|
|
## Repository Structure
|
|
|
|
```
|
|
JsonParserByLeo/
|
|
├── Src/ # Full code (Defines, Node, Parser)
|
|
└── Test/ # Test and Benchmarks (vs)
|
|
```
|
|
|
|
---
|
|
|
|
## License
|
|
|
|
**[Read Full License](./LICENSE)**
|
|
By downloading or using this repository, you accept the license terms.
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
See [dependencies.json](./dependencies.json) for the full list.
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
cd "C:\Users\YOUR_USER\AppData\Roaming\MetaQuotes\Terminal\YOUR_ID\MQL5\Shared Projects"
|
|
tsndep install "https://forge.mql5.io/nique_372/JsonParserByLeo.git"
|
|
```
|
|
|
|
Requires the `tsndep` package, available on [PyPI](https://pypi.org/project/tsndep). It automatically downloads and installs all declared dependencies.
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
**1. Include the library:**
|
|
|
|
```mql5
|
|
#include "..\\JsonParserByLeo\\Src\\JsonNode.mqh"
|
|
```
|
|
|
|
**2. Parse and access:**
|
|
|
|
```mql5
|
|
TSN::CJsonParser parser;
|
|
parser.Assing(my_json_string);
|
|
parser.CorrectPadingForSwar();
|
|
if(parser.Parse())
|
|
{
|
|
TSN::CJsonNode root = parser.GetRoot();
|
|
double price = root["price"].ToDouble(0.0);
|
|
}
|
|
```
|
|
|
|
**3. Re-parse without re-copying (benchmark pattern):**
|
|
|
|
```mql5
|
|
parser.Assing(raw_string); // copy once
|
|
parser.CorrectPadingForSwar();
|
|
for(int i = 0; i < 1000; i++)
|
|
parser.Parse(); // re-parse in-place
|
|
```
|
|
|
|
---
|
|
|
|
## Contact
|
|
|
|
- **Platform:** [MQL5 Community](https://www.mql5.com/es/users/nique_372)
|
|
- **Profile:** https://www.mql5.com/es/users/nique_372
|
|
- **Articles:** https://www.mql5.com/es/users/nique_372/publications
|