Base repository for parsers of structured languages (JSON, YAML, TOML, or custom formats) in MQL5.
Provides an abstract parser, a read-only node/navigation layer, a mutable DOM, a precompiled pointer system (JSON Pointer), a shared unescape engine and a small query VM — all shared infrastructure that concrete parsers ( like JsonParserByLeo) extend rather than reimplement.

--- ## Main Features This repository does not parse any format by itself. It provides the building blocks a concrete "structured language" parser is built from, and every layer is meant to be extended, not used directly. - **Abstract parser base (`CBaseStructuredLan`, `BaseClass/Main.mqh`)**: holds the parsed data as a flat "cinta" (a `ulong[]` tape) plus the raw `uchar[]` buffer, owns the perfect-hash tables used for O(1) key lookups (`m_tables`, built with [PerfectHashByLeo](https://forge.mql5.io/nique_372/PerfectHashByLeo)), and handles saving/loading a fully compiled representation (tape + tables) to a binary cache file (`SaveCompiled` / `AssingFile`, with the raw text either embedded or referenced by path). It also normalizes UTF-16 / UTF-8 BOM input (`NormalizeUft`). A concrete parser extends this class and implements `CalcRegType` (how to detect the file type) plus its own lexer/tokenizer that fills the tape (see JsonParserByLeo's `CJsonParser : public CBaseStructuredLan`). - **Read-only node/navigation layer (`CNodeSFLBase`, `Node/NodeBase.mqh`)**: generic template for walking the tape without copying data — `operator[]` / `Get(hash)` by key, `At` / `AtObj` by index, `HasKey`, `GetKeys`, `ToArray`, `ExistInArray`, array/object iterators (`BeginArr`, `BeginArrHomogeneo`, `BeginObj`), raw token movement, and JIT perfect-hashing (an object is scanned linearly while its key hashes are cached lazily on the tape; if it keeps being accessed, a perfect-hash table is built for it on the fly). A concrete format defines its own node type on top, e.g. `CJsonNode : CNodeSFLBase`. The exact contract `TCtx` and `TOut` must fulfill is documented in [`Src/Node/SPEC.md`](./Src/Node/SPEC.md). - **JSON Pointer (`BSFLPointerCompiler` + `BSFLPointerQuery`, `ExtraDef/Pointer/`)**: RFC 6901-style paths (`/a/0/b~1c`, with `~0` / `~1` escapes) are **compiled once** into a query (each key already FNV-1a hashed, each array index already parsed) and then reused as many times as you want. The same `BSFLPointerQuery` resolves against the read-only tape (`node[query]`, `Node/Pointer.mqh`) and against the mutable DOM (`dom[query]`, `DOM/DomPointer.mqh`). Failures are not silent: `query.m_last_err`, `ErrorInfo()` / `ErrorFormat()` point at the exact segment of the path that failed (key not found, index out of range, node is not an object/array), and `CintaTypesInfo()` dumps the compiled instructions. - **Mutable DOM (`CDomNodeBase` + `CDomNodeManager`, `DOM/`)**: a fully independent, writable tree — arenas with free-lists for strings, array nodes and objects (objects are backed by `CHashMapFast` from [FastCollectionsByLeo](https://forge.mql5.io/nique_372/FastCollectionsByLeo)). Supports `operator[]` (creates the key if missing) / `Get` / `Set` / `Remove`, `At` / `AtSafe` / `AddItem` / `RemoveItem`, Python-style `PySlicing`, `Clear`, `NewObj` / `NewArr`, typed reads (`ToInt`, `ToUint`, `ToDouble`, `ToBool`, `ToString`, plus `*Safe` variants) and custom type bridges through `Load` / `GetAs`. Any read-only node can be materialized into this DOM via `ToDom()` (which calls the format's `SerializeToDom`), which is how you turn a parsed-but-frozen document into something you can mutate, rebuild, or hand off between formats. - **Unescape engine (`CBSFLUnescape`, `UE/`)**: one generic implementation of `\` escape handling for any serialization language that uses `"` strings (JSON, YAML, TOML...). The format only injects a backend with a `s_unescape_table[256]` (built from the `TSN_TABLE_UNESCAPE_*` sentinels of [TsnTables](https://forge.mql5.io/nique_372/TsnTables)), and gets `MakeString` (UTF-16 `string`), `MakeUArray` (UTF-8 `uchar[]`, surrogates included) and `MakeHash` (FNV-1a over the unescaped bytes, so escaped keys hash the same as their plain form). See [`Src/UE/README.md`](./Src/UE/README.md). - **Query VM (`Query/`)**: a small register-based bytecode VM plus its own assembly-like compiler, for expressing JSONPath-style queries (filters, iteration, math, comparisons, function calls, slicing) that run directly over the DOM. `CSLDomQueryCompiler` compiles a `.jsonpasm` text (or loads a precompiled `.jsonpb` bytecode cache saved with `SaveCompiled`) into bytecode; `CSLDomQueryVm : public CSLDomQueryCompiler` executes it via `Run()`. Supports named jump labels, registers, `SAVE` / `SAVE_NO_RESET`, arithmetic and bitwise ops, `COMPARE_EQ/MENOR/MENOR_EQ/IS`, user-registered native functions (`SetFunction`, read arguments with `FReadIntegerAs` / `FReadDouble` / `FReadString` / `FReadBool` / `FReadNode`, return values with `FPushInteger` / `FPushValue`), array slicing (Python-style start/end/step) and `COPY_ALL` / `APPEND` to collect results. Number and label semantics are specified in [`Src/Query/SPEC.md`](./Src/Query/SPEC.md). - **Shared utilities (`Utils/`)**: number-parsing helpers used by the pointer and query compilers, and `CParserErrFormat`, the formatter that prints parse errors with line/column context. ### Usage example — extending the base parser (pattern used by JsonParserByLeo) ```mql5 // 1. Extend the abstract parser class CJsonParser : public TSN::CBaseStructuredLan { private: // Aqui podrian ir pilas, o lo que use tu lengauje depende quizas lo haces recusivo o con pilas... int CalcRegType(const string& file_name) override final; // detecta .json / cache public: static const uchar s_unescape_table[256]; // sentinels de TsnTables, lo usa CBSFLUnescape static const string s_dtype_to_str[]; // nombre de cada tipo ("" = tipo invalido) bool Parse(); // tu lexer propio, llena m_cinta }; // 2. Extend the node layer for typed, read-only navigation // struct CJsonNode : TSN::CNodeSFLBase { // constructores, EMPTY_NODE, OnMute, ToString/ToInt/ToDouble... (ver Node/SPEC.md) }; // 3. Uso (con JsonParserByLeo) CJsonParser parser; parser.Assing(json); parser.CorrectPadding(); parser.Parse(); CJsonNode root = parser.GetRoot(); string val = root["precios"]["libro1"].ToString(""); ``` ### Usage example — JSON Pointer (tape and DOM) ```mql5 // Se compila UNA vez y se reutiliza (tape o DOM, mismo query) TSN::BSFLPointerCompiler compiler; TSN::BSFLPointerQuery query("/config~1general/riesgo~0maximo", compiler); // Sobre el nodo de solo lectura (cinta) TSN::CJsonNode root = parser.GetRoot(); Print(root[query].ToDouble(0.0)); // Reusar el mismo query con otra ruta query.Set("/ordenes/2/tags/2", compiler); Print(root[query].ToString("")); // Sobre el DOM mutable TSN::CDomNodeManager manager; TSN::CDomNodeBase node(&manager); root.ToDom(&node, &manager); TSN::CDomNodeBase* found = node[query]; // NULL si falla if(found == NULL) query.ErrorInfo(); // imprime el segmento exacto que fallo ``` ### Usage example — JSONPath-style query (`.jsonpasm`) ``` ; $.libros[?(@.comando * 20 < @.valor && @.sumando.size()].sumando[*] ;--- Example in ASM DOM ; Registers %reg 0 = 20 %reg 1 = 1 ; Code INS_ACCESO_KEY_F K"libros" INS_INICIAR_ITERACION 1 @Iteracion: INS_ITER_ASSING 'Final' INS_ACCESO_KEY 'Iteracion' K"comando" INS_SAVE 256 'Iteracion' INS_M_MUL 256 0 256 'Iteracion' INS_ACCESO_KEY 'Iteracion' K"valor" INS_SAVE 257 'Iteracion' INS_COMPARE_MENOR 256 257 1 'Iteracion' INS_ACCESO_KEY 'Iteracion' K"sumando" INS_SAVE_NO_RESET 256 'Iteracion' INS_CALL 256 1 'size' INS_COMPARE_EQ 257 1 1 'Iteracion' INS_COMPARE_IS 258 'Iteracion' 1 INS_COPY_ALL 'Iteracion' INS_JUMP 'Iteracion' @Final: INS_SALIDA ``` > Compiled with `CSLDomQueryCompiler::CompileAsm()` and executed against a `CDomNodeBase*` tree via `CSLDomQueryVm::Run()`. Native functions such as `size` are registered beforehand with `SetFunction("size", MyFunc)`. ### Extending to a new format (TOML, YAML, custom) The intended path for a new "structured language" is the same one JsonParserByLeo follows — see its `JsonCore/JsonParser.mqh` and `JsonNode/JsonNode.mqh` for a concrete reference, and [`Src/ComoHeredar/README.md`](./Src/ComoHeredar/README.md) for the recommended project layout and parsing style: 1. Extend `CBaseStructuredLan` with your own tokenizer/lexer that fills `m_cinta` using this repo's tape layout (see `BaseClass/Def.mqh` and [`Src/Node/SPEC.md`](./Src/Node/SPEC.md) for the layout of ARR/OBJ/KEY/INT/FLT/BOL/STR slots). 2. Define `s_unescape_table` (from the `TSN_TABLE_UNESCAPE_*` sentinels) and reuse `CBSFLUnescape` for strings and key hashes. 3. Extend `CNodeSFLBase` with a typed node struct for your format (key access, array access, `ToString`/`ToInt`/etc., following `CJsonNode` as a template) and implement the `TCtx` functions the node calls (`GetStep`, `NodeAt`, `NodeFindByHashFvn1A64`, `NodeHashing`, `SerializeToDom`...). 4. Reuse `CDomNodeBase` / `CDomNodeManager` as-is for the mutable DOM — no changes needed there. 5. Reuse `BSFLPointerCompiler` / `BSFLPointerQuery` as-is: pointer resolution over your node and over the DOM comes for free once step 3 is done. 6. Reuse the Query VM (`Query/`) as-is if you want JSONPath-style queries over your format's DOM. --- ## Repository Structure ``` BasesParserSLan/ ├── Src/ │ ├── BaseClass/ # CBaseLanFormat / CBaseStructuredLan (tape, perfect-hash tables, cache) │ ├── Node/ # CNodeSFLBase (read-only navigation, JIT hashing, iterators) + pointer resolver │ │ └── SPEC.md # contract for TCtx (parser) and TOut (node) │ ├── DOM/ # Mutable DOM: CDomNodeBase, CDomNodeManager, DomPointer │ ├── ExtraDef/ │ │ └── Pointer/ # BSFLPointerCompiler / BSFLPointerQuery (JSON Pointer, compiled once) │ ├── UE/ # CBSFLUnescape (generic unescape engine) │ ├── Query/ # Register-based VM + assembler (.jsonpasm / .jsonpb) │ │ ├── Table/ # Generated perfect-hash table of instruction names │ │ └── SPEC.md # number / label semantics of the assembly │ ├── Utils/ # Number parsing helpers, parse-error formatter │ └── ComoHeredar/ # Guide: how to build your own parser on this backend └── dependencies.json ``` --- ## Requirements See [dependencies.json](./dependencies.json) for the full list. - MetaTrader 5, build 5430+ - [TsnTables](https://forge.mql5.io/nique_372/TsnTables), [PerfectHashByLeo](https://forge.mql5.io/nique_372/PerfectHashByLeo) and [FastCollectionsByLeo](https://forge.mql5.io/nique_372/FastCollectionsByLeo) (installed automatically by `tsndep`) --- ## Installation ```bash cd "C:\Users\YOUR_USER\AppData\Roaming\MetaQuotes\Terminal\YOUR_ID\MQL5\Shared Projects" tsndep install "https://forge.mql5.io/nique_372/BasesParserSLan.git" ``` Requires the `tsndep` package, available on [PyPI](https://pypi.org/project/tsndep). It automatically downloads and installs all declared dependencies. --- ## Quick Start **1. Include the layer you need:** ```mql5 #include // parser base (CBaseStructuredLan) #include // + node layer #include // + JSON Pointer over the node layer #include // + mutable DOM #include // + JSON Pointer over the DOM #include // + unescape engine #include // + query VM ``` **2. This repo alone does nothing** — extend `CBaseStructuredLan` and `CNodeSFLBase` for your format, or depend on a repo that already did (e.g. JsonParserByLeo). --- ## License **[Read Full License](./LICENSE)** By downloading or using this repository, you accept the license terms. --- ## Contact - **Platform:** [MQL5 Community](https://www.mql5.com/es/users/nique_372) - **Profile:** https://www.mql5.com/es/users/nique_372 - **Articles:** https://www.mql5.com/es/users/nique_372/publications