Base repository for parsers of structured languages (JSON, YAML, TOML, or custom formats) in MQL5.
Provides an abstract parser, a read-only node/navigation layer, a mutable DOM, a precompiled pointer system (JSON Pointer), a shared unescape engine and a small query VM — all shared infrastructure that concrete parsers ( like JsonParserByLeo) extend rather than reimplement.

---

Main Features

This repository does not parse any format by itself. It provides the building blocks a concrete "structured language" parser is built from, and every layer is meant to be extended, not used directly.

• Abstract parser base (CBaseStructuredLan, BaseClass/Main.mqh): holds the parsed data as a flat "cinta" (a ulong[] tape) plus the raw uchar[] buffer, owns the perfect-hash tables used for O(1) key lookups (m_tables, built with PerfectHashByLeo), and handles saving/loading a fully compiled representation (tape + tables) to a binary cache file (SaveCompiled / AssingFile, with the raw text either embedded or referenced by path). It also normalizes UTF-16 / UTF-8 BOM input (NormalizeUft). A concrete parser extends this class and implements CalcRegType (how to detect the file type) plus its own lexer/tokenizer that fills the tape (see JsonParserByLeo's CJsonParser : public CBaseStructuredLan).
• Read-only node/navigation layer (CNodeSFLBase<TCtx, TType, TOut>, Node/NodeBase.mqh): generic template for walking the tape without copying data — operator[] / Get(hash) by key, At / AtObj by index, HasKey, GetKeys, ToArray, ExistInArray, array/object iterators (BeginArr, BeginArrHomogeneo, BeginObj), raw token movement, and JIT perfect-hashing (an object is scanned linearly while its key hashes are cached lazily on the tape; if it keeps being accessed, a perfect-hash table is built for it on the fly). A concrete format defines its own node type on top, e.g. CJsonNode : CNodeSFLBase<CJsonParser, ENUM_JSON_VTYPE, CJsonNode>. The exact contract TCtx and TOut must fulfill is documented in Src/Node/SPEC.md.
• JSON Pointer (BSFLPointerCompiler + BSFLPointerQuery, ExtraDef/Pointer/): RFC 6901-style paths (/a/0/b~1c, with ~0 / ~1 escapes) are compiled once into a query (each key already FNV-1a hashed, each array index already parsed) and then reused as many times as you want. The same BSFLPointerQuery resolves against the read-only tape (node[query], Node/Pointer.mqh) and against the mutable DOM (dom[query], DOM/DomPointer.mqh). Failures are not silent: query.m_last_err, ErrorInfo() / ErrorFormat() point at the exact segment of the path that failed (key not found, index out of range, node is not an object/array), and CintaTypesInfo() dumps the compiled instructions.
• Mutable DOM (CDomNodeBase + CDomNodeManager, DOM/): a fully independent, writable tree — arenas with free-lists for strings, array nodes and objects (objects are backed by CHashMapFast from FastCollectionsByLeo). Supports operator[] (creates the key if missing) / Get / Set / Remove, At / AtSafe / AddItem / RemoveItem, Python-style PySlicing, Clear, NewObj / NewArr, typed reads (ToInt, ToUint, ToDouble, ToBool, ToString, plus *Safe variants) and custom type bridges through Load<T> / GetAs<T>. Any read-only node can be materialized into this DOM via ToDom() (which calls the format's SerializeToDom), which is how you turn a parsed-but-frozen document into something you can mutate, rebuild, or hand off between formats.
• Unescape engine (CBSFLUnescape, UE/): one generic implementation of \ escape handling for any serialization language that uses " strings (JSON, YAML, TOML...). The format only injects a backend with a s_unescape_table[256] (built from the TSN_TABLE_UNESCAPE_* sentinels of TsnTables), and gets MakeString (UTF-16 string), MakeUArray (UTF-8 uchar[], surrogates included) and MakeHash (FNV-1a over the unescaped bytes, so escaped keys hash the same as their plain form). See Src/UE/README.md.
• Query VM (Query/): a small register-based bytecode VM plus its own assembly-like compiler, for expressing JSONPath-style queries (filters, iteration, math, comparisons, function calls, slicing) that run directly over the DOM. CSLDomQueryCompiler compiles a .jsonpasm text (or loads a precompiled .jsonpb bytecode cache saved with SaveCompiled) into bytecode; CSLDomQueryVm : public CSLDomQueryCompiler executes it via Run(). Supports named jump labels, registers, SAVE / SAVE_NO_RESET, arithmetic and bitwise ops, COMPARE_EQ/MENOR/MENOR_EQ/IS, user-registered native functions (SetFunction, read arguments with FReadIntegerAs / FReadDouble / FReadString / FReadBool / FReadNode, return values with FPushInteger / FPushValue), array slicing (Python-style start/end/step) and COPY_ALL / APPEND to collect results. Number and label semantics are specified in Src/Query/SPEC.md.
• Shared utilities (Utils/): number-parsing helpers used by the pointer and query compilers, and CParserErrFormat, the formatter that prints parse errors with line/column context.

Usage example — extending the base parser (pattern used by JsonParserByLeo)

// 1. Extend the abstract parser
class CJsonParser : public TSN::CBaseStructuredLan
 {
private:
 // Aqui podrian ir pilas, o lo que use tu lengauje depende quizas lo haces recusivo o con pilas...
  int CalcRegType(const string& file_name) override final; // detecta .json / cache
public:
  static const uchar  s_unescape_table[256];  // sentinels de TsnTables, lo usa CBSFLUnescape
  static const string s_dtype_to_str[];       // nombre de cada tipo ("" = tipo invalido)

  bool Parse(); // tu lexer propio, llena m_cinta
 };

// 2. Extend the node layer for typed, read-only navigation
// <Tu clase de pasing, Tu Enumeracino de los tipos disponibles, y El mismo nombre del nodo>
struct CJsonNode : TSN::CNodeSFLBase<CJsonParser, ENUM_JSON_VTYPE, CJsonNode>
 {
  // constructores, EMPTY_NODE, OnMute, ToString/ToInt/ToDouble... (ver Node/SPEC.md)
 };

// 3. Uso (con JsonParserByLeo)
CJsonParser parser;
parser.Assing(json);
parser.CorrectPadding();
parser.Parse();
CJsonNode root = parser.GetRoot();
string val = root["precios"]["libro1"].ToString("");

Usage example — JSON Pointer (tape and DOM)

// Se compila UNA vez y se reutiliza (tape o DOM, mismo query)
TSN::BSFLPointerCompiler compiler;
TSN::BSFLPointerQuery query("/config~1general/riesgo~0maximo", compiler);

// Sobre el nodo de solo lectura (cinta)
TSN::CJsonNode root = parser.GetRoot();
Print(root[query].ToDouble(0.0));

// Reusar el mismo query con otra ruta
query.Set("/ordenes/2/tags/2", compiler);
Print(root[query].ToString(""));

// Sobre el DOM mutable
TSN::CDomNodeManager manager;
TSN::CDomNodeBase node(&manager);
root.ToDom(&node, &manager);
TSN::CDomNodeBase* found = node[query];   // NULL si falla
if(found == NULL)
   query.ErrorInfo();                // imprime el segmento exacto que fallo

Usage example — JSONPath-style query (.jsonpasm)

; $.libros[?(@.comando * 20 < @.valor && @.sumando.size()].sumando[*]

;--- Example in ASM DOM
; Registers
%reg 0 = 20
%reg 1 = 1

; Code
INS_ACCESO_KEY_F K"libros"
INS_INICIAR_ITERACION 1
@Iteracion:
  INS_ITER_ASSING 'Final'
  INS_ACCESO_KEY 'Iteracion' K"comando"
  INS_SAVE 256 'Iteracion'
  INS_M_MUL 256 0 256 'Iteracion'
  INS_ACCESO_KEY 'Iteracion' K"valor"
  INS_SAVE 257 'Iteracion'
  INS_COMPARE_MENOR 256 257 1 'Iteracion'
  INS_ACCESO_KEY 'Iteracion' K"sumando"
  INS_SAVE_NO_RESET 256 'Iteracion'
  INS_CALL 256 1 'size'
  INS_COMPARE_EQ 257 1 1 'Iteracion'
  INS_COMPARE_IS 258 'Iteracion' 1
  INS_COPY_ALL 'Iteracion'
  INS_JUMP 'Iteracion'
@Final:
  INS_SALIDA

Compiled with CSLDomQueryCompiler::CompileAsm() and executed against a CDomNodeBase* tree via CSLDomQueryVm::Run(). Native functions such as size are registered beforehand with SetFunction("size", MyFunc).

Extending to a new format (TOML, YAML, custom)

The intended path for a new "structured language" is the same one JsonParserByLeo follows — see its JsonCore/JsonParser.mqh and JsonNode/JsonNode.mqh for a concrete reference, and Src/ComoHeredar/README.md for the recommended project layout and parsing style:

1. Extend CBaseStructuredLan with your own tokenizer/lexer that fills m_cinta using this repo's tape layout (see BaseClass/Def.mqh and Src/Node/SPEC.md for the layout of ARR/OBJ/KEY/INT/FLT/BOL/STR slots).
2. Define s_unescape_table (from the TSN_TABLE_UNESCAPE_* sentinels) and reuse CBSFLUnescape for strings and key hashes.
3. Extend CNodeSFLBase<TCtx, TType, TOut> with a typed node struct for your format (key access, array access, ToString/ToInt/etc., following CJsonNode as a template) and implement the TCtx functions the node calls (GetStep, NodeAt, NodeFindByHashFvn1A64, NodeHashing, SerializeToDom...).
4. Reuse CDomNodeBase / CDomNodeManager as-is for the mutable DOM — no changes needed there.
5. Reuse BSFLPointerCompiler / BSFLPointerQuery as-is: pointer resolution over your node and over the DOM comes for free once step 3 is done.
6. Reuse the Query VM (Query/) as-is if you want JSONPath-style queries over your format's DOM.

---

Repository Structure

BasesParserSLan/
├── Src/
│   ├── BaseClass/        # CBaseLanFormat / CBaseStructuredLan (tape, perfect-hash tables, cache)
│   ├── Node/             # CNodeSFLBase (read-only navigation, JIT hashing, iterators) + pointer resolver
│   │   └── SPEC.md       # contract for TCtx (parser) and TOut (node)
│   ├── DOM/              # Mutable DOM: CDomNodeBase, CDomNodeManager, DomPointer
│   ├── ExtraDef/
│   │   └── Pointer/      # BSFLPointerCompiler / BSFLPointerQuery (JSON Pointer, compiled once)
│   ├── UE/               # CBSFLUnescape (generic unescape engine)
│   ├── Query/            # Register-based VM + assembler (.jsonpasm / .jsonpb)
│   │   ├── Table/        # Generated perfect-hash table of instruction names
│   │   └── SPEC.md       # number / label semantics of the assembly
│   ├── Utils/            # Number parsing helpers, parse-error formatter
│   └── ComoHeredar/      # Guide: how to build your own parser on this backend
└── dependencies.json

---

Requirements

See dependencies.json for the full list.

• MetaTrader 5, build 5430+
• TsnTables, PerfectHashByLeo and FastCollectionsByLeo (installed automatically by tsndep)

---

Installation

cd "C:\Users\YOUR_USER\AppData\Roaming\MetaQuotes\Terminal\YOUR_ID\MQL5\Shared Projects"
tsndep install "https://forge.mql5.io/nique_372/BasesParserSLan.git"

Requires the tsndep package, available on PyPI. It automatically downloads and installs all declared dependencies.

---

Quick Start

1. Include the layer you need:

#include <TSN\\BSFL\\Main.mqh>          // parser base (CBaseStructuredLan)
#include <TSN\\BSFL\\Node.mqh>          // + node layer
#include <TSN\\BSFL\\NodePointer.mqh>   // + JSON Pointer over the node layer
#include <TSN\\BSFL\\Dom.mqh>           // + mutable DOM
#include <TSN\\BSFL\\DomPointer.mqh>    // + JSON Pointer over the DOM
#include <TSN\\BSFL\\UE.mqh>            // + unescape engine
#include <TSN\\BSFL\\Query.mqh>         // + query VM

2. This repo alone does nothing — extend CBaseStructuredLan and CNodeSFLBase for your format, or depend on a repo that already did (e.g. JsonParserByLeo).

---

License

Read Full License By downloading or using this repository, you accept the license terms.

---

Contact

• Platform: MQL5 Community
• Profile: https://www.mql5.com/es/users/nique_372
• Articles: https://www.mql5.com/es/users/nique_372/publications