BasesParserSLan/README.md
206 lines
13 KiB
Markdown
Nique_372
2026-08-01 18:05:18 -05:00
<p align="center">
<img src="https://img.shields.io/badge/Language-MQL5-1B6CA8?style=flat-square"/>
<img src="https://img.shields.io/badge/Platform-MetaTrader%205-0D1B2A?style=flat-square"/>
<img src="https://img.shields.io/badge/Author-nique__372-C9D6DF?style=flat-square&logoColor=white"/>
<img src="https://img.shields.io/badge/MQL5.com-nique__372-1B6CA8?style=flat-square"/>
Nique_372
2026-08-01 18:06:13 -05:00
<a href="./LICENSE">
<img src="https://img.shields.io/badge/License-Nique%26Leo%20NL--NC-yellow.svg"/>
</a>
Nique_372
2026-08-01 18:05:18 -05:00
</p>
Niquel Mendoza
2026-07-18 15:46:31 +00:00
Nique_372
2026-08-01 18:05:18 -05:00
<p align="center">
Base repository for parsers of structured languages (JSON, YAML, TOML, or custom formats) in MQL5.<br/>
Nique_372
2026-10-19 17:03:25 -05:00
Provides an abstract parser, a read-only node/navigation layer, a mutable DOM, a precompiled pointer system (JSON Pointer),
a shared unescape engine and a small query VM — all shared infrastructure that concrete parsers (
Nique_372
2026-08-01 18:05:18 -05:00
like <a href="https://forge.mql5.io/nique_372/JsonParserByLeo">JsonParserByLeo</a>) extend rather than reimplement.
</p>
---
## Main Features
This repository does not parse any format by itself. It provides the building blocks a concrete "structured language" parser is built from, and every layer is meant to be extended, not used directly.
Nique_372
2026-10-19 17:03:25 -05:00
- **Abstract parser base (`CBaseStructuredLan`, `BaseClass/Main.mqh`)**: holds the parsed data as a flat "cinta" (a `ulong[]` tape) plus the raw `uchar[]` buffer, owns the perfect-hash tables used for O(1) key lookups (`m_tables`, built with [PerfectHashByLeo](https://forge.mql5.io/nique_372/PerfectHashByLeo)), and handles saving/loading a fully compiled representation (tape + tables) to a binary cache file (`SaveCompiled` / `AssingFile`, with the raw text either embedded or referenced by path). It also normalizes UTF-16 / UTF-8 BOM input (`NormalizeUft`). A concrete parser extends this class and implements `CalcRegType` (how to detect the file type) plus its own lexer/tokenizer that fills the tape (see JsonParserByLeo's `CJsonParser : public CBaseStructuredLan`).
- **Read-only node/navigation layer (`CNodeSFLBase<TCtx, TType, TOut>`, `Node/NodeBase.mqh`)**: generic template for walking the tape without copying data — `operator[]` / `Get(hash)` by key, `At` / `AtObj` by index, `HasKey`, `GetKeys`, `ToArray`, `ExistInArray`, array/object iterators (`BeginArr`, `BeginArrHomogeneo`, `BeginObj`), raw token movement, and JIT perfect-hashing (an object is scanned linearly while its key hashes are cached lazily on the tape; if it keeps being accessed, a perfect-hash table is built for it on the fly). A concrete format defines its own node type on top, e.g. `CJsonNode : CNodeSFLBase<CJsonParser, ENUM_JSON_VTYPE, CJsonNode>`. The exact contract `TCtx` and `TOut` must fulfill is documented in [`Src/Node/SPEC.md`](./Src/Node/SPEC.md).
- **JSON Pointer (`BSFLPointerCompiler` + `BSFLPointerQuery`, `ExtraDef/Pointer/`)**: RFC 6901-style paths (`/a/0/b~1c`, with `~0` / `~1` escapes) are **compiled once** into a query (each key already FNV-1a hashed, each array index already parsed) and then reused as many times as you want. The same `BSFLPointerQuery` resolves against the read-only tape (`node[query]`, `Node/Pointer.mqh`) and against the mutable DOM (`dom[query]`, `DOM/DomPointer.mqh`). Failures are not silent: `query.m_last_err`, `ErrorInfo()` / `ErrorFormat()` point at the exact segment of the path that failed (key not found, index out of range, node is not an object/array), and `CintaTypesInfo()` dumps the compiled instructions.
- **Mutable DOM (`CDomNodeBase` + `CDomNodeManager`, `DOM/`)**: a fully independent, writable tree — arenas with free-lists for strings, array nodes and objects (objects are backed by `CHashMapFast` from [FastCollectionsByLeo](https://forge.mql5.io/nique_372/FastCollectionsByLeo)). Supports `operator[]` (creates the key if missing) / `Get` / `Set` / `Remove`, `At` / `AtSafe` / `AddItem` / `RemoveItem`, Python-style `PySlicing`, `Clear`, `NewObj` / `NewArr`, typed reads (`ToInt`, `ToUint`, `ToDouble`, `ToBool`, `ToString`, plus `*Safe` variants) and custom type bridges through `Load<T>` / `GetAs<T>`. Any read-only node can be materialized into this DOM via `ToDom()` (which calls the format's `SerializeToDom`), which is how you turn a parsed-but-frozen document into something you can mutate, rebuild, or hand off between formats.
- **Unescape engine (`CBSFLUnescape`, `UE/`)**: one generic implementation of `\` escape handling for any serialization language that uses `"` strings (JSON, YAML, TOML...). The format only injects a backend with a `s_unescape_table[256]` (built from the `TSN_TABLE_UNESCAPE_*` sentinels of [TsnTables](https://forge.mql5.io/nique_372/TsnTables)), and gets `MakeString` (UTF-16 `string`), `MakeUArray` (UTF-8 `uchar[]`, surrogates included) and `MakeHash` (FNV-1a over the unescaped bytes, so escaped keys hash the same as their plain form). See [`Src/UE/README.md`](./Src/UE/README.md).
- **Query VM (`Query/`)**: a small register-based bytecode VM plus its own assembly-like compiler, for expressing JSONPath-style queries (filters, iteration, math, comparisons, function calls, slicing) that run directly over the DOM. `CSLDomQueryCompiler` compiles a `.jsonpasm` text (or loads a precompiled `.jsonpb` bytecode cache saved with `SaveCompiled`) into bytecode; `CSLDomQueryVm : public CSLDomQueryCompiler` executes it via `Run()`. Supports named jump labels, registers, `SAVE` / `SAVE_NO_RESET`, arithmetic and bitwise ops, `COMPARE_EQ/MENOR/MENOR_EQ/IS`, user-registered native functions (`SetFunction`, read arguments with `FReadIntegerAs` / `FReadDouble` / `FReadString` / `FReadBool` / `FReadNode`, return values with `FPushInteger` / `FPushValue`), array slicing (Python-style start/end/step) and `COPY_ALL` / `APPEND` to collect results. Number and label semantics are specified in [`Src/Query/SPEC.md`](./Src/Query/SPEC.md).
- **Shared utilities (`Utils/`)**: number-parsing helpers used by the pointer and query compilers, and `CParserErrFormat`, the formatter that prints parse errors with line/column context.
Nique_372
2026-08-01 18:05:18 -05:00
### Usage example — extending the base parser (pattern used by JsonParserByLeo)
```mql5
// 1. Extend the abstract parser
class CJsonParser : public TSN::CBaseStructuredLan
{
private:
// Aqui podrian ir pilas, o lo que use tu lengauje depende quizas lo haces recusivo o con pilas...
Nique_372
2026-10-19 17:03:25 -05:00
int CalcRegType(const string& file_name) override final; // detecta .json / cache
Nique_372
2026-08-01 18:05:18 -05:00
public:
Nique_372
2026-10-19 17:03:25 -05:00
static const uchar s_unescape_table[256]; // sentinels de TsnTables, lo usa CBSFLUnescape
static const string s_dtype_to_str[]; // nombre de cada tipo ("" = tipo invalido)
Nique_372
2026-08-01 18:05:18 -05:00
bool Parse(); // tu lexer propio, llena m_cinta
};
// 2. Extend the node layer for typed, read-only navigation
// <Tu clase de pasing, Tu Enumeracino de los tipos disponibles, y El mismo nombre del nodo>
struct CJsonNode : TSN::CNodeSFLBase<CJsonParser, ENUM_JSON_VTYPE, CJsonNode>
{
Nique_372
2026-10-19 17:03:25 -05:00
// constructores, EMPTY_NODE, OnMute, ToString/ToInt/ToDouble... (ver Node/SPEC.md)
Nique_372
2026-08-01 18:05:18 -05:00
};
// 3. Uso (con JsonParserByLeo)
CJsonParser parser;
Nique_372
2026-10-19 17:03:25 -05:00
parser.Assing(json);
parser.CorrectPadding();
Nique_372
2026-08-01 18:05:18 -05:00
parser.Parse();
CJsonNode root = parser.GetRoot();
Nique_372
2026-10-19 17:03:25 -05:00
string val = root["precios"]["libro1"].ToString("");
```
### Usage example — JSON Pointer (tape and DOM)
```mql5
// Se compila UNA vez y se reutiliza (tape o DOM, mismo query)
TSN::BSFLPointerCompiler compiler;
TSN::BSFLPointerQuery query("/config~1general/riesgo~0maximo", compiler);
// Sobre el nodo de solo lectura (cinta)
TSN::CJsonNode root = parser.GetRoot();
Print(root[query].ToDouble(0.0));
// Reusar el mismo query con otra ruta
query.Set("/ordenes/2/tags/2", compiler);
Print(root[query].ToString(""));
// Sobre el DOM mutable
TSN::CDomNodeManager manager;
TSN::CDomNodeBase node(&manager);
root.ToDom(&node, &manager);
TSN::CDomNodeBase* found = node[query]; // NULL si falla
if(found == NULL)
query.ErrorInfo(); // imprime el segmento exacto que fallo
Nique_372
2026-08-01 18:05:18 -05:00
```
### Usage example — JSONPath-style query (`.jsonpasm`)
```
; $.libros[?(@.comando * 20 < @.valor && @.sumando.size()].sumando[*]
;--- Example in ASM DOM
; Registers
%reg 0 = 20
%reg 1 = 1
; Code
INS_ACCESO_KEY_F K"libros"
INS_INICIAR_ITERACION 1
@Iteracion:
INS_ITER_ASSING 'Final'
INS_ACCESO_KEY 'Iteracion' K"comando"
INS_SAVE 256 'Iteracion'
INS_M_MUL 256 0 256 'Iteracion'
INS_ACCESO_KEY 'Iteracion' K"valor"
INS_SAVE 257 'Iteracion'
INS_COMPARE_MENOR 256 257 1 'Iteracion'
INS_ACCESO_KEY 'Iteracion' K"sumando"
INS_SAVE_NO_RESET 256 'Iteracion'
INS_CALL 256 1 'size'
INS_COMPARE_EQ 257 1 1 'Iteracion'
INS_COMPARE_IS 258 'Iteracion' 1
INS_COPY_ALL 'Iteracion'
INS_JUMP 'Iteracion'
@Final:
INS_SALIDA
```
Nique_372
2026-10-19 17:03:25 -05:00
> Compiled with `CSLDomQueryCompiler::CompileAsm()` and executed against a `CDomNodeBase*` tree via `CSLDomQueryVm::Run()`. Native functions such as `size` are registered beforehand with `SetFunction("size", MyFunc)`.
Nique_372
2026-08-01 18:05:18 -05:00
### Extending to a new format (TOML, YAML, custom)
Nique_372
2026-10-19 17:03:25 -05:00
The intended path for a new "structured language" is the same one JsonParserByLeo follows — see its `JsonCore/JsonParser.mqh` and `JsonNode/JsonNode.mqh` for a concrete reference, and [`Src/ComoHeredar/README.md`](./Src/ComoHeredar/README.md) for the recommended project layout and parsing style:
Nique_372
2026-08-01 18:05:18 -05:00
Nique_372
2026-10-19 17:03:25 -05:00
1. Extend `CBaseStructuredLan` with your own tokenizer/lexer that fills `m_cinta` using this repo's tape layout (see `BaseClass/Def.mqh` and [`Src/Node/SPEC.md`](./Src/Node/SPEC.md) for the layout of ARR/OBJ/KEY/INT/FLT/BOL/STR slots).
2. Define `s_unescape_table` (from the `TSN_TABLE_UNESCAPE_*` sentinels) and reuse `CBSFLUnescape` for strings and key hashes.
3. Extend `CNodeSFLBase<TCtx, TType, TOut>` with a typed node struct for your format (key access, array access, `ToString`/`ToInt`/etc., following `CJsonNode` as a template) and implement the `TCtx` functions the node calls (`GetStep`, `NodeAt`, `NodeFindByHashFvn1A64`, `NodeHashing`, `SerializeToDom`...).
4. Reuse `CDomNodeBase` / `CDomNodeManager` as-is for the mutable DOM — no changes needed there.
5. Reuse `BSFLPointerCompiler` / `BSFLPointerQuery` as-is: pointer resolution over your node and over the DOM comes for free once step 3 is done.
6. Reuse the Query VM (`Query/`) as-is if you want JSONPath-style queries over your format's DOM.
Nique_372
2026-08-01 18:05:18 -05:00
---
## Repository Structure
```
BasesParserSLan/
Nique_372
2026-10-19 17:03:25 -05:00
├── Src/
│ ├── BaseClass/ # CBaseLanFormat / CBaseStructuredLan (tape, perfect-hash tables, cache)
│ ├── Node/ # CNodeSFLBase (read-only navigation, JIT hashing, iterators) + pointer resolver
│ │ └── SPEC.md # contract for TCtx (parser) and TOut (node)
│ ├── DOM/ # Mutable DOM: CDomNodeBase, CDomNodeManager, DomPointer
│ ├── ExtraDef/
│ │ └── Pointer/ # BSFLPointerCompiler / BSFLPointerQuery (JSON Pointer, compiled once)
│ ├── UE/ # CBSFLUnescape (generic unescape engine)
│ ├── Query/ # Register-based VM + assembler (.jsonpasm / .jsonpb)
│ │ ├── Table/ # Generated perfect-hash table of instruction names
│ │ └── SPEC.md # number / label semantics of the assembly
│ ├── Utils/ # Number parsing helpers, parse-error formatter
│ └── ComoHeredar/ # Guide: how to build your own parser on this backend
└── dependencies.json
Nique_372
2026-08-01 18:05:18 -05:00
```
---
## Requirements
See [dependencies.json](./dependencies.json) for the full list.
- MetaTrader 5, build 5430+
Nique_372
2026-10-19 17:03:25 -05:00
- [TsnTables](https://forge.mql5.io/nique_372/TsnTables), [PerfectHashByLeo](https://forge.mql5.io/nique_372/PerfectHashByLeo) and [FastCollectionsByLeo](https://forge.mql5.io/nique_372/FastCollectionsByLeo) (installed automatically by `tsndep`)
Nique_372
2026-08-01 18:05:18 -05:00
---
## Installation
```bash
cd "C:\Users\YOUR_USER\AppData\Roaming\MetaQuotes\Terminal\YOUR_ID\MQL5\Shared Projects"
tsndep install "https://forge.mql5.io/nique_372/BasesParserSLan.git"
```
Requires the `tsndep` package, available on [PyPI](https://pypi.org/project/tsndep). It automatically downloads and installs all declared dependencies.
---
## Quick Start
**1. Include the layer you need:**
```mql5
Nique_372
2026-10-19 17:03:25 -05:00
#include <TSN\\BSFL\\Main.mqh> // parser base (CBaseStructuredLan)
#include <TSN\\BSFL\\Node.mqh> // + node layer
#include <TSN\\BSFL\\NodePointer.mqh> // + JSON Pointer over the node layer
#include <TSN\\BSFL\\Dom.mqh> // + mutable DOM
#include <TSN\\BSFL\\DomPointer.mqh> // + JSON Pointer over the DOM
#include <TSN\\BSFL\\UE.mqh> // + unescape engine
#include <TSN\\BSFL\\Query.mqh> // + query VM
Nique_372
2026-08-01 18:05:18 -05:00
```
**2. This repo alone does nothing** — extend `CBaseStructuredLan` and `CNodeSFLBase` for your format, or depend on a repo that already did (e.g. JsonParserByLeo).
---
## License
**[Read Full License](./LICENSE)**
By downloading or using this repository, you accept the license terms.
---
## Contact
- **Platform:** [MQL5 Community](https://www.mql5.com/es/users/nique_372)
- **Profile:** https://www.mql5.com/es/users/nique_372
Nique_372
2026-10-19 17:03:25 -05:00
- **Articles:** https://www.mql5.com/es/users/nique_372/publications