A high-performance, memory-free .set file parser for MQL5, based on a flat tape architecture.
Single-pass iterative state machine (as is, without extra loops, "pure" O(n)): no recursion, no dynamic memory fragmentation, no overhead from function calls.
---
## Main Features
- **Tape-based zero-alloc model**: the entire `.set` file is parsed into a single contiguous `long[]` array
- **Single flat loop**: one `switch` over a token enum, no helper function calls during parsing, strictly O(n)
- **JIT perfect-hash over the ROOT**: unlike JSON, a `.set` file is always a flat `key=value` map with no nested objects or arrays, so there can only ever be **one** hashable table per file (the ROOT). Access starts as linear scanning (`StrEquals`); once the ROOT passes `TSN_SETFILE_JIT_MIN_REF_TO_HASING` accesses, a minimal perfect hash table is built for it on the fly (`k = n`, no slack), promoting subsequent lookups to O(1). No FNV hashing is computed upfront during parsing.
- **Two value formats per line**: `key=value` (non-optimizable) and `key=value||start||step||stop||Y` / `...||N` (optimizable, with an enabled/disabled flag)
- **Handle-based navigation**: `CSetNode` is a lightweight struct (pointer + two ints), copying is free
- **In-place mutation**: `SetInt`, `SetDbl`, `SetBool`, `SetEnabled` modify the tape directly without re-parsing
- **Iterator support**: `CSetIteratorObj` to walk every entry in the file
- **Comment detection**: lines starting with `;` are skipped during parsing
- **Compiled tape cache (`.settc`)**: `SaveCompiled` persists the tape, the source `.set` (or a reference to its file), and the perfect-hash table already built — so a reload via `AssingFile` skips parsing entirely and keeps the ROOT already hashed at O(1)
---
### Parse and access
```mql5
#include "Src\Node.mqh"
TSN::CSetFileParser setfile;
const string set = "InpValor=10\n"
"InpVa=aaaaaaaa\n"
"InpS=10.0\n"
"InpMul=10||20||1||30||Y\n"
"InpBoolean=true||false||0||true||Y\n";
setfile.Assing(set);
setfile.CorrectPading();
setfile.Parse();
TSN::CSetNode root = setfile.GetRoot();
int valor = (int)root["InpValor"].Value().ToInt();
string va = root["InpVa"].Value().ToString();
int step = (int)root["InpMul"].Step().ToInt();
bool b = root["InpBoolean"].Value().ToBool();
```
### Iterate every entry
```mql5
TSN::CSetIteratorObj it = root.BeginObj();
while(it.IsValid())
{
PrintFormat("%s : %s", it.Key(), it.Val().ToString());
it.Next();
}
```
### Positional and key-based access
```mql5
string keys[];
int total = root.GetKeys(keys);
string key_two = root.AtObjKey(2);
int pos = root.KeyToPosition(key_two);
bool exists = root.HasKey("InpMul");
```
### Parse from file
```mql5
TSN::CSetFileParser setfile;
setfile.AssingFile("data.set", false);
setfile.Parse();
```
### Compile and reload a `.settc` tape cache
`SaveCompiled` writes the parsed tape (plus the source `.set` and the perfect-hash table, if already built) to a binary `.settc` file. Reloading it via `AssingFile` skips parsing entirely:
```mql5
//--- Compile once
TSN::CSetFileParser setfile;
setfile.AssingFile("data.set", false);
setfile.Parse();
setfile.GetRoot()["InpMul"]; // (optional) warm up the perfect-hash table before saving
setfile.SaveCompiled("data.settc", false);
//--- Reload later — no Parse() needed, tape and hash table are restored as-is
TSN::CSetFileParser setfile2;
setfile2.AssingFile("data.settc", false);
TSN::CSetNode root = setfile2.GetRoot();
```
There's also an overload that stores a reference to an external `.set` file instead of embedding it:
```mql5
setfile.SaveCompiled("data.settc", false, "data.set", false);
```
---
## Performance
Benchmark: `TestOptim.set` (100.0 KB, 1891 lines — a mix of strings, booleans, numbers, and optimizable entries), 6000 iterations, same file and same machine.
> Note: here we measure how quickly the `.set` file is parsed.
> All benchmarks are located in: `Test\`
>
> There are currently no other public `.set` parsing libraries in MQL5 to compare against, so as a reference we use `Test\AiCode.mqh` — an alternative, AI-generated implementation over the same tape format.
| Parser | Language | Time (ms, total / 6000 iter) |
|--------|----------|-------------------------------|
| **SetFileByLeo** | MQL5 | **1050.3** |
| Claude Sonnet 5 (Effort=Max) (AI reference, tape-based, 90 minutes for devoloping fast setfile parser, iterations +20) | MQL5 | 1503.9 |
### Performance notes
- Machine: Laptop Lenovo (~2017), Windows 10 build 19045, Intel Core i5-8250U @ 1.60GHz, AVX2, GMT-5
- MQL5 run: MetaTrader 5 x64 build 5836
---
## Repository Structure
```
SetFileByLeo/
├── Src/ # Full code (Def, Node, Parser)
└── Test/ # Test and Benchmarks (vs)
```
---
## License
**[Read Full License](./LICENSE)**
By downloading or using this repository, you accept the license terms.
---
## Requirements
See [dependencies.json](./dependencies.json) for the full list.
---
## Installation
```bash
cd "C:\Users\YOUR_USER\AppData\Roaming\MetaQuotes\Terminal\YOUR_ID\MQL5\Shared Projects"
tsndep install "https://forge.mql5.io/nique_372/SetFileByLeo.git"
```
Requires the `tsndep` package, available on [PyPI](https://pypi.org/project/tsndep). It automatically downloads and installs all declared dependencies.
---
## Quick Start
**1. Include the library:**
```mql5
#include "..\SetFileByLeo\Src\Node.mqh"
```
**2. Parse and access:**
```mql5
TSN::CSetFileParser setfile;
setfile.Assing(my_set_string);
setfile.CorrectPading();
if(setfile.Parse())
{
TSN::CSetNode root = setfile.GetRoot();
double lot = root["LotSize"].Value().ToDouble(0.0);
}
```
**3. Re-parse without re-copying (benchmark pattern):**
```mql5
setfile.Assing(raw_string); // copy once
for(int i = 0; i < 1000; i++)
setfile.Parse(); // re-parse in-place
```
---
## Contact
- **Platform:** [MQL5 Community](https://www.mql5.com/es/users/nique_372)
- **Profile:** https://www.mql5.com/es/users/nique_372
- **Articles:** https://www.mql5.com/es/users/nique_372/publications