SetFileByLeo/README.md

<p align="center">
  <img src="https://img.shields.io/badge/Language-MQL5-1B6CA8?style=flat-square"/>
  <img src="https://img.shields.io/badge/Platform-MetaTrader%205-0D1B2A?style=flat-square"/>
  <img src="https://img.shields.io/badge/Author-nique__372-C9D6DF?style=flat-square&logoColor=white"/>
  <img src="https://img.shields.io/badge/MQL5.com-nique__372-1B6CA8?style=flat-square"/>
</p>

<p align="center">
A high-performance, memory-free <code>.set</code> file parser for MQL5, based on a flat tape architecture.<br/>
Single-pass iterative state machine (as is, without extra loops, "pure" O(n)): no recursion, no dynamic memory fragmentation, no overhead from function calls.
</p>

---

## Main Features

- **Tape-based zero-alloc model**: the entire `.set` file is parsed into a single contiguous `long[]` array
- **Single flat loop**: one `switch` over a token enum, no helper function calls during parsing, strictly O(n)
- **JIT perfect-hash over the ROOT**: unlike JSON, a `.set` file is always a flat `key=value` map with no nested objects or arrays, so there can only ever be **one** hashable table per file (the ROOT). Access starts as linear scanning (`StrEquals`); once the ROOT passes `TSN_SETFILE_JIT_MIN_REF_TO_HASING` accesses, a minimal perfect hash table is built for it on the fly (`k = n`, no slack), promoting subsequent lookups to O(1). No FNV hashing is computed upfront during parsing.
- **Two value formats per line**: `key=value` (non-optimizable) and `key=value||start||step||stop||Y` / `...||N` (optimizable, with an enabled/disabled flag)
- **Handle-based navigation**: `CSetNode` is a lightweight struct (pointer + two ints), copying is free
- **In-place mutation**: `SetInt`, `SetDbl`, `SetBool`, `SetEnabled` modify the tape directly without re-parsing
- **Iterator support**: `CSetIteratorObj` to walk every entry in the file
- **Comment detection**: lines starting with `;` are skipped during parsing
- **Compiled tape cache (`.settc`)**: `SaveCompiled` persists the tape, the source `.set` (or a reference to its file), and the perfect-hash table already built — so a reload via `AssingFile` skips parsing entirely and keeps the ROOT already hashed at O(1)

---

### Parse and access

```mql5
#include "Src\Node.mqh"

TSN::CSetFileParser setfile;
const string set = "InpValor=10\n"
                    "InpVa=aaaaaaaa\n"
                    "InpS=10.0\n"
                    "InpMul=10||20||1||30||Y\n"
                    "InpBoolean=true||false||0||true||Y\n";

setfile.Assing(set);
setfile.CorrectPading();
setfile.Parse();

TSN::CSetNode root = setfile.GetRoot();
int valor = (int)root["InpValor"].Value().ToInt();
string va = root["InpVa"].Value().ToString();
int step = (int)root["InpMul"].Step().ToInt();
bool b = root["InpBoolean"].Value().ToBool();
```

### Iterate every entry

```mql5
TSN::CSetIteratorObj it = root.BeginObj();
while(it.IsValid())
 {
  PrintFormat("%s : %s", it.Key(), it.Val().ToString());
  it.Next();
 }
```

### Positional and key-based access

```mql5
string keys[];
int total = root.GetKeys(keys);

string key_two = root.AtObjKey(2);
int pos  = root.KeyToPosition(key_two);
bool exists = root.HasKey("InpMul");
```

### Parse from file

```mql5
TSN::CSetFileParser setfile;
setfile.AssingFile("data.set", false);
setfile.Parse();
```

### Compile and reload a `.settc` tape cache

`SaveCompiled` writes the parsed tape (plus the source `.set` and the perfect-hash table, if already built) to a binary `.settc` file. Reloading it via `AssingFile` skips parsing entirely:

```mql5
//--- Compile once
TSN::CSetFileParser setfile;
setfile.AssingFile("data.set", false);
setfile.Parse();
setfile.GetRoot()["InpMul"]; // (optional) warm up the perfect-hash table before saving
setfile.SaveCompiled("data.settc", false);

//--- Reload later — no Parse() needed, tape and hash table are restored as-is
TSN::CSetFileParser setfile2;
setfile2.AssingFile("data.settc", false);
TSN::CSetNode root = setfile2.GetRoot();
```

There's also an overload that stores a reference to an external `.set` file instead of embedding it:

```mql5
setfile.SaveCompiled("data.settc", false, "data.set", false);
```

---

## Performance

Benchmark: `TestOptim.set` (100.0 KB, 1891 lines — a mix of strings, booleans, numbers, and optimizable entries), 6000 iterations, same file and same machine.

> Note: here we measure how quickly the `.set` file is parsed.
> All benchmarks are located in: `Test\`
>
> There are currently no other public `.set` parsing libraries in MQL5 to compare against, so as a reference we use `Test\AiCode.mqh` — an alternative, AI-generated implementation over the same tape format.

| Parser | Language | Time (ms, total / 6000 iter) |
|--------|----------|-------------------------------|
| **SetFileByLeo** | MQL5 | **1050.3** |
| Claude Sonnet 5 (Effort=Max) (AI reference, tape-based, 90 minutes for devoloping fast setfile parser, iterations +20) | MQL5 | 1503.9 |

### Performance notes

- Machine: Laptop Lenovo (~2017), Windows 10 build 19045, Intel Core i5-8250U @ 1.60GHz, AVX2, GMT-5
- MQL5 run: MetaTrader 5 x64 build 5836

---

## Repository Structure

```
SetFileByLeo/
├── Src/  # Full code (Def, Node, Parser)
└── Test/ # Test and Benchmarks (vs)
```

---

## License

**[Read Full License](./LICENSE)**
By downloading or using this repository, you accept the license terms.

---

## Requirements

See [dependencies.json](./dependencies.json) for the full list.

---

## Installation

```bash
cd "C:\Users\YOUR_USER\AppData\Roaming\MetaQuotes\Terminal\YOUR_ID\MQL5\Shared Projects"
tsndep install "https://forge.mql5.io/nique_372/SetFileByLeo.git"
```

Requires the `tsndep` package, available on [PyPI](https://pypi.org/project/tsndep). It automatically downloads and installs all declared dependencies.

---

## Quick Start

**1. Include the library:**

```mql5
#include "..\SetFileByLeo\Src\Node.mqh"
```

**2. Parse and access:**

```mql5
TSN::CSetFileParser setfile;
setfile.Assing(my_set_string);
setfile.CorrectPading();
if(setfile.Parse())
 {
  TSN::CSetNode root = setfile.GetRoot();
  double lot = root["LotSize"].Value().ToDouble(0.0);
 }
```

**3. Re-parse without re-copying (benchmark pattern):**

```mql5
setfile.Assing(raw_string);  // copy once
for(int i = 0; i < 1000; i++)
  setfile.Parse();            // re-parse in-place
```

---

## Contact

- **Platform:** [MQL5 Community](https://www.mql5.com/es/users/nique_372)
- **Profile:** https://www.mql5.com/es/users/nique_372
- **Articles:** https://www.mql5.com/es/users/nique_372/publications