Tick_Forge/README.md

# Tick Forge

Build custom backtesting symbols in MetaTrader 5 from **your broker's own history**:
shift the timestamps, rescale or reshape the prices with a typed function, and test your
Expert Advisor on the result in the normal Strategy Tester.

Free and open source (MIT). No DLL. No internet access. It reads your terminal's downloaded
history and writes one custom symbol - nothing else.

## Why

An EA that works on one stretch of price history may only work because of that stretch's
price level, volatility or calendar. Tick Forge lets you ask the question directly:

- *Does it still work if the same market is 6 hours earlier or 3 days later?* (time offset)
- *Does it still work if gold costs 1,000 instead of 4,000?* (price scale)
- *What if the moves were twice as big, or the trend ramped up over the period?* (price function)

Run the original and the changed data side by side. If the result only exists on the original
data, you have learned something before it cost you money.

## Quick start

1. Install (below), attach **Tick Forge** to a chart.
2. Pick a source symbol. Leave the name (`<symbol>_TF`) or type your own.
3. Pick a date range (or **Use full range**), a data type, a time offset and a price function. Press **Preview**.
4. Press **GENERATE SYMBOL**. When it says DONE, the symbol is in Market Watch (group "Tick Forge").
5. In the Strategy Tester choose the new symbol. Real-tick data pairs with the model *Every tick based on real ticks*,
   M1-only data with *Every tick* / *1 minute OHLC* / *Open prices only*.

Do a **control run first**: generate with function `p` and offset 0, test your EA on it, and only then test the changed
data. Differences are then caused by your change, not by the pipeline.

## Install

**From the MQL5 Market:** install the free product, then drag **Tick Forge** from the Navigator onto a chart.

**From source:**

1. Copy `Tick_Forge.mq5`, `TF_Engine.mqh` and `TF_Expr.mqh` into one folder under
   `MQL5\Experts\Tick_Forge\` of your terminal (File > Open Data Folder).
   (`market/Tick_Forge.mq5` is the same program as one file with the two headers inlined; that is what the Market compiles.)
2. Compile `Tick_Forge.mq5` in MetaEditor (F7).
3. Drag **Tick Forge** from the Navigator onto any chart. It is a utility Expert: it never trades,
   so AutoTrading can stay off. (MetaTrader's window library only works in Experts and indicators,
   which is why it is not a script.) Closing the window removes it from the chart.

The window needs a chart of at least about 1070 x 640 pixels; if the chart is smaller you get a message on the chart.
It does nothing when loaded in the Strategy Tester.

**Close MetaTrader normally** after generating (File > Exit, not a task kill): the terminal saves the list of custom
symbols on a normal exit.

## Using the window

| Field | What it does |
|---|---|
| Source symbol | Any symbol in your Market Watch. Its history is the raw material. |
| Custom symbol name | Name of the symbol to create. An existing *custom* symbol asks before it is overwritten. A real broker symbol is refused. |
| Broker data available | First and last M1 bar your broker serves. "Scan tick availability" lists the months that have ticks. |
| From / to | Date range (inclusive). "Use full range" fills the whole M1 history. |
| Data type | **Real ticks**: every broker tick is transformed and written, M1 bars are rebuilt from them. Use with the tester model "Every tick based on real ticks". **M1 bars only**: the tester synthesizes ticks (models "Every tick", "1 minute OHLC", "Open prices only"). |
| Time offset | Moves the raw timestamps by hours or days. Weekends and sessions move with the data (the clock is shifted, not re-aligned). |
| Price function | The new price as a function of the old one. See below. |
| Spread follows the price factor | Spread is multiplied by the same factor as the price, so the relative trading cost stays the same. Untick to keep the broker's absolute spread. |
| Add digits when prices shrink | If prices become much smaller, digits and tick size are increased so the spread is still resolvable. |
| Preview button | Draws the whole date range: the source prices on top, the result below (after the time shift and the function), each on its own price axis, with dates along the bottom. Check the shape before generating. |
| Bars / Ticks stored in | Where the generated data sits on disk (read-only fields you can copy). |

## Price functions

The function returns the **new price**.

| Variable | Meaning |
|---|---|
| `p` (or `x`) | original price |
| `p0` | first price in the range |
| `r` | `p / p0`, the relative price |
| `t` | progress through the range, 0 to 1 |
| `d` | days since the start of the range |

Operators `+ - * / ^`, parentheses, unary minus. Functions `abs sqrt ln log10 exp sin cos pow(a,b) min(a,b) max(a,b)`. Constants `pi`, `e`.

| Goal | Function |
|---|---|
| Unchanged (control run) | `p` |
| Everything costs twice as much | `p*2` |
| Prices rescaled to a lower level | `p*0.5` |
| Moves squared (extreme), same start price | `p0*r^2` |
| Multiplier ramps from 1 to 2 over the range | `p*(1+t)` |
| Add a fixed amount | `p+1000` |
| Mirror the moves around the start price | `p0*p0/p` |

A function that returns zero, a negative number or an invalid value for any price stops the
run with an error - nothing is written past that point. Always do a **Preview** first, and
always run a control (`p`, no offset) through the same pipeline next to your changed data.

## Things to know

- Only data your terminal can obtain is used. Tick history is downloaded from the broker on demand and
  its depth varies by broker; "Scan tick availability" shows what exists. Use "M1 bars only" for older periods.
- The custom symbol keeps the contract (size, margin, swap, sessions) of the source symbol.
- Custom symbols are stored under `<data folder>\bases\Custom\`. Delete one from
  Market Watch > Symbols > Custom.
- This is a research tool. A backtest on altered data is a stress test, not a prediction.

## Audit set and the runner

The **Audit set (7 symbols)** button builds `<name>_C` (control), `_S7`, `_S14`, `_S28` (shifted 7/14/28 days), `_L50`
(prices x0.5), `_L200` (x2) and `_MIR` (mirrored moves) in one go, after a confirmation that states the number of symbols
and days. **Delete audit set** removes them (only symbols Tick Forge created). Each symbol uses the disk space of one generated
symbol; real ticks use far more than M1 bars. It also writes `MQL5\Files\TickForgeudit_<name>.json`.

MQL5 cannot start the Strategy Tester, so `tools/tf_audit.py` does that part on a terminal you choose and writes a comparison
page (trades, net, profit factor, drawdown, share of control entries reproduced at the shifted time, flags):

    python tools/tf_audit.py run --manifest audit_<name>.json --expert Folder\MyEA --terminal "<testing terminal64.exe>" --data "<its data folder>" --period M15 --model 1 --set my.set
    python tools/tf_audit.py report --manifest audit_<name>.json --reports <dir with the collected reports>

Use a testing install, never a terminal logged in to a live-money account. The script refuses to start if that terminal is
running and never kills anything. `examples/` holds two reports made with `tools/TF_DemoEA.mq5`, a deliberately simple demo
Expert: one trading a rule every weekday, one trading the same rule only on a fitted list of dates.
Shifted data that would lie after today cannot be tested; the report then compares the overlapping part only.

## Developer tools (`tools/`)

`TF_Dev.mq5` is a developer Expert that runs the same window plus headless generation (`InpAuto`), a form-driven run
(`InpGuiRun`) and screenshot mode (`InpScreenshot`); `tf_run.py` drives it on a **testing** terminal.
`build_market.py` builds and compiles the single-file Market versions (Expert and indicator), `make_icon.py` redraws the logos,
`tf_run.py` / `shots.py` drive a **testing** terminal headless, `tf_tester_check.py` + `TF_TestEA.mq5` check that the
Strategy Tester replays a generated symbol. Never point them at a live-money terminal.

## Licence

MIT - see `LICENSE`. You may use, change and redistribute it, including commercially, as long as
the copyright notice stays. Contributions welcome.

Author: Joshua Dunnink

## Ready-made binary

`release/Tick_Forge.ex5` is the compiled 1.03 Expert (utility) build of `market/Tick_Forge.mq5`; copy it to
`MQL5\Experts\` of your terminal and attach it to a chart. `release/Tick_Forge_Indicator.ex5` is the same window as an
indicator (copy to `MQL5\Indicators\`). Both compile with 0 errors and 0 warnings; you can also compile the source yourself.