Article-24649-GrangerMTF-Ca.../README.md

239 lines
12 KiB
Markdown

2026-09-22 18:39:54 +00:00
# Article-24649-GrangerMTF-Causality-Detection
2026-09-22 19:29:53 +00:00
2026-09-22 19:28:33 +00:00
This repository is an article-derived reference project based on the original MQL5 article. It does not claim to reproduce the full original source code unless files are explicitly attached.
2026-09-22 18:39:54 +00:00
2026-09-22 19:29:53 +00:00
## Overview
2026-09-22 19:28:33 +00:00
This project documents a complete multi-timeframe causality detection pipeline for MetaTrader 5. Instead of assuming that a higher timeframe always informs the lower, the system tests that relationship statistically and uses the result to filter signals at runtime.
The pipeline automatically:
2026-09-22 19:29:53 +00:00
- Fetches historical EURUSD OHLC data for M5, M15, M30 and H1 directly from MetaTrader 5
- Converts prices to log returns and verifies stationarity with an ADF test
- Builds look-ahead-free design matrices aligned so each lower-TF bar only sees already-closed higher-TF bars
- Selects lag structure by BIC and fits restricted and unrestricted OLS models for each timeframe pair
- Runs a Granger F-test, cross-checks it against statsmodels, and adds an HC3 robust Wald test
- Scans rolling windows to measure how stable each relationship is over time
- Corrects for multiple comparisons across all four pairs with Benjamini–Hochberg
- Exports a calibration table to `Terminal\Common\Files` for the EA to read
- Repeats the same F-test live on a rolling window inside MQL5 at runtime
- Marks each pair VALID or INVALID based on both the offline table and the live result
- Generates BUY/SELL signals from the votes of causally valid pairs only
- Gates every trade through the live causality check before execution
- Manages positions with ATR-based stop loss and take profit
- Publishes pair statuses as terminal global variables for use by other EAs
2026-09-22 19:28:33 +00:00
The project demonstrates how a statistical test developed in Python can be reproduced exactly in MQL5 so that the offline research and the live system stay faithful to the same rules.
2026-09-22 19:29:53 +00:00
## Original Article
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
**Article ID:** 24649
**Author:** Hlomohang John Borotho
**Category:** Experts
**URL:** [https://www.mql5.com/en/articles/24649](https://www.mql5.com/en/articles/24649)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
## Repository Purpose
2026-09-22 19:28:33 +00:00
This repository should be treated as a reference or reconstruction project derived from the article content.
Its purpose is to:
2026-09-22 19:29:53 +00:00
- Test whether higher timeframes statistically Granger-cause lower timeframes before trading on that assumption
- Demonstrate look-ahead-free multi-timeframe alignment in both Python and MQL5
- Reproduce an F-test and rolling causality monitor natively in MQL5 without external libraries
- Provide a reusable causality-filtering framework that works on any symbol or timeframe pair
- Serve as a foundation for regime-aware Expert Advisor development
## Key Concepts
### Timeframe Pairs
2026-09-22 19:28:33 +00:00
The system tests four lead → lag relationships. The lead must always be the higher timeframe.
2026-09-22 19:29:53 +00:00
| Pair | Lead | Lag | EURUSD Result |
|------|------|-----|---------------|
| H1 → M15 | H1 | M15 | INVALID (F ≈ 0, stab 9%) |
| M30 → M15 | M30 | M15 | VALID (F = 9.64, stab 43%) |
| M15 → M5 | M15 | M5 | INVALID (F = 4.84, stab 20%) |
| H1 → M5 | H1 | M5 | INVALID (F ≈ 0, stab 6%) |
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### Look-Ahead-Free Alignment
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
For each lower-TF bar opening at time `t`, only higher-TF bars whose open time is at or before `t - lead_period` are used as regressors. This rule is identical in Python and MQL5. Any deviation between the two stages would make the offline coefficients meaningless at runtime.
### Granger F-Test
2026-09-22 19:28:33 +00:00
Two OLS models are compared per pair:
2026-09-22 19:29:53 +00:00
- **Restricted model** — constant plus `q` own lags of the lower-TF return
- **Unrestricted model** — restricted model plus `p` lags of the completed higher-TF return
2026-09-22 19:28:33 +00:00
If adding the higher-TF lags reduces the residual sum of squares by more than chance would allow, the F-statistic is large and the pair is a candidate for VALID status.
2026-09-22 19:29:53 +00:00
### Rolling Stability
The F-test is repeated on every consecutive window of `WINDOW` bars, stepping forward by `STEP`. The fraction of windows where the pair is significant is its stability score. A pair passes only when both its adjusted p-value and its stability clear their thresholds.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### VALID / INVALID Decision
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
| Condition | Required |
|-----------|----------|
| Adjusted p-value (BH) < ALPHA | Yes |
| Rolling stability ≥ STABILITY_MIN | Yes |
| OOS directional edge > 0 | Optional (REQUIRE_OOS_EDGE) |
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
## Algorithm / Architecture Summary
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 1. Data Fetch (Python)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
OHLCV data is pulled from MetaTrader 5 using `copy_rates_range` for all four timeframes. `copy_rates_from_pos` is not used because it fails with Invalid params at the bar counts required for M5 over 150 days. The last (forming) bar is always dropped before any calculation.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 2. Feature Engineering (Python)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
Prices are converted to log returns in basis points (`SCALE * np.log(close / close.shift(1))`). ADF tests confirm prices are non-stationary and returns are stationary. The design matrix is built with `np.searchsorted` to enforce the look-ahead-free cutoff without any forward fill.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 3. Label Generation (Python)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
The target for each lower-TF bar is its own log return, in basis points. This is a regression problem, not a classification problem. The "label" is the raw return the model is trying to forecast. Sign of the forecast becomes the vote direction; magnitude relative to return std controls the threshold via `InpMinEdgeSigma`.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 4. Granger F-Test (Python)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
Lags are selected by BIC over a grid of `p` from 1 to `MAX_P` and `q` from 1 to `MAX_Q` on the training 70%. The hand-written F-test is cross-checked against `sm.OLS(...).compare_f_test(...)` with an assert. An HC3 robust Wald test on the higher-TF coefficients is run alongside to flag cases where heteroskedasticity inflates the classical F.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 5. Rolling Stability and Correction (Python)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
The rolling scan uses the same `WINDOW` and `ALPHA` the EA will use live. Benjamini–Hochberg is applied across all four pairs to control the false discovery rate.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 6. Export (Python)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
The calibration table is written to `Terminal\Common\Files` with `\r\n` line endings and plain ASCII so the EA can read it with `FILE_TXT | FILE_ANSI`. One row per pair: `lead_tf, lag_tf, p, q, window, alpha, f_stat, p_value, p_adj, stability, oos_hit, oos_edge_bps, status`.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 7. EA Deployment (MQL5)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
The EA reads the CSV on `OnInit`. On each new signal bar it re-runs the F-test on the most recent `WINDOW` bars per pair (every `InpRetestBars` lower-TF bars), computes a return forecast from the fitted coefficients, and casts a vote. Votes from VALID pairs only are aggregated into the final signal. A trade opens only when the signal changes.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
### 8. Position Management (MQL5)
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
Stop loss and take profit are set as ATR multiples on the signal timeframe at entry. The opposite-signal exit is optional via `InpCloseOnOpposite`.
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
## Signal Generation
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
**Buy signal** — at least `InpMinVotes` VALID pairs forecast a positive return, zero forecast a negative return
**Sell signal** — at least `InpMinVotes` VALID pairs forecast a negative return, zero forecast a positive return
## Risk Management
2026-09-22 19:28:33 +00:00
Stop loss distance:
2026-09-22 19:29:53 +00:00
```
2026-09-22 19:28:33 +00:00
SL = ATR(InpATRPeriod) × InpSL_ATR
2026-09-22 19:29:53 +00:00
```
2026-09-22 19:28:33 +00:00
Take profit distance:
2026-09-22 19:29:53 +00:00
```
2026-09-22 19:28:33 +00:00
TP = ATR(InpATRPeriod) × InpTP_ATR
2026-09-22 19:29:53 +00:00
```
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
## A/B Testing
Setting `InpUseCausalityFilter = false` disables the causality gate entirely. The EA then signals from all pairs regardless of VALID/INVALID status, acting as the unfiltered baseline. Running both configurations over the same tester period isolates the filter's contribution to performance.
2026-09-22 19:28:33 +00:00
Note: because the filter changes the signal frequency significantly, the two modes may require separate optimisation passes. Applying filter-OFF optimised inputs with the filter ON often results in no trades.
2026-09-22 19:29:53 +00:00
## Granger Causality: Concept Mapping
| Granger Term | Trading Interpretation |
|---|---|
| Lead series | Higher timeframe bar returns |
| Lag series | Lower timeframe bar returns |
| Restricted model | Forecast from own past returns only |
| Unrestricted model | Forecast from own past returns plus completed higher-TF returns |
| F-statistic | Evidence that the higher TF adds predictive information |
| p-value (adjusted) | Probability of the observed F under the null of no causality |
| Rolling stability | Fraction of time windows where the relationship is significant |
| VALID | Pair cleared all thresholds — allowed to vote |
| INVALID | Pair failed — blocked from voting when filter is on |
| Vote | Sign of the forecast: +1 BUY, -1 SELL, 0 flat |
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
## Mentioned or Attached Files
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
**Primary Components**
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
- `GrangerMTF_Causality.ipynb`
- `GrangerMTF.mq5`
- `README.md`
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
**Generated Artifacts**
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
- `granger_mtf_causality.csv` — calibration table written to `Terminal\Common\Files` by the notebook
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
## Statistics
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
**Pipeline Components**
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
- Data Fetch Module: 1
- Stationarity Check Module: 1
- Alignment and Feature Engineering Module: 1
- Granger F-Test Module: 1
- Rolling Stability Module: 1
- Multiple Testing Correction Module: 1
- Export Module: 1
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
**EA Components**
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
- CSV Loader: 1
- OLS Solver (Gaussian elimination): 1
- Live F-Test Engine: 1
- Forecast and Vote Engine: 1
- Signal Aggregator: 1
- Trade Execution Module: 1
- Panel and Visualisation Module: 1
- Global Variable Export Module: 1
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
**Timeframe Pairs**
2026-09-22 19:28:33 +00:00
2026-09-22 19:29:53 +00:00
- Total pairs tested: 4
- Pairs marked VALID on EURUSD (150 days): 1
## Tags
`MQL5` `MetaTrader5` `GrangerCausality` `MultiTimeframe` `StatisticalFilter` `OLS` `F-Test` `TimeSeries` `Expert-Advisor` `Algorithmic-Trading` `Python` `Statsmodels` `Scipy` `Feature-Engineering` `Strategy-Testing` `RollingWindow` `BenjaminiHochberg`
## Difficulty
2026-09-22 19:28:33 +00:00
Moderate to Advanced — requires familiarity with:
2026-09-22 19:29:53 +00:00
- MQL5 Expert Advisor development
- Python data science toolchain (pandas, numpy, scipy, statsmodels)
- Linear regression and F-test mechanics
- Time series stationarity concepts
- Multi-timeframe bar alignment and look-ahead bias
- Rolling window statistical analysis
- Multiple hypothesis testing correction
## Limitations
- Causality is tested on a single symbol — same-symbol timeframes share the same price process, so a VALID result means longer history adds predictive information beyond short own lags, not that an external driver exists.
- Rolling stability at 43% for M30 → M15 means the relationship is absent in more windows than it is present. The live re-test mitigates this but does not eliminate it.
- The 150-day calibration window overlaps the backtest period in the article. Offline VALID/INVALID labels are partly in-sample relative to what the EA traded.
- Out-of-sample directional hit rate for the one VALID pair was 50.0%, below the restricted-model baseline. Statistical significance did not translate to bar-by-bar directional accuracy.
- The OLS solver in MQL5 adds a small ridge term (1e-9) for numerical stability. This is not present in the Python version and introduces a negligible but non-zero difference in coefficients.
## Future Improvements
2026-09-22 19:28:33 +00:00
Potential enhancements include:
2026-09-22 19:29:53 +00:00
- Calibrating the notebook on data that ends before the backtest starts for a fully out-of-sample test
- Regime-specific VALID/INVALID thresholds (trending vs. ranging conditions)
- Cross-symbol Granger tests where lead and lag are different instruments
- Adaptive `STABILITY_MIN` that tightens in low-volatility regimes
- Confidence-based position sizing proportional to the forecast magnitude
- Periodic notebook recalibration and automatic CSV refresh
- Multi-symbol training to identify which pairs Granger-cause each other across instruments
- Integration with the Part 10 AutoML pipeline to use Granger-filtered signals as ML features
## Reference
Original MQL5 article: [https://www.mql5.com/en/articles/24649](https://www.mql5.com/en/articles/24649)