Article-24649-GrangerMTF-Causality-Detection/README.md

# Article-24649-GrangerMTF-Causality-Detection

This repository is an article-derived reference project based on the original MQL5 article. It does not claim to reproduce the full original source code unless files are explicitly attached.

## Overview

This project documents a complete multi-timeframe causality detection pipeline for MetaTrader 5. Instead of assuming that a higher timeframe always informs the lower, the system tests that relationship statistically and uses the result to filter signals at runtime.

The pipeline automatically:

- Fetches historical EURUSD OHLC data for M5, M15, M30 and H1 directly from MetaTrader 5
- Converts prices to log returns and verifies stationarity with an ADF test
- Builds look-ahead-free design matrices aligned so each lower-TF bar only sees already-closed higher-TF bars
- Selects lag structure by BIC and fits restricted and unrestricted OLS models for each timeframe pair
- Runs a Granger F-test, cross-checks it against statsmodels, and adds an HC3 robust Wald test
- Scans rolling windows to measure how stable each relationship is over time
- Corrects for multiple comparisons across all four pairs with Benjamini–Hochberg
- Exports a calibration table to `Terminal\Common\Files` for the EA to read
- Repeats the same F-test live on a rolling window inside MQL5 at runtime
- Marks each pair VALID or INVALID based on both the offline table and the live result
- Generates BUY/SELL signals from the votes of causally valid pairs only
- Gates every trade through the live causality check before execution
- Manages positions with ATR-based stop loss and take profit
- Publishes pair statuses as terminal global variables for use by other EAs

The project demonstrates how a statistical test developed in Python can be reproduced exactly in MQL5 so that the offline research and the live system stay faithful to the same rules.

## Original Article

**Article ID:** 24649
**Author:** Hlomohang John Borotho
**Category:** Experts
**URL:** [https://www.mql5.com/en/articles/24649](https://www.mql5.com/en/articles/24649)

## Repository Purpose

This repository should be treated as a reference or reconstruction project derived from the article content.

Its purpose is to:

- Test whether higher timeframes statistically Granger-cause lower timeframes before trading on that assumption
- Demonstrate look-ahead-free multi-timeframe alignment in both Python and MQL5
- Reproduce an F-test and rolling causality monitor natively in MQL5 without external libraries
- Provide a reusable causality-filtering framework that works on any symbol or timeframe pair
- Serve as a foundation for regime-aware Expert Advisor development

## Key Concepts

### Timeframe Pairs

The system tests four lead → lag relationships. The lead must always be the higher timeframe.

| Pair | Lead | Lag | EURUSD Result |
|------|------|-----|---------------|
| H1 → M15 | H1 | M15 | INVALID (F ≈ 0, stab 9%) |
| M30 → M15 | M30 | M15 | VALID (F = 9.64, stab 43%) |
| M15 → M5 | M15 | M5 | INVALID (F = 4.84, stab 20%) |
| H1 → M5 | H1 | M5 | INVALID (F ≈ 0, stab 6%) |

### Look-Ahead-Free Alignment

For each lower-TF bar opening at time `t`, only higher-TF bars whose open time is at or before `t - lead_period` are used as regressors. This rule is identical in Python and MQL5. Any deviation between the two stages would make the offline coefficients meaningless at runtime.

### Granger F-Test

Two OLS models are compared per pair:

- **Restricted model** — constant plus `q` own lags of the lower-TF return
- **Unrestricted model** — restricted model plus `p` lags of the completed higher-TF return

If adding the higher-TF lags reduces the residual sum of squares by more than chance would allow, the F-statistic is large and the pair is a candidate for VALID status.

### Rolling Stability

The F-test is repeated on every consecutive window of `WINDOW` bars, stepping forward by `STEP`. The fraction of windows where the pair is significant is its stability score. A pair passes only when both its adjusted p-value and its stability clear their thresholds.

### VALID / INVALID Decision

| Condition | Required |
|-----------|----------|
| Adjusted p-value (BH) < ALPHA | Yes |
| Rolling stability ≥ STABILITY_MIN | Yes |
| OOS directional edge > 0 | Optional (REQUIRE_OOS_EDGE) |

## Algorithm / Architecture Summary

### 1. Data Fetch (Python)

OHLCV data is pulled from MetaTrader 5 using `copy_rates_range` for all four timeframes. `copy_rates_from_pos` is not used because it fails with Invalid params at the bar counts required for M5 over 150 days. The last (forming) bar is always dropped before any calculation.

### 2. Feature Engineering (Python)

Prices are converted to log returns in basis points (`SCALE * np.log(close / close.shift(1))`). ADF tests confirm prices are non-stationary and returns are stationary. The design matrix is built with `np.searchsorted` to enforce the look-ahead-free cutoff without any forward fill.

### 3. Label Generation (Python)

The target for each lower-TF bar is its own log return, in basis points. This is a regression problem, not a classification problem. The "label" is the raw return the model is trying to forecast. Sign of the forecast becomes the vote direction; magnitude relative to return std controls the threshold via `InpMinEdgeSigma`.

### 4. Granger F-Test (Python)

Lags are selected by BIC over a grid of `p` from 1 to `MAX_P` and `q` from 1 to `MAX_Q` on the training 70%. The hand-written F-test is cross-checked against `sm.OLS(...).compare_f_test(...)` with an assert. An HC3 robust Wald test on the higher-TF coefficients is run alongside to flag cases where heteroskedasticity inflates the classical F.

### 5. Rolling Stability and Correction (Python)

The rolling scan uses the same `WINDOW` and `ALPHA` the EA will use live. Benjamini–Hochberg is applied across all four pairs to control the false discovery rate.

### 6. Export (Python)

The calibration table is written to `Terminal\Common\Files` with `\r\n` line endings and plain ASCII so the EA can read it with `FILE_TXT | FILE_ANSI`. One row per pair: `lead_tf, lag_tf, p, q, window, alpha, f_stat, p_value, p_adj, stability, oos_hit, oos_edge_bps, status`.

### 7. EA Deployment (MQL5)

The EA reads the CSV on `OnInit`. On each new signal bar it re-runs the F-test on the most recent `WINDOW` bars per pair (every `InpRetestBars` lower-TF bars), computes a return forecast from the fitted coefficients, and casts a vote. Votes from VALID pairs only are aggregated into the final signal. A trade opens only when the signal changes.

### 8. Position Management (MQL5)

Stop loss and take profit are set as ATR multiples on the signal timeframe at entry. The opposite-signal exit is optional via `InpCloseOnOpposite`.

## Signal Generation

**Buy signal** — at least `InpMinVotes` VALID pairs forecast a positive return, zero forecast a negative return

**Sell signal** — at least `InpMinVotes` VALID pairs forecast a negative return, zero forecast a positive return

## Risk Management

Stop loss distance:

```
SL = ATR(InpATRPeriod) × InpSL_ATR
```

Take profit distance:

```
TP = ATR(InpATRPeriod) × InpTP_ATR
```

## A/B Testing

Setting `InpUseCausalityFilter = false` disables the causality gate entirely. The EA then signals from all pairs regardless of VALID/INVALID status, acting as the unfiltered baseline. Running both configurations over the same tester period isolates the filter's contribution to performance.

Note: because the filter changes the signal frequency significantly, the two modes may require separate optimisation passes. Applying filter-OFF optimised inputs with the filter ON often results in no trades.

## Granger Causality: Concept Mapping

| Granger Term | Trading Interpretation |
|---|---|
| Lead series | Higher timeframe bar returns |
| Lag series | Lower timeframe bar returns |
| Restricted model | Forecast from own past returns only |
| Unrestricted model | Forecast from own past returns plus completed higher-TF returns |
| F-statistic | Evidence that the higher TF adds predictive information |
| p-value (adjusted) | Probability of the observed F under the null of no causality |
| Rolling stability | Fraction of time windows where the relationship is significant |
| VALID | Pair cleared all thresholds — allowed to vote |
| INVALID | Pair failed — blocked from voting when filter is on |
| Vote | Sign of the forecast: +1 BUY, -1 SELL, 0 flat |

## Mentioned or Attached Files

**Primary Components**

- `GrangerMTF_Causality.ipynb`
- `GrangerMTF.mq5`
- `README.md`

**Generated Artifacts**

- `granger_mtf_causality.csv` — calibration table written to `Terminal\Common\Files` by the notebook

## Statistics

**Pipeline Components**

- Data Fetch Module: 1
- Stationarity Check Module: 1
- Alignment and Feature Engineering Module: 1
- Granger F-Test Module: 1
- Rolling Stability Module: 1
- Multiple Testing Correction Module: 1
- Export Module: 1

**EA Components**

- CSV Loader: 1
- OLS Solver (Gaussian elimination): 1
- Live F-Test Engine: 1
- Forecast and Vote Engine: 1
- Signal Aggregator: 1
- Trade Execution Module: 1
- Panel and Visualisation Module: 1
- Global Variable Export Module: 1

**Timeframe Pairs**

- Total pairs tested: 4
- Pairs marked VALID on EURUSD (150 days): 1

## Tags

`MQL5` `MetaTrader5` `GrangerCausality` `MultiTimeframe` `StatisticalFilter` `OLS` `F-Test` `TimeSeries` `Expert-Advisor` `Algorithmic-Trading` `Python` `Statsmodels` `Scipy` `Feature-Engineering` `Strategy-Testing` `RollingWindow` `BenjaminiHochberg`

## Difficulty

Moderate to Advanced — requires familiarity with:

- MQL5 Expert Advisor development
- Python data science toolchain (pandas, numpy, scipy, statsmodels)
- Linear regression and F-test mechanics
- Time series stationarity concepts
- Multi-timeframe bar alignment and look-ahead bias
- Rolling window statistical analysis
- Multiple hypothesis testing correction

## Limitations

- Causality is tested on a single symbol — same-symbol timeframes share the same price process, so a VALID result means longer history adds predictive information beyond short own lags, not that an external driver exists.
- Rolling stability at 43% for M30 → M15 means the relationship is absent in more windows than it is present. The live re-test mitigates this but does not eliminate it.
- The 150-day calibration window overlaps the backtest period in the article. Offline VALID/INVALID labels are partly in-sample relative to what the EA traded.
- Out-of-sample directional hit rate for the one VALID pair was 50.0%, below the restricted-model baseline. Statistical significance did not translate to bar-by-bar directional accuracy.
- The OLS solver in MQL5 adds a small ridge term (1e-9) for numerical stability. This is not present in the Python version and introduces a negligible but non-zero difference in coefficients.

## Future Improvements

Potential enhancements include:

- Calibrating the notebook on data that ends before the backtest starts for a fully out-of-sample test
- Regime-specific VALID/INVALID thresholds (trending vs. ranging conditions)
- Cross-symbol Granger tests where lead and lag are different instruments
- Adaptive `STABILITY_MIN` that tightens in low-volatility regimes
- Confidence-based position sizing proportional to the forecast magnitude
- Periodic notebook recalibration and automatic CSV refresh
- Multi-symbol training to identify which pairs Granger-cause each other across instruments
- Integration with the Part 10 AutoML pipeline to use Granger-filtered signals as ML features

## Reference

Original MQL5 article: [https://www.mql5.com/en/articles/24649](https://www.mql5.com/en/articles/24649)