TICC/README.md

# TICC

Toeplitz Inverse Covariance Clustering in native MQL5. Estimates a sparse precision
matrix per market regime, so a regime is defined by which symbols are conditionally
dependent and at what lag, not by how volatile the market is.

Code for the article
[Toeplitz Inverse Covariance Clustering in MQL5](https://www.mql5.com/en/articles/24064).
The article explains the method and reports the measured results. This repo is the
source it describes.

## What is here

A correlation matrix over FX majors is mostly the dollar leg showing up five times.
Inverting it gives the precision matrix, whose off-diagonal entries are partial
correlations: the relationship between two symbols once every other symbol is
accounted for. An L1 penalty drives unsupported entries to exact zero, so the
estimate reads as a graph instead of a heatmap.

Each bar is stacked into a vector of `n` symbols over `w` bars, so the precision
matrix covers lead and lag as well as same-bar structure. The block-Toeplitz
constraint forces the model to be time invariant inside the window. For n = 5 and
w = 5 that collapses 625 matrix entries to 115 free parameters.

The solver is ADMM. The theta step reduces to one scalar quadratic per eigenvalue,
so there is no matrix inversion anywhere and the result is positive definite by
construction. The Z step averages each Toeplitz group and soft-thresholds it, which
is the exact proximal operator, not a projection applied afterwards.

Regimes are fitted by alternating the solver with a Viterbi pass over the whole
label path, penalising switches by `beta`. Labels are therefore persistent rather
than flipping on single-bar noise.

Linear algebra uses the MQL5 `matrix` and `vector` types. ALGLIB is used only to
seed the initial clustering. No Python, no DLLs.

## Layout

```
MQL5/Include/TICC/TICC_Types.mqh        config, regime and report structs, file paths
MQL5/Include/TICC/TICC_Math.mqh         block-Toeplitz group map, penalty taper, eigen helpers
MQL5/Include/TICC/TICC_GraphLasso.mqh   Toeplitz graphical lasso by ADMM
MQL5/Include/TICC/TICC_Engine.mqh       k-means seeding, Viterbi assignment, EM loop, persistence
MQL5/Include/TICC/TICC_Data.mqh         timestamp alignment, log returns, standardisation
MQL5/Include/TICC/TICC_Live.mqh         chart-side causal labelling
MQL5/Scripts/TICC/TICC_SolverTest.mq5   solver checks against a known sparsity pattern
MQL5/Scripts/TICC/TICC_Fit.mq5          fits regimes, model selection, held-out diagnostics
MQL5/Indicators/TICC/TICC_Regime.mq5    non-repainting regime ribbon
MQL5/Indicators/TICC/TICC_GraphPanel.mq5  dependency graph of the active regime
MQL5/Files/TICC/ticc_c_k6_b100.txt      fitted model the indicators default to
```

Copy the folders into your terminal's `MQL5` directory and compile.

## Running it

Run in this order. Each step assumes the previous one passed.

1. `TICC_SolverTest.mq5` on any chart. Needs no market data, takes a few seconds,
   prints twenty checks. If any fail, stop here.
2. `TICC_Fit.mq5` with `InpSaveModel` set to a file name. Put every basket symbol in
   Market Watch first, since it pulls history for all of them. Writes to
   `MQL5\Files\TICC\`, then reloads and verifies the copy reproduces the same labels.
3. `TICC_Regime.mq5` on a chart, `InpModelFile` set to that file. The model carries
   its own basket, so the chart symbol does not have to be one of them.
4. `TICC_GraphPanel.mq5` on the same chart, same model file, same commit lag.

The shipped model `ticc_c_k6_b100.txt` lets you skip step 2 and attach the
indicators directly.

## Settings behind the shipped model

EURUSD, GBPUSD, USDJPY, AUDUSD, USDCHF on H1, w = 5, K = 6, lambda = 0.01,
taper sigma = 2.5, beta = 100, commit lag 5, 120-bar causal rolling standardisation,
two-thirds training split.

`InpBars` counts back from whenever you run it, so refitting on a different day
gives a different window. ALGLIB seeds k-means from `MathRand`, so set `InpSeed`
if you want a fit to reproduce.

## Non-repainting

The batch fit runs a backward pass and therefore sees the future. The indicators do
not use it. They freeze the fitted regimes and run the recursion forward only,
committing a label once the path behind it can no longer be revised. The cost is
that the label arrives `InpCommitLag` bars late, so the newest bars stay unpainted.
That gap is real and painting it would be repainting.

## Things that cost me time

Four MQL5 behaviours worth knowing, all of which fail silently:

- `EigenSymmetricDC` rejects a matrix that is asymmetric in the last bit. A
  covariance built as `X^T X` always is, because the two dot products accumulate in
  different orders. Symmetrise before every call. Hand-built test matrices pass, so
  this only shows up on real data.
- ALGLIB seeds its k-means from `MathRand` on first use. Without an explicit
  `MathSrand` a fit will not reproduce even on identical data.
- A convergence test that can never be met turns the iteration cap into a hidden
  hyperparameter that decides the answer.
- An indicator that latches its recalculation guard before doing the work turns one
  early failure into a permanent one. Latch it only after a pass that painted.
  Returning `rates_total` when nothing was drawn tells every `iCustom` caller that
  an empty buffer is a finished answer.

## References

- Hallac, Vare, Boyd, Leskovec (2017). Toeplitz Inverse Covariance-Based Clustering
  of Multivariate Time Series Data. KDD 2017.
- Boyd, Parikh, Chu, Peleato, Eckstein (2011). Distributed Optimization and
  Statistical Learning via the Alternating Direction Method of Multipliers.
  Foundations and Trends in Machine Learning 3(1).
- Friedman, Hastie, Tibshirani (2008). Sparse Inverse Covariance Estimation with the
  Graphical Lasso. Biostatistics 9(3).
- Dempster (1972). Covariance Selection. Biometrics 28(1).

## Disclaimer

Educational code. Past behaviour of any model or dataset says nothing about future
results. Test on your own data and broker conditions before drawing conclusions.