# VALIDATION_PROTOCOL.md

## 1. Primary method: chronological walk-forward

```
training window
  -> forecast future observation/window
  -> observe actual outcome at origin+H
  -> advance origin
  -> repeat
```

supported windows: expanding, rolling, with configurable train length and horizon.

All data used in a forecast is strictly before the forecast origin. The outcome
is revealed only after the prediction is frozen (origin + H).

## 2. Baselines

| Variant | Description |
| ------- | ----------- |
| A | Naive (random-walk / last value), normalized zero-return where appropriate |
| B | ARIMA only |
| C | SAX only |
| D | ARIMA + SAX |
| E | Agreement-only filter |
| F | Disagreement/uncertainty rejection |

No variant may claim superiority unless it beats the same protocol baselines.

## 3. Metrics

- **Forecast**: MAE, RMSE, MASE (scale-aware), directional accuracy, calibration
  of directional probability.
- **Distribution/Evidence**: analog count dist, interval widths, SAX p25-p75,
  ARIMA PI width, agreement/disagreement/no-edge/insufficient rates.
- **Economic** (only after forecast metrics): gross/net expected return, expectancy,
  profit factor, max drawdown, Sharpe/Sortino where appropriate, trade frequency,
  turnover, cost sensitivity. Cost assumptions explicit.

## 4. Ablation matrix and experiments

E1: ARIMA > naive?  E2: SAX > naive?  E3: SAX + info beyond ARIMA?
E4: ARIMA + info beyond SAX?  E5: agreement -> improved conditional performance?
E6: disagreement -> poor/unstable regime?  E7: survives costs?
E8: stability across walk-forward periods?

Do not proceed to optimization before E1–E8 are answered.

## 5. Reproducibility

Every report records: commit, source version, config, symbol, timeframe, data range,
data source, horizon `H`, model parameters, seeds, cost assumptions, validation
protocol, artifacts.

## 6. No-lookahead enforcement

A single integration test drives the whole pipeline; it fails hard on any
forward-contamination. A failing no-lookahead test is a **HARD STOP**.