Warrior_EA/research/altdata/DESIGN.md

# Alternative Data — Design & Source Adjudication

2026-08-16. Context: the private-use pivot (marketplace variant dropped; DLLs,
Python, WebRequest all permitted). Motivation: every price-derived signal
family measured to date is real-but-priced-to-cost (see the cost-boundary
verdicts). Alternative data is information the retail cost boundary has not
consumed — but only if the lookahead discipline below is followed exactly.

## Decisions taken (user, 2026-08-16)

- MQL5 Marketplace variant **dropped**. Private bot; any future sale is via the
  user's own site, unrestricted. Consequences: DLL import restrictions gone,
  WebRequest whitelisting is our own choice, the OpenCL-as-client-tier
  constraint is obsolete, The5ers copy-trading collision dissolved.
- Classic signals: to be **removed** for codebase hygiene (staged, separate
  refactor — see "EA-side plan").
- Wyckoff: **keep** (only feature family with replicated context value);
  optimize the auto-tuner instead of deleting; prune input-menu surface.
- Bridge preference: MQL5 + files/DLLs. Python is an external collector only;
  the EA must never block on Python at trade time.

## The two rules that keep alt data honest

1. **Vintage rule.** Every row carries `observed` (the period described) and
   `published` (when it became publicly knowable, UTC). All joins to price bars
   use `published <= bar_open`. Revised series (FRED, EIA) must be fetched as
   FIRST PRINTS (ALFRED vintages), never current values — the calendar
   recorder's post-revision trap generalizes to every macro source.
2. **Family-wise rule.** Every feature family added here multiplies the
   best-of-N space. Any "this alt feature clears" claim goes under the null of
   the maximum, same as always. Alt data changes the information set, not the
   statistics discipline.

## Source adjudication (the pasted list, corrected)

| Source | Verdict | Notes |
|---|---|---|
| **CFTC COT** (not in the list!) | **BUILT, no key, 2010→now on disk** | Weekly positioning: TFF (ES, VIX, BTC, EUR/JPY/CAD/GBP futures) + Disaggregated (GC, CL). The best free source for this project: 16y of history = trainable TODAY, documented literature (positioning extremes / COT index), weekly cadence fits D1. Sign flip needed for USDJPY/USDCAD (futures are JPY/USD, CAD/USD). |
| **FRED/ALFRED** | **BUILT, needs free key** | 800k series, but the value is ALFRED vintages (first prints). Daily: VIX, yields, breakevens, dollar index, WTI spot; weekly: NFCI, Fed balance sheet. |
| **EIA weekly petroleum** | **BUILT, needs free key** | Crude stocks/production/refinery utilization for XTIUSD. Weekly, Wednesday 10:30 ET release. The *surprise* vs consensus is the tradeable object; consensus history is the hard part (record forward). |
| Options OI / GEX — **CBOE delayed-quotes CDN** | **BUILT (recorder), no key** | `cdn.cboe.com/api/global/delayed_quotes/options/{TICKER}.json` — supersedes the yfinance route and the earlier "CBOE is UA-gated" note, which was about a *different* CBOE path: this CDN returns 200 with no User-Agent at all (verified 2026-08-16, four UA variants), so MQL5 `WebRequest` reaches it unaided. Ships per-contract **gamma and open interest**, so no pricing model — and no model risk — enters the recorded data. Validated on `_SPX`: 29,362 rows, 20,993 with nonzero gamma, 54 expiries, 13 MB, total GEX +90.7 Bn/1%. Still SNAPSHOT-ONLY: the EA records one row per trading day (`gex_{CANONICAL}.csv`) and feeds nothing until the file is long enough to screen. |
| **FlashAlpha** | **NOT BUYING — history is paywalled at $1,499/mo** | Sells point-in-time chain replay since 2018-04 with per-strike OI + gamma, which would have let us compute *our own* GEX construction on their history and screen years early. Killed by pricing (checked 2026-08-16): **the Historical API is Alpha-tier exclusive at $1,499/month**; Basic ($79, 250 req/day) and Growth ($299) are LIVE only. The archive was the only thing we wanted, so no affordable tier helps — Basic would buy 15-second freshness for a once-a-day snapshot, worth nothing. **Free tier probed 2026-08-16 (3 requests), tier gates now known empirically:** ETFs (SPY/QQQ/IWM) need Basic; full-chain all-expirations needs Growth; historical needs Alpha — so free/Basic are single-expiry-per-request. The one 200 response (`AAPL?expiration=2026-09-18`) had the right schema but a **nearly empty payload**: 13 of 93 strikes with any OI, total call OI 4,296 vs CBOE's 373,253 for the same expiry, put OI zero, every near-money strike blank. **No validation was possible** — nothing to compare against, and from outside we cannot separate free-tier degradation from their flow-signed methodology (strikes without classified tape may simply not appear). Net: **the free CBOE CDN is strictly better than Basic here** — complete chains, all expiries and strikes, gamma + OI populated, unlimited, $0. Note `github.com/FlashAlpha-lab/awesome-options-analytics` is this vendor's own list, not a neutral survey. |
| Historical option-chain archives (if we ever want to skip the wait) | **Price-check bulk sellers, not analytics APIs** | The distinction that matters: analytics APIs charge for *live convenience* and meter requests; bulk vendors sell the *archive* as files, which is what a one-off backfill actually needs. Candidates to price: CBOE DataShop (one-time dataset purchase), historicaloptiondata.com, Thetadata, Polygon, ORATS. But note the EV: the recorder banks this history for free in ~12 months, and on this project's own base rate most alt-data families die at the incremental gate (EIA null, tick-flow direction null, drift dead to financing). Paying four figures to test GEX a year early is a poor trade; waiting costs nothing. |
| Finnhub news/sentiment | Deferred | Free tier is real but news sentiment history is shallow; social sentiment endpoints have moved tiers repeatedly. Forward-record if used; never backfill from a "historical sentiment" endpoint whose values were computed retroactively. |
| Stocktwits | Deferred | API locked down significantly; scraping ToS-fragile. Low priority. |
| NASA FIRMS (flares) | Parked | Reachable, free key. Creative but unproven proxy chain (thermal anomaly → refinery outage → CL price) with no cheap validation path. Revisit after COT/EIA prove or disprove the pipeline. Re-raised by the user 2026-08-16 — nothing changed: still point detections needing aggregation through the same proxy chain. |
| **NASA POWER** (daily weather, no key) | **QUEUED behind NATGAS becoming a traded symbol** | The one NASA API with a mechanism-bearing, numeric, daily series for a catalog instrument: temperature → heating/cooling degree days → natural-gas demand is the textbook gas fundamental. But it is per-lat/lon point data, so the feature (US population-weighted HDD/CDD) must be CONSTRUCTED from ~10 city points — and NOAA CPC publishes ready-made national degree-day series, likely the better source for the same object. Neither of the currently traded symbols (SP500/EURUSD/USDJPY/XAUUSD) is weather-linked at range horizon (weather→equities is the Hirshleifer sunshine literature: direction, tiny, sentiment-based — not range). Wire when a NATGAS/agri chart exists, WITH bars to screen against; writing fetch code today for a symbol nobody attaches means it first runs months from now, unobserved — the silent-FRED incident's exact shape. |
| NASA GIBS (satellite imagery) | Not consumable | Imagery tile layers, not numeric series. Extracting a tradeable number needs a CV pipeline; our CONV is 1D by design (see the satellite adjudication — NASA's own CNNs already ship EXTRACTED point products, which is what FIRMS is). If imagery-derived data ever matters, buy the extracted product, don't rebuild the extractor. |
| Binance/CoinGecko (BTC) | Parked until broker BTC confirmed | Binance Vision does have deep free history (trades, funding, OI) — genuinely good if BTCUSD becomes a traded symbol. COT Bitcoin already collected (2017→now). |
| `PythonExecute()` from MQL5 | **Does not exist** | MT5's Python support is the `MetaTrader5` package: an external Python process connecting TO the terminal. The EA-side bridge is files (chosen) or a DLL; WebRequest is available for live-time pulls if ever needed. |

## Architecture

```
[scheduled task, e.g. hourly]
python -m altdata.cot / fred / eia / recorder      (research/altdata/*)
        │  writes observed,published,value CSVs
        ▼
c:\...\Workspaces\Market Data\altdata\{cot,fred,eia,...}\*.csv   (research root)
        │  python -m altdata.join  (as-of join, per-symbol feature matrix)
        ▼
Common\Files\Warrior_EA\AltData\{SYMBOL}_D1.csv    (EA-facing contract)
        │  EA: FileOpen(...,FILE_READ|FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_COMMON)
        ▼
feature window → NN input block (alt features appended like the meta descriptor)
```

EA contract (to implement): one CSV per symbol, rows `bar_time;f1;..;fn`,
header row names the features, sidecar `.meta` carries feature count + last
update stamp. Missing/stale file ⇒ features fed as "missing" flags, EA trades
on (degraded, never blocked). Tester reads the same files — history is fully
reproducible because every row was as-of joined.

## Feature derivation (join.py, next)

Per symbol from COT (the literature-standard set, deliberately small):
- net speculative position / open interest (Lev_Money or M_Money net ÷ OI)
- 1y and 3y COT index (percentile of net position) — positioning EXTREMES
- 4-week change in net position
- dealer/commercial net (the other side)
- USDJPY/USDCAD: sign flipped at this step; sidecar documents it
From FRED: level + 5-day change of VIX, T10Y2Y, DTWEXBGS, T5YIE; NFCI weekly.
From EIA (XTIUSD only): stocks level vs 5y seasonal norm, 1w change.

First measurement (before any NN touches this): the existing MI/permutation
harness over these features vs the excursion targets (RANGE first — the proven
predictable channel — then the barrier label), family-wise gated, on D1. Cheap,
uses only already-validated machinery, and tells us which families earn a slot
in the feature window.

## EA-side plan (staged, in order)

1. `join.py` + first MI measurement (offline only, no EA changes).
2. AltData reader module in the EA (mirrors MetaCorpus.mqh patterns; share
   flags per the tester FileCopy lesson).
3. Classic-signal removal refactor (separate branch; the sweep/corpus and
   signal-DB machinery is entangled with classic patterns — the candidate
   stream they provided has to be replaced or retired in the same commit).
4. Wyckoff tuner optimization + input-menu pruning.

## Keys needed (all free registrations, user action)

Put in `Market Data\altdata\keys.json` as `{"fred": "...", "eia": "..."}`:
- FRED: https://fred.stlouisfed.org/docs/api/api_key.html
- EIA: https://www.eia.gov/opendata/register.php
- (later, optional) Finnhub, NASA FIRMS.