Warrior_EA/research/altdata/DESIGN.md

Alternative Data — Design & Source Adjudication

2026-08-16. Context: the private-use pivot (marketplace variant dropped; DLLs, Python, WebRequest all permitted). Motivation: every price-derived signal family measured to date is real-but-priced-to-cost (see the cost-boundary verdicts). Alternative data is information the retail cost boundary has not consumed — but only if the lookahead discipline below is followed exactly.

Decisions taken (user, 2026-08-16)

• MQL5 Marketplace variant dropped. Private bot; any future sale is via the user's own site, unrestricted. Consequences: DLL import restrictions gone, WebRequest whitelisting is our own choice, the OpenCL-as-client-tier constraint is obsolete, The5ers copy-trading collision dissolved.
• Classic signals: to be removed for codebase hygiene (staged, separate refactor — see "EA-side plan").
• Wyckoff: keep (only feature family with replicated context value); optimize the auto-tuner instead of deleting; prune input-menu surface.
• Bridge preference: MQL5 + files/DLLs. Python is an external collector only; the EA must never block on Python at trade time.

The two rules that keep alt data honest

1. Vintage rule. Every row carries observed (the period described) and published (when it became publicly knowable, UTC). All joins to price bars use published <= bar_open. Revised series (FRED, EIA) must be fetched as FIRST PRINTS (ALFRED vintages), never current values — the calendar recorder's post-revision trap generalizes to every macro source.
2. Family-wise rule. Every feature family added here multiplies the best-of-N space. Any "this alt feature clears" claim goes under the null of the maximum, same as always. Alt data changes the information set, not the statistics discipline.

Source adjudication (the pasted list, corrected)

Source	Verdict	Notes
CFTC COT (not in the list!)	BUILT, no key, 2010→now on disk	Weekly positioning: TFF (ES, VIX, BTC, EUR/JPY/CAD/GBP futures) + Disaggregated (GC, CL). The best free source for this project: 16y of history = trainable TODAY, documented literature (positioning extremes / COT index), weekly cadence fits D1. Sign flip needed for USDJPY/USDCAD (futures are JPY/USD, CAD/USD).
FRED/ALFRED	BUILT, needs free key	800k series, but the value is ALFRED vintages (first prints). Daily: VIX, yields, breakevens, dollar index, WTI spot; weekly: NFCI, Fed balance sheet.
EIA weekly petroleum	BUILT, needs free key	Crude stocks/production/refinery utilization for XTIUSD. Weekly, Wednesday 10:30 ET release. The surprise vs consensus is the tradeable object; consensus history is the hard part (record forward).
Options OI / GEX — CBOE delayed-quotes CDN	BUILT (recorder), no key	cdn.cboe.com/api/global/delayed_quotes/options/{TICKER}.json — supersedes the yfinance route and the earlier "CBOE is UA-gated" note, which was about a different CBOE path: this CDN returns 200 with no User-Agent at all (verified 2026-08-16, four UA variants), so MQL5 WebRequest reaches it unaided. Ships per-contract gamma and open interest, so no pricing model — and no model risk — enters the recorded data. Validated on _SPX: 29,362 rows, 20,993 with nonzero gamma, 54 expiries, 13 MB, total GEX +90.7 Bn/1%. Still SNAPSHOT-ONLY: the EA records one row per trading day (gex_{CANONICAL}.csv) and feeds nothing until the file is long enough to screen.
FlashAlpha	NOT BUYING — history is paywalled at $1,499/mo	Sells point-in-time chain replay since 2018-04 with per-strike OI + gamma, which would have let us compute our own GEX construction on their history and screen years early. Killed by pricing (checked 2026-08-16): the Historical API is Alpha-tier exclusive at $1,499/month; Basic ($79, 250 req/day) and Growth ($299) are LIVE only. The archive was the only thing we wanted, so no affordable tier helps — Basic would buy 15-second freshness for a once-a-day snapshot, worth nothing. Free tier probed 2026-08-16 (3 requests), tier gates now known empirically: ETFs (SPY/QQQ/IWM) need Basic; full-chain all-expirations needs Growth; historical needs Alpha — so free/Basic are single-expiry-per-request. The one 200 response (AAPL?expiration=2026-09-18) had the right schema but a nearly empty payload: 13 of 93 strikes with any OI, total call OI 4,296 vs CBOE's 373,253 for the same expiry, put OI zero, every near-money strike blank. No validation was possible — nothing to compare against, and from outside we cannot separate free-tier degradation from their flow-signed methodology (strikes without classified tape may simply not appear). Net: the free CBOE CDN is strictly better than Basic here — complete chains, all expiries and strikes, gamma + OI populated, unlimited, $0. Note github.com/FlashAlpha-lab/awesome-options-analytics is this vendor's own list, not a neutral survey.
Historical option-chain archives (if we ever want to skip the wait)	Price-check bulk sellers, not analytics APIs	The distinction that matters: analytics APIs charge for live convenience and meter requests; bulk vendors sell the archive as files, which is what a one-off backfill actually needs. Candidates to price: CBOE DataShop (one-time dataset purchase), historicaloptiondata.com, Thetadata, Polygon, ORATS. But note the EV: the recorder banks this history for free in ~12 months, and on this project's own base rate most alt-data families die at the incremental gate (EIA null, tick-flow direction null, drift dead to financing). Paying four figures to test GEX a year early is a poor trade; waiting costs nothing.
Finnhub news/sentiment	Deferred	Free tier is real but news sentiment history is shallow; social sentiment endpoints have moved tiers repeatedly. Forward-record if used; never backfill from a "historical sentiment" endpoint whose values were computed retroactively.
Stocktwits	Deferred	API locked down significantly; scraping ToS-fragile. Low priority.
NASA FIRMS (flares)	Parked	Reachable, free key. Creative but unproven proxy chain (thermal anomaly → refinery outage → CL price) with no cheap validation path. Revisit after COT/EIA prove or disprove the pipeline. Re-raised by the user 2026-08-16 — nothing changed: still point detections needing aggregation through the same proxy chain.
NASA POWER (daily weather, no key)	QUEUED behind NATGAS becoming a traded symbol	The one NASA API with a mechanism-bearing, numeric, daily series for a catalog instrument: temperature → heating/cooling degree days → natural-gas demand is the textbook gas fundamental. But it is per-lat/lon point data, so the feature (US population-weighted HDD/CDD) must be CONSTRUCTED from ~10 city points — and NOAA CPC publishes ready-made national degree-day series, likely the better source for the same object. Neither of the currently traded symbols (SP500/EURUSD/USDJPY/XAUUSD) is weather-linked at range horizon (weather→equities is the Hirshleifer sunshine literature: direction, tiny, sentiment-based — not range). Wire when a NATGAS/agri chart exists, WITH bars to screen against; writing fetch code today for a symbol nobody attaches means it first runs months from now, unobserved — the silent-FRED incident's exact shape.
NASA GIBS (satellite imagery)	Not consumable	Imagery tile layers, not numeric series. Extracting a tradeable number needs a CV pipeline; our CONV is 1D by design (see the satellite adjudication — NASA's own CNNs already ship EXTRACTED point products, which is what FIRMS is). If imagery-derived data ever matters, buy the extracted product, don't rebuild the extractor.
Binance/CoinGecko (BTC)	Parked until broker BTC confirmed	Binance Vision does have deep free history (trades, funding, OI) — genuinely good if BTCUSD becomes a traded symbol. COT Bitcoin already collected (2017→now).
PythonExecute() from MQL5	Does not exist	MT5's Python support is the MetaTrader5 package: an external Python process connecting TO the terminal. The EA-side bridge is files (chosen) or a DLL; WebRequest is available for live-time pulls if ever needed.

Architecture

[scheduled task, e.g. hourly]
python -m altdata.cot / fred / eia / recorder      (research/altdata/*)
        │  writes observed,published,value CSVs
        ▼
c:\...\Workspaces\Market Data\altdata\{cot,fred,eia,...}\*.csv   (research root)
        │  python -m altdata.join  (as-of join, per-symbol feature matrix)
        ▼
Common\Files\Warrior_EA\AltData\{SYMBOL}_D1.csv    (EA-facing contract)
        │  EA: FileOpen(...,FILE_READ|FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_COMMON)
        ▼
feature window → NN input block (alt features appended like the meta descriptor)

EA contract (to implement): one CSV per symbol, rows bar_time;f1;..;fn, header row names the features, sidecar .meta carries feature count + last update stamp. Missing/stale file ⇒ features fed as "missing" flags, EA trades on (degraded, never blocked). Tester reads the same files — history is fully reproducible because every row was as-of joined.

Feature derivation (join.py, next)

Per symbol from COT (the literature-standard set, deliberately small):

• net speculative position / open interest (Lev_Money or M_Money net ÷ OI)
• 1y and 3y COT index (percentile of net position) — positioning EXTREMES
• 4-week change in net position
• dealer/commercial net (the other side)
• USDJPY/USDCAD: sign flipped at this step; sidecar documents it From FRED: level + 5-day change of VIX, T10Y2Y, DTWEXBGS, T5YIE; NFCI weekly. From EIA (XTIUSD only): stocks level vs 5y seasonal norm, 1w change.

First measurement (before any NN touches this): the existing MI/permutation harness over these features vs the excursion targets (RANGE first — the proven predictable channel — then the barrier label), family-wise gated, on D1. Cheap, uses only already-validated machinery, and tells us which families earn a slot in the feature window.

EA-side plan (staged, in order)

1. join.py + first MI measurement (offline only, no EA changes).
2. AltData reader module in the EA (mirrors MetaCorpus.mqh patterns; share flags per the tester FileCopy lesson).
3. Classic-signal removal refactor (separate branch; the sweep/corpus and signal-DB machinery is entangled with classic patterns — the candidate stream they provided has to be replaced or retired in the same commit).
4. Wyckoff tuner optimization + input-menu pruning.

Keys needed (all free registrations, user action)

Put in Market Data\altdata\keys.json as {"fred": "...", "eia": "..."}:

• FRED: https://fred.stlouisfed.org/docs/api/api_key.html
• EIA: https://www.eia.gov/opendata/register.php
• (later, optional) Finnhub, NASA FIRMS.