Same harness as the D1 screens, forward 30 H4 bars (~5 days), 199 perms, on
the surviving htf mid bars (19.8k-36.4k bars per symbol). Question: does the
wired alt/volume information carry to H4, for the chart-timeframe decision.
Answer: the signal survives but is roughly halved, and the cost side worsens
2.6x per step down. SP500 vix_chg5 clears the family bar with MI|vol 0.0112
(vs 0.031 at D1); volLevel50 (the EA activity feature) is incremental on all
four symbols at H4 - USDJPY family-clean, and it is EURUSD strongest
non-control signal there too. Gold gvz_chg5 stays incremental (0.0038 vs
0.0197 at D1 - a fifth of the strength). COT is null at H4 on FX (weekly
cadence pasted across 30 bars/week dilutes it below detection on EURUSD/JPY;
survives conditionally on SP500).
Cost table (median spread/ATR; the 1.74xATR geometry in spread units):
SP500 138->52->25, EURUSD 282->113->57, USDJPY 230->87->44, XAUUSD 92->34->16
for D1->H4->H1. Every step down multiplies the cost share ~2.6x.
Also fixes the disaggregated-COT column name for gold/WTI in the screen
(M_Money vs Lev_Money - the same crash the EA-side catalog documents).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POWER is the real find of the three: daily temperature -> degree days ->
natural-gas demand is the textbook gas fundamental, numeric and daily. But it
is point data needing construction into a national series (NOAA CPC ships that
ready-made), and its target symbol is not traded yet - fetch code written for
a chart nobody attaches first runs months later, unobserved, which is the
silent-FRED failure shape. Queued for the AvaTrade expansion, not refused.
FIRMS: re-raised, nothing changed since it was parked - point fire detections
behind the same unproven proxy chain. GIBS: imagery tiles, not numbers; our
CONV is 1D and NASA already sells the extracted products.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner decision (stated twice): available data gets wired; the networks judge
usefulness; the deploy gate remains the arbiter of what trades. Implemented:
MACRO block (6) on every symbol: 10y yield 20d change, curve slope, 5y
breakeven 20d change, Fed-ECB policy gap, CPI yoy, unemployment 12m change.
Screened null vs forward range on all four research symbols - recorded as
the honest prior in the catalog comment, wired regardless.
RISK block (3) extended to every symbol (FX majors, metals, energy, BTC all
now carry vix/vix_chg5/usd_chg5).
IVOL pair extended with the level alongside the change.
Vintage integrity kept where it is free: CPI is fetched as CPIAUCNS (NSA,
essentially never revised) so the plain-FRED backfill stays first-print-clean;
yields/curve/breakevens/policy rates are unrevised by nature. UNRATE is the
one exception (seasonal refits, ~0.1-0.2pp) - the EA cannot run the ALFRED
protocol, accepted and documented at the declaration site.
UpdateFred gains a staleDays parameter so the monthly series do not fire a
pointless fetch attempt every hour for three weeks after each print.
FeatureValue now takes the day and does its own as-of lookups - adding a
source no longer widens a parameter list. Feature counts: 12-15 per symbol;
symbol feature-order changed, safe only because no models exist yet.
export.py mirrors the new catalog for the five research symbols (13-15
features), smoke-tested: all five CSVs written, 6,072 daily rows each.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Screened yields (DGS2/DGS10), curve slope, inflation breakevens, Fed policy,
the Fed-ECB policy differential, and monthly US unemployment and CPI - all on
ALFRED first prints, 499 permutations, against forward 5-day range.
NOT ONE macro feature clears the family-wise bar on any symbol. The only thing
that clears anywhere is the trailing-range positive control, which is what it
is there to do. Best a-priori candidate, the Fed-ECB differential on EURUSD,
came in at MI 0.00170 p=0.088 - nothing. The two features flagged INCREMENTAL
(dgs2_chg5 on SP500) have null marginal MI and are isolated conditional cells
at the expected false-positive rate, not findings.
The `distinct` column quantifies the power argument instead of asserting it:
unemployment takes 51-66 distinct values across 3,745-6,159 bars, CPI 174-277,
against 6,159 for a continuous feature. A monthly series pasted onto daily bars
carries about 1% of the resolution, and it showed - the monthly features were
among the weakest in every table.
The contrast with the implied-vol screen is the useful part: the options
market FORWARD-LOOKING view of an instrument (gvz_chg5 on gold, MI|vol 0.0197)
carries real information about its range, while the economy BACKWARD-LOOKING
state carries none. Mismatched timescales - rate levels move over months,
5-day range moves daily.
Also makes load_bars fall back to htf/{SYM}_D1_mid.npz when the tick-derived
build is absent (the 2026-08-16 disk cleanup removed bars/ but htf/ survived),
with need_ticks=True turning that fallback into a loud failure for the
order-flow screen rather than silently testing flow features on OHLC data.
No EA change: nothing survived to wire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the screen_ivol survivors (41d726c). New per-symbol `ivolSeries` in the
catalog feeds a generic `ivol_chg5` feature from whichever CBOE vol index the
instrument owns, so one code path serves every symbol:
XAUUSD + ivol_chg5 (GVZ) - MI|vol 0.01971 p=0.002, 3.6x the positive
control and 4.6x the vix_chg5 gold had alone.
vix_chg5 KEPT: this appends, it does not replace.
EURUSD + vix_chg5 - screened, incremental p<=0.006, and its first
real feature ever (it had only exploratory EIA).
USDJPY + vix_chg5 - screened, incremental.
NAS100 / US30 / US2000 + ivol_chg5 (VXN / VXD / RVX) - exploratory by analogy.
XTIUSD / XBRUSD + ivol_chg5 (OVX) - exploratory, no oil bars to screen yet.
SP500 unchanged - its features already screened clean and VXN/VIX3M edging
out VIX is a correlated within-family best-of-N, not a real ranking.
On EURUSD/USDJPY the screen put VXD marginally above VIX, but they are
near-duplicates and the gap sits inside the noise, so the tie is broken by a
rule rather than by the number: take the series already in the fetch path.
Also fixes a real collector bug: fetch_vintaged built ALFRED realtime windows
out to 2028, and FRED rejects realtime_end after today - so every REVISED
series (unemployment, CPI, GDP: exactly the ones needing the vintage path) was
unreachable, while unrevised series never noticed because they bail earlier.
UNRATE and CPIAUCSL now return first prints correctly.
Adds screen_macro.py (rates, curve, breakevens, Fed/ECB policy differential,
plus monthly country stats) with a `distinct` column that reports the honest
effective sample size - a monthly series pasted onto D1 bars is a step
function, and that column is what decides whether it can clear a gate at all.
Not yet run: the Market Data bars directory is being regenerated right now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No free historical GEX exists: probed the CBOE chain endpoint with date/dt query
params (both silently ignored, returned today) and dated/historical paths (403),
and the CBOE index-history CSVs are 403 too. The forward recorder stays the only
path to GEX history.
But the options market publishes its per-instrument view of future range as the
CBOE vol indices, and FRED carries the whole family free with 15-25 years of
history - screenable today with the existing collector and harness. Fetched
GVZ (gold), OVX (oil), VXN, VXD, RVX, VIX3M.
HEADLINE - XAUUSD: gvz_chg5 (gold IV 5-day change) MI 0.02103, MI|vol 0.01971,
p=0.002. That is 3.6x the trailing-range positive control and 4.6x the vix_chg5
this project currently ships on gold - the second-largest incremental MI of the
whole campaign, on a symbol that carries exactly one screened feature today.
Vol-change is incremental on all four symbols: SP500 (known), USDJPY vxd_chg5
0.00492, and EURUSD vxd_chg5 0.00412 / vix_chg5 0.00379 - notable because
EURUSD has no screened features at all and its own trailing range is a weak
control there, so external vol carries information its own history does not.
Caveats recorded in the script and memory: SP500 within-family ordering
(VXN > VIX3M > VIX, all ~0.031-0.038 conditional) is a best-of-N artifact and
must not be cherry-picked; XAUUSD noise control misbehaved this run (MI|vol
0.00271 p=0.002), so anything under ~0.003 conditional on gold is unresolved -
gvz_chg5 at 7x that floor is unaffected; EVZ (euro IV) is DISCONTINUED since
2025-03 and must never be wired.
Nothing wired - the EA is mid-deploy and this would re-key every model again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spent 3 of 5 daily requests. All three were informative:
ETF data (SPY/QQQ/IWM) requires Basic - free tier is single stocks only.
Full-chain GEX (all expirations) requires Growth - free and Basic must
query one expiration per request, so even with history a full-chain
backfill would be 24-54 requests per day of history.
AAPL?expiration=2026-09-18 returned 200 with the right schema but a nearly
empty payload: 13 of 93 strikes carried any open interest, total call OI
4,296 against CBOE 373,253 for the same expiry, put OI zero, and every
near-the-money strike blank.
So the construction could not be validated - not because the math disagreed
but because there was nothing to compare against. From outside it is not
possible to tell free-tier degradation from their flow-signed methodology,
and finding out costs $1,499/month.
Verdict hardens: the free CBOE CDN is strictly better than Basic for this
project - complete chains, every expiry and strike, gamma and open interest
populated, unlimited, $0. Our own AAPL figures were internally coherent
(+0.929 Bn/1% total, Sep-18 expiry +0.154 Bn, near-money gammas 0.013-0.019).
GEX stays externally unvalidated; if that ever matters, use a different vendor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pricing checked: Free $0 (5/day), Basic $79 (250/day), Growth $299 (2,500/day),
Alpha $1,499 (unlimited) - and the Historical API is ALPHA-EXCLUSIVE. Basic and
Growth serve live data only.
The archive was the only thing worth buying from this vendor, so nothing in
budget helps: Basic would spend $79/month to make a once-a-day snapshot 15
seconds fresh instead of 15 minutes. Not subscribing.
The free key keeps one genuine use: a single live call to compare their GEX
against our CBOE-computed number, validating the recorder formula against a
commercial implementation (sign and magnitude only - they sign strikes from
classified tape, we use the standard open-interest assumption).
Recorded the EV argument for future sessions: the recorder banks this history
for free in ~12 months, and on this project base rate most alt-data families
die at the incremental gate. Paying four figures to test GEX a year early is a
poor trade. If revisited, price bulk ARCHIVE sellers, not analytics APIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Option open interest is a snapshot source - no free history exists anywhere -
so the series only accrues from the day recording starts. That is why this
ships BEFORE the redeploy: every day the EA is not running is a day of history
that cannot be recovered later.
Records one row per weekday after 21:00 UTC to gex_{CANONICAL}.csv: net/call/put
dollar GEX per 1% move, call and put OI, the three nearest expiries and the
front expiry code. Feeds NOTHING - wiring a feature that is missing across ~100%
of the training sample would waste input width and hand batch-norm a constant.
It becomes a screening candidate at ~250 rows, gated like every other feature.
Thesis: dealer gamma is a RANGE mechanism (long gamma -> hedging sells rallies
and buys dips, range compresses; short gamma amplifies both ways), and range is
this project's one proven channel.
Verified in situ against the live SPX chain before writing any MQL5: 29,362
contracts, 20,993 with nonzero gamma, 54 expiries, total +90.7 Bn/1% (calls
+305.7, puts -215.0), and 100% of net GEX inside 5% of spot. The CDN publishes
per-contract gamma directly, so no pricing model - and no model risk - enters
the recorded data. Also verified the CDN does NOT gate on User-Agent (the old
"CBOE is UA-gated" note in DESIGN.md was a different CBOE path), so plain
WebRequest reaches it.
Dropped a zero-gamma "flip level" field: the probe returned a crossing above
spot while total GEX was strongly positive, which is incoherent - a static
gamma snapshot cannot give a flip level without repricing. Recording a
plausible-looking wrong number is worse than recording nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VPIN-style toxicity (|imbalance|): null-to-marginal everywhere. But tick
ACTIVITY (count vs 20d mean) clears the family bar on ALL FOUR symbols for
forward range AND survives conditioning on trailing realized range
(SP500 MI|vol 0.025, XAUUSD 0.0088, USDJPY 0.0069, EURUSD 0.0049, all
p<=0.006). On EURUSD it beats the trailing-range positive control itself -
resolving the void-control anomaly: EURUSD D1 range IS predictable, just
not by its own trailing range. USDJPY spread_stress (max/mean) also
family-clean + incremental. Direction: nothing beyond the known SP500
leverage effect. Validates the EA's volume feature block for the RANGE
objective the tuner now optimizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
12 features (5 EIA petroleum + 3 COT managed-money + VIX/USD + controls)
vs forward 5-bar range and direction on 1,789 D1 bars resampled from the
decoded XTIUSD M1 file, 499 circular-shift perms. No feature clears the
family-wise bar on either target; every EIA fundamental is null even
marginally (best p=0.13). Sample is short (~7y) so a weak effect is not
excluded - but per the gate, no EIA feature ships. The EIA key stays in
keys.txt for future use (longer history / recorded surprise-vs-consensus).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MI|vol column = I(X; target | trailing-range tercile), same circular-shift
null. Range target 499 perms: SP500 vix_chg5 survives conditioning at 0.031
(3x trailing range's own within-tercile residual); VIX LEVEL emerges
conditionally (variance-risk-premium structure); USDJPY COT family survives.
Direction target: SP500 vol/VIX-chg clear marginally (equity leverage
effect) but drop to p~0.05-0.06 conditional = redundant with price vol;
USDJPY/XAUUSD/EURUSD direction null across all alt features.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- fred.py: ALFRED output_type=4 first prints, chunked realtime windows
(2000-vintage cap), unrevised-series fallback (published=observed+1d);
NFCI excluded from features (revised, no vintage archive)
- eia.py: 4 weekly petroleum series on disk (1982->now)
- screen.py: as-of joined alt features vs forward 5-bar range/ATR on D1,
3x3 MI, circular-shift null, family-wise max bar, +/- controls
First readings (199 perms): SP500 vix_chg5 MI 0.047 (1.5x the positive
control) + usd_chg5 clear family bar; USDJPY 4 COT positioning features
clear family bar BEATING the positive control; XAUUSD vix_chg5 tops control
but sub-family-bar; EURUSD positive control FAILS -> table void per the
excursion-target rule, needs investigation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
