feat(altdata): EIA wired, 24-instrument symbol catalog, mapping dialog for unknown symbols
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 16:18:29 -04:00
|
|
|
WARRIOR EA - URLs TO WHITELIST FOR ALTERNATIVE DATA
|
|
|
|
|
===================================================
|
|
|
|
|
|
|
|
|
|
The EA downloads its own alternative-data history (COT positioning, VIX / dollar
|
|
|
|
|
index, EIA petroleum) and keeps it updated while it runs. MetaTrader blocks all
|
|
|
|
|
outbound HTTP until you list the hosts explicitly.
|
|
|
|
|
|
|
|
|
|
WHERE
|
|
|
|
|
-----
|
|
|
|
|
MT5 menu: Tools > Options > Expert Advisors
|
|
|
|
|
Tick: "Allow WebRequest for listed URL"
|
|
|
|
|
Then add each line below as its OWN entry.
|
|
|
|
|
|
|
|
|
|
THE URLS (copy-paste one per line)
|
|
|
|
|
----------------------------------
|
|
|
|
|
https://publicreporting.cftc.gov
|
|
|
|
|
https://api.stlouisfed.org
|
|
|
|
|
https://api.eia.gov
|
feat(altdata): GEX forward recorder (CBOE delayed-quotes CDN, no key)
Option open interest is a snapshot source - no free history exists anywhere -
so the series only accrues from the day recording starts. That is why this
ships BEFORE the redeploy: every day the EA is not running is a day of history
that cannot be recovered later.
Records one row per weekday after 21:00 UTC to gex_{CANONICAL}.csv: net/call/put
dollar GEX per 1% move, call and put OI, the three nearest expiries and the
front expiry code. Feeds NOTHING - wiring a feature that is missing across ~100%
of the training sample would waste input width and hand batch-norm a constant.
It becomes a screening candidate at ~250 rows, gated like every other feature.
Thesis: dealer gamma is a RANGE mechanism (long gamma -> hedging sells rallies
and buys dips, range compresses; short gamma amplifies both ways), and range is
this project's one proven channel.
Verified in situ against the live SPX chain before writing any MQL5: 29,362
contracts, 20,993 with nonzero gamma, 54 expiries, total +90.7 Bn/1% (calls
+305.7, puts -215.0), and 100% of net GEX inside 5% of spot. The CDN publishes
per-contract gamma directly, so no pricing model - and no model risk - enters
the recorded data. Also verified the CDN does NOT gate on User-Agent (the old
"CBOE is UA-gated" note in DESIGN.md was a different CBOE path), so plain
WebRequest reaches it.
Dropped a zero-gamma "flip level" field: the probe returned a crossing above
spot while total GEX was strongly positive, which is incoherent - a static
gamma snapshot cannot give a flip level without repricing. Recording a
plausible-looking wrong number is worse than recording nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 16:30:39 -04:00
|
|
|
https://cdn.cboe.com
|
feat(altdata): EIA wired, 24-instrument symbol catalog, mapping dialog for unknown symbols
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 16:18:29 -04:00
|
|
|
|
|
|
|
|
WHAT EACH ONE IS FOR
|
|
|
|
|
--------------------
|
|
|
|
|
https://publicreporting.cftc.gov
|
|
|
|
|
CFTC Commitments of Traders (weekly positioning). No API key needed.
|
|
|
|
|
Feeds: cot_spec_net, cot_idx_1y, cot_idx_3y, cot_chg_4w.
|
|
|
|
|
|
|
|
|
|
https://api.stlouisfed.org
|
|
|
|
|
FRED (Federal Reserve Bank of St. Louis). Needs the FredApiKey input.
|
|
|
|
|
Feeds: vix, vix_chg5 (VIXCLS) and usd_chg5 (DTWEXBGS broad dollar index).
|
|
|
|
|
|
|
|
|
|
https://api.eia.gov
|
|
|
|
|
U.S. Energy Information Administration, Weekly Petroleum Status Report.
|
|
|
|
|
Needs the EiaApiKey input.
|
|
|
|
|
Feeds: eia_stk_idx1y, eia_stk_chg4, eia_util.
|
|
|
|
|
|
feat(altdata): GEX forward recorder (CBOE delayed-quotes CDN, no key)
Option open interest is a snapshot source - no free history exists anywhere -
so the series only accrues from the day recording starts. That is why this
ships BEFORE the redeploy: every day the EA is not running is a day of history
that cannot be recovered later.
Records one row per weekday after 21:00 UTC to gex_{CANONICAL}.csv: net/call/put
dollar GEX per 1% move, call and put OI, the three nearest expiries and the
front expiry code. Feeds NOTHING - wiring a feature that is missing across ~100%
of the training sample would waste input width and hand batch-norm a constant.
It becomes a screening candidate at ~250 rows, gated like every other feature.
Thesis: dealer gamma is a RANGE mechanism (long gamma -> hedging sells rallies
and buys dips, range compresses; short gamma amplifies both ways), and range is
this project's one proven channel.
Verified in situ against the live SPX chain before writing any MQL5: 29,362
contracts, 20,993 with nonzero gamma, 54 expiries, total +90.7 Bn/1% (calls
+305.7, puts -215.0), and 100% of net GEX inside 5% of spot. The CDN publishes
per-contract gamma directly, so no pricing model - and no model risk - enters
the recorded data. Also verified the CDN does NOT gate on User-Agent (the old
"CBOE is UA-gated" note in DESIGN.md was a different CBOE path), so plain
WebRequest reaches it.
Dropped a zero-gamma "flip level" field: the probe returned a crossing above
spot while total GEX was strongly positive, which is incoherent - a static
gamma snapshot cannot give a flip level without repricing. Recording a
plausible-looking wrong number is worse than recording nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 16:30:39 -04:00
|
|
|
https://cdn.cboe.com
|
|
|
|
|
CBOE delayed option quotes - open interest and gamma per contract.
|
|
|
|
|
No API key needed.
|
|
|
|
|
Feeds: NOTHING YET. This is a forward RECORDER: it appends one row a day
|
|
|
|
|
to gex_{SYMBOL}.csv because no free history of option open interest
|
|
|
|
|
exists anywhere. It becomes a feature only once the file is long enough
|
|
|
|
|
to screen (roughly a year). Every day the EA is not running is a day of
|
|
|
|
|
that history you cannot get back later.
|
|
|
|
|
Recorded once per weekday after 21:00 UTC. The SPX chain is ~13 MB, so
|
|
|
|
|
this is the one large download the EA makes - once a day.
|
|
|
|
|
|
feat(altdata): EIA wired, 24-instrument symbol catalog, mapping dialog for unknown symbols
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 16:18:29 -04:00
|
|
|
NOTES
|
|
|
|
|
-----
|
|
|
|
|
* Takes effect IMMEDIATELY. A running EA picks the whitelist up on its next
|
|
|
|
|
hourly retry - you do not need to remove and re-attach it.
|
|
|
|
|
* If a URL is missing the EA prints error 4014, raises ONE alert, and writes
|
|
|
|
|
these same copy-paste lines into the Experts log, then retries hourly.
|
|
|
|
|
* Until then the EA runs normally on whatever it has already cached - features
|
|
|
|
|
stay valid but stop updating past their last downloaded date.
|
|
|
|
|
* Enter the HOST only, exactly as written above. Do not add paths, query
|
|
|
|
|
strings or trailing slashes.
|
|
|
|
|
* The Strategy Tester cannot make web requests at all (platform rule). Tester
|
|
|
|
|
runs read the cached files, which is what makes them reproducible.
|
|
|
|
|
* Keys live in the FredApiKey / EiaApiKey inputs - see docs/API_KEYS.txt.
|