Screened yields (DGS2/DGS10), curve slope, inflation breakevens, Fed policy,
the Fed-ECB policy differential, and monthly US unemployment and CPI - all on
ALFRED first prints, 499 permutations, against forward 5-day range.
NOT ONE macro feature clears the family-wise bar on any symbol. The only thing
that clears anywhere is the trailing-range positive control, which is what it
is there to do. Best a-priori candidate, the Fed-ECB differential on EURUSD,
came in at MI 0.00170 p=0.088 - nothing. The two features flagged INCREMENTAL
(dgs2_chg5 on SP500) have null marginal MI and are isolated conditional cells
at the expected false-positive rate, not findings.
The `distinct` column quantifies the power argument instead of asserting it:
unemployment takes 51-66 distinct values across 3,745-6,159 bars, CPI 174-277,
against 6,159 for a continuous feature. A monthly series pasted onto daily bars
carries about 1% of the resolution, and it showed - the monthly features were
among the weakest in every table.
The contrast with the implied-vol screen is the useful part: the options
market FORWARD-LOOKING view of an instrument (gvz_chg5 on gold, MI|vol 0.0197)
carries real information about its range, while the economy BACKWARD-LOOKING
state carries none. Mismatched timescales - rate levels move over months,
5-day range moves daily.
Also makes load_bars fall back to htf/{SYM}_D1_mid.npz when the tick-derived
build is absent (the 2026-08-16 disk cleanup removed bars/ but htf/ survived),
with need_ticks=True turning that fallback into a loud failure for the
order-flow screen rather than silently testing flow features on OHLC data.
No EA change: nothing survived to wire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the screen_ivol survivors (41d726c). New per-symbol `ivolSeries` in the
catalog feeds a generic `ivol_chg5` feature from whichever CBOE vol index the
instrument owns, so one code path serves every symbol:
XAUUSD + ivol_chg5 (GVZ) - MI|vol 0.01971 p=0.002, 3.6x the positive
control and 4.6x the vix_chg5 gold had alone.
vix_chg5 KEPT: this appends, it does not replace.
EURUSD + vix_chg5 - screened, incremental p<=0.006, and its first
real feature ever (it had only exploratory EIA).
USDJPY + vix_chg5 - screened, incremental.
NAS100 / US30 / US2000 + ivol_chg5 (VXN / VXD / RVX) - exploratory by analogy.
XTIUSD / XBRUSD + ivol_chg5 (OVX) - exploratory, no oil bars to screen yet.
SP500 unchanged - its features already screened clean and VXN/VIX3M edging
out VIX is a correlated within-family best-of-N, not a real ranking.
On EURUSD/USDJPY the screen put VXD marginally above VIX, but they are
near-duplicates and the gap sits inside the noise, so the tie is broken by a
rule rather than by the number: take the series already in the fetch path.
Also fixes a real collector bug: fetch_vintaged built ALFRED realtime windows
out to 2028, and FRED rejects realtime_end after today - so every REVISED
series (unemployment, CPI, GDP: exactly the ones needing the vintage path) was
unreachable, while unrevised series never noticed because they bail earlier.
UNRATE and CPIAUCSL now return first prints correctly.
Adds screen_macro.py (rates, curve, breakevens, Fed/ECB policy differential,
plus monthly country stats) with a `distinct` column that reports the honest
effective sample size - a monthly series pasted onto D1 bars is a step
function, and that column is what decides whether it can clear a gate at all.
Not yet run: the Market Data bars directory is being regenerated right now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No free historical GEX exists: probed the CBOE chain endpoint with date/dt query
params (both silently ignored, returned today) and dated/historical paths (403),
and the CBOE index-history CSVs are 403 too. The forward recorder stays the only
path to GEX history.
But the options market publishes its per-instrument view of future range as the
CBOE vol indices, and FRED carries the whole family free with 15-25 years of
history - screenable today with the existing collector and harness. Fetched
GVZ (gold), OVX (oil), VXN, VXD, RVX, VIX3M.
HEADLINE - XAUUSD: gvz_chg5 (gold IV 5-day change) MI 0.02103, MI|vol 0.01971,
p=0.002. That is 3.6x the trailing-range positive control and 4.6x the vix_chg5
this project currently ships on gold - the second-largest incremental MI of the
whole campaign, on a symbol that carries exactly one screened feature today.
Vol-change is incremental on all four symbols: SP500 (known), USDJPY vxd_chg5
0.00492, and EURUSD vxd_chg5 0.00412 / vix_chg5 0.00379 - notable because
EURUSD has no screened features at all and its own trailing range is a weak
control there, so external vol carries information its own history does not.
Caveats recorded in the script and memory: SP500 within-family ordering
(VXN > VIX3M > VIX, all ~0.031-0.038 conditional) is a best-of-N artifact and
must not be cherry-picked; XAUUSD noise control misbehaved this run (MI|vol
0.00271 p=0.002), so anything under ~0.003 conditional on gold is unresolved -
gvz_chg5 at 7x that floor is unaffected; EVZ (euro IV) is DISCONTINUED since
2025-03 and must never be wired.
Nothing wired - the EA is mid-deploy and this would re-key every model again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spent 3 of 5 daily requests. All three were informative:
ETF data (SPY/QQQ/IWM) requires Basic - free tier is single stocks only.
Full-chain GEX (all expirations) requires Growth - free and Basic must
query one expiration per request, so even with history a full-chain
backfill would be 24-54 requests per day of history.
AAPL?expiration=2026-09-18 returned 200 with the right schema but a nearly
empty payload: 13 of 93 strikes carried any open interest, total call OI
4,296 against CBOE 373,253 for the same expiry, put OI zero, and every
near-the-money strike blank.
So the construction could not be validated - not because the math disagreed
but because there was nothing to compare against. From outside it is not
possible to tell free-tier degradation from their flow-signed methodology,
and finding out costs $1,499/month.
Verdict hardens: the free CBOE CDN is strictly better than Basic for this
project - complete chains, every expiry and strike, gamma and open interest
populated, unlimited, $0. Our own AAPL figures were internally coherent
(+0.929 Bn/1% total, Sep-18 expiry +0.154 Bn, near-money gammas 0.013-0.019).
GEX stays externally unvalidated; if that ever matters, use a different vendor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pricing checked: Free $0 (5/day), Basic $79 (250/day), Growth $299 (2,500/day),
Alpha $1,499 (unlimited) - and the Historical API is ALPHA-EXCLUSIVE. Basic and
Growth serve live data only.
The archive was the only thing worth buying from this vendor, so nothing in
budget helps: Basic would spend $79/month to make a once-a-day snapshot 15
seconds fresh instead of 15 minutes. Not subscribing.
The free key keeps one genuine use: a single live call to compare their GEX
against our CBOE-computed number, validating the recorder formula against a
commercial implementation (sign and magnitude only - they sign strikes from
classified tape, we use the standard open-interest assumption).
Recorded the EV argument for future sessions: the recorder banks this history
for free in ~12 months, and on this project base rate most alt-data families
die at the incremental gate. Paying four figures to test GEX a year early is a
poor trade. If revisited, price bulk ARCHIVE sellers, not analytics APIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Free tier is 5 requests per DAY, so the quota is reserved rather than spent:
one request answers whether a historical timestamp returns the whole chain or
one expiration, and that decides whether backfilling 2018->now is ~2,100
requests (GEX screenable now) or infeasible (compare bulk vendors instead).
Deliberately NOT an EA input, unlike FRED/EIA: the EA must never depend on a
paid, rate-limited vendor in its live path. Research-side backfill only, so the
key lives in the gitignored Market Data\altdata\keys.json plus this backup.
Auth is an X-Api-Key HEADER, not a query parameter - worth noting because the
EA's HttpGet currently sends no headers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Option open interest is a snapshot source - no free history exists anywhere -
so the series only accrues from the day recording starts. That is why this
ships BEFORE the redeploy: every day the EA is not running is a day of history
that cannot be recovered later.
Records one row per weekday after 21:00 UTC to gex_{CANONICAL}.csv: net/call/put
dollar GEX per 1% move, call and put OI, the three nearest expiries and the
front expiry code. Feeds NOTHING - wiring a feature that is missing across ~100%
of the training sample would waste input width and hand batch-norm a constant.
It becomes a screening candidate at ~250 rows, gated like every other feature.
Thesis: dealer gamma is a RANGE mechanism (long gamma -> hedging sells rallies
and buys dips, range compresses; short gamma amplifies both ways), and range is
this project's one proven channel.
Verified in situ against the live SPX chain before writing any MQL5: 29,362
contracts, 20,993 with nonzero gamma, 54 expiries, total +90.7 Bn/1% (calls
+305.7, puts -215.0), and 100% of net GEX inside 5% of spot. The CDN publishes
per-contract gamma directly, so no pricing model - and no model risk - enters
the recorded data. Also verified the CDN does NOT gate on User-Agent (the old
"CBOE is UA-gated" note in DESIGN.md was a different CBOE path), so plain
WebRequest reaches it.
Dropped a zero-gamma "flip level" field: the probe returned a crossing above
spot while total GEX was strongly positive, which is incoherent - a static
gamma snapshot cannot give a flip level without repricing. Recording a
plausible-looking wrong number is worse than recording nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The AltData folder in Common\Files gets wiped before every fresh test, and
keys.txt died with it (2026-08-16 silent-FRED incident). The credential now
travels with the EA: FredApiKey input, owner key as default; keys.txt demoted
to a fallback consulted only when the input is blanked. EiaApiKey stored the
same way - reserved, nothing consumes it since the WTI screen came back null.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-08-16 first live fetch looked complete but was not: COT (keyless)
downloaded 1053 reports, then FredKey() hit the absent keys.txt, returned ""
with no log line, UpdateFred bailed, and the SP500_D1.csv rebuild - gated on
all three raw series - never happened. The panel stayed at 0 features with
nothing in the journal explaining why.
FredKey() now logs loudly when keys.txt is missing, and only latches once a
key is actually FOUND: the file is re-read on each hourly-throttled attempt,
so dropping keys.txt in after attach recovers without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VPIN-style toxicity (|imbalance|): null-to-marginal everywhere. But tick
ACTIVITY (count vs 20d mean) clears the family bar on ALL FOUR symbols for
forward range AND survives conditioning on trailing realized range
(SP500 MI|vol 0.025, XAUUSD 0.0088, USDJPY 0.0069, EURUSD 0.0049, all
p<=0.006). On EURUSD it beats the trailing-range positive control itself -
resolving the void-control anomaly: EURUSD D1 range IS predictable, just
not by its own trailing range. USDJPY spread_stress (max/mean) also
family-clean + incremental. Direction: nothing beyond the known SP500
leverage effect. Validates the EA's volume feature block for the RANGE
objective the tuner now optimizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- PeriodMA/MA_Type/PeriodRSI: input -> const seeds (closing the set: every
indicator parameter is now tuner-owned)
- Variables\TunedPeriods.mqh: chart-level tuned-period state. A gated
install writes TunedPeriods_{SYM}_{TF}.cfg; next attach reads it BEFORE
the DB fingerprint and classic-signal config, so classic votes, DB key,
and tuner seeds always describe the same indicators regardless of
classic/AI/hybrid use. Restart-grained adoption by design (no mid-run
handle churn); new periods re-key the signal DB (semantics rule).
- EnableAltData input in AI Input Features (consumption gate only;
collection keeps running); |ALT DB-fingerprint token; opt-out on an
alt-trained model correctly starts fresh via the width compare.
- Defaults: all four classic votes OFF (AI-first; WARRIOR_MARKET_BUILD
branches collapsed with the marketplace pivot), order-flow/Wyckoff NN
features OFF (alt data is the default information diet; toggles stay).
Compiles 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 18 inputs added 2026-08-08 (when the tuner defaulted off and the
values needed an operator path) become compile-time aliases of their own
defaults - same names, zero consumer churn, byte-identical values. The
tuner is now the only path by which these values move: it defaults ON
(the 08-08 off-flip was measured against the direction target's flat
landscape; the objective is now RANGE, which has signal), searches from
the seeds under the Sidak family-wise gate, and persists winners in the
.nnw beside the weights. ADP fingerprint token retired (deviation now
impossible by construction; tuned values were never its job).
Menu shrinks 102 -> 84 inputs. Compiles 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- MI_TUNE_TARGET = MI_TARGET_EXC_RANGE: the coordinate sweep scored
candidates against the barrier label - measured noise - so it climbed a
flat landscape and the gate rightly rejected every winner. It now
selects indicator settings for MI vs realised RANGE (4x null, positive
control), the channel the excursion head consumes these features for.
Winner gate re-tests on the same target. Barrier-label report unchanged.
- AltDataFetch 4014 handling: Alert popup + once-per-session walkthrough
with the two whitelist URLs on their own journal lines (copy-paste
ready); hourly-backoff retry instead of a permanent latch, so the
whitelist fix takes effect without re-attaching.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
12 features (5 EIA petroleum + 3 COT managed-money + VIX/USD + controls)
vs forward 5-bar range and direction on 1,789 D1 bars resampled from the
decoded XTIUSD M1 file, 499 circular-shift perms. No feature clears the
family-wise bar on either target; every EIA fundamental is null even
marginally (best p=0.13). Sample is short (~7y) so a weak effect is not
excluded - but per the gate, no EIA feature ships. The EIA key stays in
keys.txt for future use (longer history / recorded surprise-vs-consensus).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
System\AltDataFetch.mqh: the EA backfills missing alt-data history at
attach and keeps appending forward while deployed - online learning never
depends on an external process. CFTC Socrata API (no key, 2006->now, one
GET per symbol; ES name variants verified, max-OI dedupe) + FRED (VIXCLS/
DTWEXBGS, key from AltData\keys.txt). Identical publication stamps and
fixed a-priori transforms as research/altdata/export.py; rebuilds the
same {SYM}_D1.csv files, so Python and EA interoperate on one format.
OnTimer hook (30-min staleness check, in-memory compares when current;
never in tester - cache files serve there) + AltDataReload() on signals.
Classic-signal removal CANCELLED per user (vote experiment later).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MI|vol column = I(X; target | trailing-range tercile), same circular-shift
null. Range target 499 perms: SP500 vix_chg5 survives conditioning at 0.031
(3x trailing range's own within-tercile residual); VIX LEVEL emerges
conditionally (variance-risk-premium structure); USDJPY COT family survives.
Direction target: SP500 vol/VIX-chg clear marginally (equity leverage
effect) but drop to p~0.05-0.06 conditional = redundant with price vol;
USDJPY/XAUUSD/EURUSD direction null across all alt features.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- fred.py: ALFRED output_type=4 first prints, chunked realtime windows
(2000-vintage cap), unrevised-series fallback (published=observed+1d);
NFCI excluded from features (revised, no vintage archive)
- eia.py: 4 weekly petroleum series on disk (1982->now)
- screen.py: as-of joined alt features vs forward 5-bar range/ATR on D1,
3x3 MI, circular-shift null, family-wise max bar, +/- controls
First readings (199 perms): SP500 vix_chg5 MI 0.047 (1.5x the positive
control) + usd_chg5 clear family bar; USDJPY 4 COT positioning features
clear family bar BEATING the positive control; XAUUSD vix_chg5 tops control
but sub-family-bar; EURUSD positive control FAILS -> table void per the
excursion-target rule, needs investigation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The derived stop/target were quantiles of EVERY bar''s excursions over a
fixed horizon - q75 adverse gave a 2.6-3.5*ATR stop against a ~1.7*ATR
target (user: "looks limiting"). That pooled measurement was correct
when direction was dead (any subset of bars had the same distribution)
and is provably mis-sized now that the gate certifies the label carries
information: the bars the model trades are the labeled bars, and their
excursions differ from the pool.
FractalDirectionLabel now records, for every Buy/Sell-labeled IS bar
during the prebuild, the favourable and adverse travel in ATR units
over exactly the LEG the label points at - entry close through the next
fractal extreme (user request: "from a fractal to the next for maximum
accuracy"). DeriveBarrierGeometry reads the same q75-adverse/q50-
favourable quantiles off that conditional sample instead of the pool,
with a logged fallback to pooled when fewer than the minimum legs
exist. Quantiles kept over averages deliberately: a mean MFE is
dominated by runaway legs and would set an unreachable target.
No circularity: the fractal label does not depend on SL/TP (the barrier
label does - this path must never feed it). Recording stops the moment
geometry is derived and pinned, so pass 2 relabels and later bars
cannot silently re-shape a certified pair.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two user-reported ensemble regressions, one cause each:
- "getting ready is very long": every member ran the full MI diagnostic
suite (headline MI, positive control, alignment, lag profile,
geometry scan + winner test - ~200 permuted draws per line) on
IDENTICAL features and labels, reporting the same numbers four times.
First member runs it, the rest adopt with one log line. Documented
caveat: if the geometry scan ever ADOPTS a winner under its gate
(it never has), the adoption becomes donor-only and the gate must be
revisited.
- "panel not responsive": four members chunks queue back-to-back on the
one chart thread - 4 x 120ms = 480ms worst-case click latency, the
exact regime the 200ms note in Training.mqh already documents as
broken. Ensemble members now use a 30ms chunk budget, restoring solo
UI latency at slightly higher dispatch overhead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All four ensemble members previously wrote their full multi-line panels
to the SAME global label objects - an ensemble chart would flicker
between four stacked panels covering the chart side (user request:
aggregate). Every AI-side SetStatusLabel call site now routes through
CExpertSignalAIBase::PublishStatus - solo charts draw the full panel
exactly as before; an ensemble member claims a slot and contributes
only its HEADLINE to one combined block ("HYBRID ensemble - N models",
then one line per model; the live line leads with the model current
signal). The combined render skips unchanged text and enforces its own
minimum redraw interval so four publishers cannot multiply
ChartRedraw() cost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user is right that no special combination logic is needed: the AI
signals are ordinary voting filters, and the aggregate already has
union semantics - abstaining filters do not dilute the average, so an
ensemble chart trades whenever ANY deployed member clears the vote
threshold and disagreeing members net out. What the ensemble preset
actually adds:
- AI_CHOICE value 4 renamed AI_CONVLSTM (the name says the front-end);
enum VALUES stable, CSignalHYBRID class and State\HYBRID\ folder kept,
so saved configs and trained models keep their identity.
- New AI_HYBRID = 6: enables PAI+CONV+LSTM+CONVLSTM together on one
chart - replaces four separate charts of the same symbol. Each member
trains and self-gates independently; only certified members ever vote.
- |ENS1 fingerprint token on every member, so an ensemble member's
weight files can never collide with a solo model of identical
settings on another chart of the same symbol (the duplicate-chart
guard would otherwise correctly fight over one .nnw).
- Private default AIType = AI_HYBRID: one D1 drop now yields every
topology's gate verdict for that symbol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first-ever family-wise gate pass (SP500 D1 PAI, +10.4pp, p=0.0081)
certifies a win rate measured on HOLD-TO-RESOLUTION outcomes: entry,
then the measured SL or TP decides. Live, three vote-driven exit routes
could close earlier - the averaged-vote close, the AI early-exit route
(both in CheckClosePosition), and CheckReverse - and the fractal
target's vote flips at swing-marker cadence (~3-5 bars), far inside the
barrier's typical travel time (median 7-8 D1 bars to target). The user
observed exactly this: an opposite arrow near an entry, trade cut,
price kept going.
On a fractal-target chart with a live direction model, all three routes
are now suppressed (m_holdToBarrier, set in InitializeSignal, loudly
logged): positions run to their broker SL/TP. Risk guards and trailing
are deliberately untouched - account protection is not signal opinion.
Barrier-target models keep the vote exits: their label is the vote's
own horizon, so for them the routes are semantically consistent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15, ~6 minutes after attach: PAI-8fea (fractal target, 37
features, D1) converged at era 299 and the plateau deploy CLEARED the
family-wise gate for the first time in project history: dir-precision
73.1% vs 63% break-even, +10.4pp on 350 test calls = 4.04 sigma,
p_family = 0.0081.
This script asks the first two hostile questions offline:
- DRIFT: always-long at the same 2.64/1.66 geometry scores 64.1% on the
last 15% of D1 history (66.7% on 30%) - drift alone clears BE by
~1-3pp, but the model is +9pp above ALWAYS-LONG, so the pass is
selection, not drift.
- SWAP EXPOSURE: median 7-8 bars to the long target = ~10 nights of
financing ~ 0.15-0.2% notional vs a ~1.7% target -> a ~1-1.5pp BE
haircut against a +10.4pp margin. Survives.
Remaining before belief: replication on other D1 symbols, and closing
the live-semantics gap (certified wins assume hold-to-barrier; live
exit paths can cut on vote flips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User direction (2026-08-15): back to predicting swing turns, D1 charts,
fractals over ZigZag pivots (their call - balances classes, matches the
reference library target, and a 5-bar fractal confirms 2 bars after its
extreme so labels resolve nearly to the present with no repaint embargo).
- TRAINING_TARGET enum + TrainingTarget input: TARGET_BARRIER (Market
default - existing models keep their meaning and fingerprints) or
TARGET_FRACTAL (private default).
- FractalDirectionLabel (Labels.mqh): per-bar 3-class label = direction
from the bar close to the next confirmed strict 5-bar fractal extreme,
costs charged in the same bid-series convention as the barrier label,
Neutral when the move cannot clear max(2 spreads, 0.10 ATR) or on an
outside bar (both-extreme bars are unorderable within OHLC).
- The barrier walk still runs in full: measured SL/TP geometry, the
expectancy scan, excursion caches and the era gate all keep scoring
what a trade at the EA's own stop/target actually collected - only the
TRAINING label changes. NOT the pre-b4a704d "is this bar the pivot"
form; that target's 31:1 imbalance stays retired.
- Fingerprint token |TGT:FRA1 so switching targets trains a separate
model; AI_META unaffected (guarded setter).
- Private defaults: AIType back to AI_HYBRID (direction topology needed)
+ TrainingTarget=TARGET_FRACTAL = drop-on-D1-chart workflow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User request: NNs predicting pivots (fractals for label density). Run on
the offline stack that demonstrably CAN learn (+2.6pp XAUUSD meta), free
of every historical in-EA training bug: pooled 4-symbol training,
per-symbol norm, scale-free causal features, target = side of current
price the next confirmed Bill Williams fractal lands on, real M1
ask/bid fills, threshold fitted on calib only, pre-registered 2-sigma
net-expectancy gate.
Result: train CE 0.682 (a whisper below the 0.693 coin), and on test no
symbol passes - EURUSD/USDJPY net zero, XAUUSD gross +0.22 pts vs a
larger spread (net -0.30), SP500 net +0.05 +/- 0.45. The gross-positive
tails are index drift plus sub-spread micro-reversion - the tick-flow
decay shape at swing scale. Seventh independent measurement of the same
fact: entry-time direction information is not in these features.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Faithful Python port of ADZigZag (stock MetaQuotes ZigZag 12/5/3,
verbatim rebrand) including the incremental prev_calculated branch, so
the indicator can be replayed bar by bar exactly as it draws live.
SP500 H1, 74,599 bars, fills at real M1 ask/bid:
- FINAL swings: 4,658 legs, mean 50.8 pts = 95 spreads. Perfect
foresight +50.2 pts/leg. The user premise (swings dwarf spread) is
fully confirmed.
- LIVE: 81% of drawn newest-pivots later repaint away entirely
(18,805 of 23,299). Holding the drawn direction at every bar close
grosses +0.38 pts/trade (t=0.8, zero cost charged) out of the
50.8-pt average swing - 0.7% of the line the chart ends up showing.
- LONG +1.66 gross / SHORT -0.90 = the index drift, nothing else;
long net +1.17 pts / 13-bar hold = ~1.5 bp, under one night financing.
The spread subtracts 0.49 pts of a swing that hands over 0.38: cost was
never the obstacle - pivot knowledge is.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User claim: swings dwarf the spread, so cost cannot be what blocks swing
trading at 1:2/1:3 RR. Measured on the validated M1 bid/ask book, SP500
H1, ATR-scaled causal ZigZag at 4 reversal thresholds:
- The premise is CONFIRMED: median swing 36-79 spreads, mean up to 110.
Perfect-foresight expectancy +28 to +59 pts/leg.
- The conclusion does not follow: trading every confirmed leg (enter on
the ZigZag confirmation close, real ask/bid fills, exit on the next
confirmation) grosses -0.2 to -0.5 pts/leg AT ZERO COST, on 3,681 to
13,871 legs. The confirmation retracement - the event that DEFINES a
pivot - consumes the entire swing before the spread is even charged.
- Long/short split is symmetric around the index drift (LONG +1.18,
SHORT -2.17 gross at 3xATR), i.e. no swing structure beyond drift.
- 3.0xATR reversal reproduces the EA ZigZag cadence exactly (median leg
17 bars, 49 legs/1000 bars vs the EA measured 17 and 44).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three additions to meta_pool.py, in the order the campaign needed them:
- memmap + float32-throughout (per-batch float64 cast): the 6.5 GB
4-symbol corpus OOMed the float64 pipeline on the training box;
- pool2: per-symbol standardization (each symbol by its own train-slice
mu/sd) + 64/32 capacity + l2 1e-3, after the naive pooled model
underfit to the prior (train CE pinned at base-rate entropy);
- curve: fixed-ladder precision-vs-threshold on calib and test side by
side - the dose-response diagnostic that closed the question.
RESULT recorded in memory: pooling transfers real skill (XAUUSD +2.6pp,
SP500 +1.2pp at fitted thresholds, >>2 sigma) but 0/8 fitted operating
points clear break-even, and the high-conviction tail is temporally
unstable - the precision-vs-threshold slope FLIPS SIGN between calib and
test on 3 of 4 symbols, so no ex-ante threshold rule exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The candidate sweep reserved bars*2 slots and guarded the bar loop with
room for only 2 appends, but every bar can append m_srcCount*2 candidates
(4 families x 2 sides) and STATE-model patterns stay active on most bars.
Two failure modes, both observed on the first multi-chart attach:
- USDJPY/XAUUSD/XTIUSD H1: mid-bar overflow -> "array out of range in
SignalMETA.mqh (369/379,25)" -> EA dead on the chart, panel frozen at
"getting ready".
- SP500 H1: the guard tripped exactly at cap (109,508 = 54,754*2), a
SILENT truncation that dropped the newest bars from the corpus - the
sweep walks oldest-first, so what fell off was the most recent history.
The arrays now grow 1.5x whenever headroom for one full bar is missing,
the loop runs to completion on every symbol, and a shrink-to-fit after
the loop returns the slack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User request: attaching a chart must need zero Inputs-tab edits. Private
(non-Market) build now defaults to AIType=META, all four classic families
ON (they are the sweep's candidate sources), Meta_ExportDataset=true.
Market-build defaults unchanged (AI_NONE, MA/RSI only, no export);
UseDatabaseRanking=true applies to both per the earlier request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user should not need a tester corpus run per symbol. Every pattern
condition in Signals\Signal{MA,RSI,MACD,Ichimoku}.mqh anchors its reads on
`int idx = StartIndex()` with zero hardcoded indices (verified), so a
name-hiding StartIndex override + EvalShift(i) on CExpertSignalCustom makes
the EXACT live ladder code answer "what would you have fired at bar i" -
the silent-divergence trap that justified the DB corpus does not exist on
this path, and neither do the GMT-offset ambiguity, the DB row caps, or
the wipe procedure.
- CExpertSignalCustom: m_evalShift + StartIndex()/EvalShift() +
SweepPrepare(bars) (deep-resizes the shared price series); the four
classic signal classes override SweepPrepare to deep-resize their own
indicator buffers.
- CSignalMETA::BuildCorpusBySweep: per bar x per source filter, run
Direction() shifted, harvest the per-side pattern slots + netVote into
the same corpus arrays the DB loader fills; entry=bar open so
MetaPrepareEra's resolution matches at offset +0 with zero price error.
DB corpus remains the fallback when classic filters are disabled.
- Warrior_EA.mq5: META gets the enabled classic filters as candidate
sources (family ids match the descriptor one-hot).
- UseDatabaseRanking default false -> true (user request): a META chart
journals + ranks out of the box.
Workflow per symbol is now: attach ONE chart with AIType=META (optionally
Meta_ExportDataset=true for the offline pool) - candidates, labels,
training and export all happen in place, ~10 seconds of sweep instead of a
tester run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Loads the EA's MetaExport .f32 datasets (UTF-16 sidecars), applies the EA's
own discipline offline: chronological 55/15/30 split with horizon-length
purges, operating point fitted on the calibration slice only via
coverage x (precision - BE) with the 25% floor, test slice touched once,
deployability at the 2-sigma edge floor. Small leaky-ReLU MLP + Adam in
numpy; `stats` / `eval <tag>` / `pool` commands.
First run on XAUUSD_16388 validated the plumbing and exposed the data:
the 2.5h gold tester run only covered 2004-07..2006-10 (3,214 candidates)
because gold tick volume is huge - and corpus builds do not need ticks at
all (journaling is bar-open-keyed, labels come from bar history later), so
"Open prices only" modeling builds the same corpus ~100x faster.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Meta_ExportDataset input: with AIType=META the chart writes its complete
training set once per attach - every resolved+labeled candidate as
[barTime|family|pattern|side|won|NetInputWidth floats] using the SAME
window builder, descriptor and label caches pass 2 trains on, so offline
examples are byte-equivalent to the EA's own. Sidecar .meta.csv carries
layout + the geometry/BE the labels were computed at. Files land in
Common\Files\Warrior_EA\MetaExport\<sym>_<period>.f32.
This is the pooling architecture decision: multi-symbol training INSIDE the
per-chart God-class would be the riskiest surgery this codebase has seen;
instead each chart exports, the pooled head trains offline (small dense+BN
net, minutes on this box), is validated per-symbol under the same
chronological splits and coverage x (p - BE) gate, and only a WINNER gets
written back into a .nnw for the EA to load natively (format fully mapped).
Also turns every future meta experiment from a 20-minute tester cycle into
minutes of offline iteration.
Cost-model note for the record (user challenge, verified): spread is 0.099
ATR = ~2% of the 4.74 ATR trade width - tiny per bar, but expressed in
win-rate points it is 0.099/4.74 = 2.1pp, which is the measured base-vs-BE
gap and the size of the entire observed skill lift. Zero-spread relabeling
would put base == BE by construction. Multi-day holds additionally pay swap,
which the label does NOT charge - the true bar is higher, not lower.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
999 eras on 13,436 H4 candidates. The cost gap halved exactly as computed
(BE 64.1 vs base 63.0 = 1.1pp) and the lift did not come with it: max
cov x (p-BE) = +0.07 in 1 era of 999, selective-threshold skill +0.47pp
mean (noise), OOS ranking slightly inverted (skips won 67% vs 62% for
trades) so the expectancy fitter correctly pinned coverage at 100%. The
lift shrank faster than the cost - the tick-flow decay shape, now measured
at the setup-conditional level. SP500 is closed at both accessible cost
points under pre-registered hypotheses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The S2 verdict localized precisely: the meta head's edge x width (0.02 x
4.74 ATR = 0.095 ATR/trade) equals the measured spread (0.099 ATR/trade) -
real signal, consumed exactly by cost. The breakdown line adds: the lift is
LONG-ONLY (shorts anti-selected) and MA-family-strongest (67-70% traded win,
<1 sigma over BE on ~350 trades, best-of-32 cells - not family-wise
evidence).
Next experiment, pre-registered in Meta_Labeling_Design.md before any H4
data exists: SP500 H4 doubles ATR against a fixed spread, halving the cost
drag (~1.3pp) that the ~+2pp lift must clear. Same pipeline end to end;
deployability still decided by the unchanged 2-sigma gate. H3 (the honest
risk) is that the lift decays with timeframe as fast as cost does - the
tick-flow failure shape - which would close the single-instrument well and
leave cross-sectional pooling as the only lever.
Enabler fixed here: LoadMetaCorpus picked the LARGEST .db on disk, so an H4
chart would have adopted the (bigger) H1 corpus and resolved candidates onto
wrong bars - and a chart could even adopt another SYMBOL's corpus. The
loader now requires a <symbol>_<period>_ filename match and says so when
nothing matches.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A resumed META model hot-looped pass 1 (0->100% scan oscillation, silent for
3 minutes until the stall reporter fired) because EVERY window failed at the
first AD/Wyckoff feature: the init-time param adoption called
ReInitADIndicators unconditionally, destroying five freshly-calculating
indicator instances to recreate them with BYTE-IDENTICAL params (verified by
parsing the .nnw header - the MI tuner had kept the configured settings), at
process start, on a box with 1 GB free of 31. The replacements sat cold for
6+ minutes while full-history resweeps starved the indicator threads harder.
- AdoptIndicatorParams: installs a loaded param set into the tuner and
rebuilds handles ONLY when the set actually differs from what the live
indicators run. Both call sites (resume init + panel reload) use it.
- Resumed models get the same 3 warm-up passes as fresh ones. The skip was
the shared root cause of the cold-ATR (ba13eef), cold-AD (2026-08-11) and
this incident - custom indicators recompute from scratch every process
start regardless of what the .nnw proves.
- Cold-sweep backoff: a pass-1 sweep in which every window failed on a
TRANSIENT cause arms a 5s era-start pause instead of an immediate
full-history resweep, so the retry loop stops consuming the CPU/memory the
warming indicators need. The stall reporter names the backoff branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
350-era S2 verdict on SP500 H1: the meta head carries REAL ranking skill
(+1.0-1.3pp mean over base, 101/350 eras clear their own 2-sigma bar, traded
subset wins 66.1% at <30% coverage vs 64.5% base) but 0/350 eras produced a
positive cov x (p - BE): the candidate stream sits 3pp under the derived
geometry's 67.5% break-even and ~2.6pp of recovered skill cannot bridge it.
Skill plateaued by mid-run (1.28pp -> 1.05pp), so more eras only buy
multiplicity, and the deploy gate correctly shipped nothing.
The aggregate can hide a deployable subset (one family/side clearing BE
blended with junk), so the META era line now decomposes the SAME traded
population into MA/RSI/MACD/Ichimoku x LONG/SHORT cells, each as
traded/candidates base->traded win rate. 32 cells is a best-of-N search by
construction - any candidate cell faces the family-wise rule before belief.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The NN now has a target that is not per-bar direction (closed, best-of-999
p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of
cost). One net for all 52 pattern-sides, AIType=AI_META.
- NetForward.mqh: the host-side softmax+CE gradient generalized total==3 ->
2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no
compute backend changes.
- SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on
disk (decoupled from the config fingerprint that burned four S1 runs); the
GMT->server offset is measured PER ROW against entryPrice vs bar open
(DST-immune, histogram logged); a window-span regime filter drops the
pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the
input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR).
- Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate
calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell
lets checkpoint selection, the edge floor, the plateau ladder and the
family-wise deploy gate run UNCHANGED: precision reads as win rate among
traded candidates, chance as the base win rate, recalls as sensitivity/
specificity. Era-end META line: coverage x (p - break-even) vs the null.
- Labels are the side-conditional triple-barrier win caches - never the DB's
stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base
rate). Live inference + online learning guarded off until S3.
- Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename
slot keep meta models fully separate from direction models.
Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with
AIType=AI_META; S3 wires the votes via the per-side hooks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The warning lived inside the VerboseMode-gated corpus report, so a
forgotten wipe silently voided an entire 18-year corpus run - the
outdated-row guard rejected the whole replay against leftover rows
and the run appended 35 rows instead of building a corpus. The check
now runs unconditionally at tester OnInit (MetaCorpusStaleCheck): 52
quiet one-row newest-key probes vs the test start, with a loud stop-
wipe-rerun instruction when the DB is newer than the test. Absent
tables probe quietly via FetchNewestTimeKey''s new quiet flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
6819bb4 called dbm.FetchRecordCount() from ProcessSignal, but the
method only existed on CDatabaseOperationsManager - CDatabaseManager
never exposed it (nothing outside the DB layer had needed it before).
The 12 compile errors were the usual MQL cascade from one unknown
member.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The historical 1000-row cap existed for a real reason: ProcessSignal
pulled BOTH full tables into MQL struct arrays on every buffered
signal, and UpdateSignalsWeights pulled all 52 per cycle -
materializing thousands of string-bearing structs per event is the
practical limit the cap protected against (SQLite itself has none).
Raising the cap for an 18-year meta-label corpus build would have
made runs crawl; sharding across databases would re-read the same
rows and inherit the same cost.
Every question is now answered inside SQLite, one row or one number
per query, flat in table size:
- FetchOpenTradeEntry: the open (NA) trade''s entryPrice for
pattern+direction, LIMIT 1
- FetchNewestTimeKey: newest row''s yyyymmddhhmm via max ROWID
(rows insert chronologically) - the duplicate/outdated guard
- FetchWinLossCounts: COALESCE''d SUM aggregates with the
before-now bound applied in SQL, replacing the tester-only array
trim (now also active live, where it is harmless by construction)
ProcessSignal semantics preserved exactly: prune -> close opposite
(stop-and-reverse still registers its own row) -> duplicate/outdated
-> one-open-trade -> register. CalculatePatternWinRate''s array walk
becomes WinRateFromCounts; the private FetchTradeRecords wrapper and
ShouldDeleteOldestEntry are gone. DB_MaxRowsPerTable=20000 is now
cheap at any table size.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first corpus build produced 3,681 rows, all 2026, from an 18-year
backtest: ProcessSignal''s outdated-row guard rejects any registration
older than a row its table already holds (correct for a live stream),
so a tester run starting before the leftover rows'' dates silently
registers nothing for the overlap. The corpus report now prints a
loud WARNING when running in the tester with DB rows newer than the
test''s start: corpus builds start from an empty DB.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements stage S1 of Meta_Labeling_Design.md, superseding the
original "training-time ladder sweep": the per-side journaling from
652bf81/195be20 already produces the exact candidate stream a sweep
would compute - every pattern instance the live ladders fire, both
sides, uncensored, with netVote and touchable entry price - so the
corpus is READ from the DB instead of re-implementing 26 ladder
conditions in training code. That eliminates the silent-divergence
trap outright: the corpus is by construction identical to live
behaviour. Accepted costs are documented in the module and the doc:
coverage equals the populating backtest, and sampling is one
candidate per fire-stretch (the right dedup for training anyway).
- Expert\AIBase\MetaCorpus.mqh: CMetaCorpus reader (52 tables ->
SMetaCandidate rows) + VerboseMode OnInit report: volume/closed/
S&R-win-rate per family, span, and the GMT->server bar-offset
match table (offsets +0..+3h) that S2''s label plumbing pins to -
measured, not assumed.
- DB_MaxRowsPerTable input (default 1000 = old MAX_TABLE_ROWS): a
corpus build raises it (e.g. 20000) so a 15-20 year backtest
isn''t pruned; wired through CExpertSignalCustom::MaxTableRows().
- Report-only stage: nothing downstream consumes the corpus yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agreed architecture pivot: classic patterns become candidate
generators (WHEN), the excursion head keeps geometry (SHAPE), and a
new meta-head predicts per-instance P(win at the EA''s own geometry)
(WHETHER), replacing the AI direction vote whose question is measured
closed. One net for all patterns, setup descriptor appended to the
existing feature window, triple-barrier side-conditional labels,
existing optimizer/selection/gate machinery reused. Honest floor
stated: if primaries carry zero structure the head degenerates to a
vol/cost/session timer - bounded, and the family-wise gate decides.
Four compile-alone stages; cross-sectional pooling is the next lever
after S4. No SQX parsing anywhere in this path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
seasonal.py gets a frame-based entry (analyse_frame) and a reusable
report() so the identical statistics - circular-rotation family-wise
null, max-|t| bar, split-half - can run on instruments whose book is
synthesised from M1 bars. breadth_seasonal.py runs it on the five
SQX-decoded instruments (FTSE100, UK100, WTI x2 feeds, USDCAD) that
share no data path with the four originals; the duplicate-market pairs
(FTSE100/UK100, WTI_d/WTI_5) double as replication checks. Caveats
stated in the module docstring: synthesised flat spread (move/spread
is approximate, no intraday spread shape) and file-time clock labels;
drift/t columns are spread-free and unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
