NOT COMPILED - user compiles.
Root cause of the Neutral collapse. Logit adjustment (Menon et al. 2020) makes a
classifier Bayes-optimal for BALANCED error by subsidising rare classes. It was
wired here when Neutral was the DOMINANT class - the "big move up / big move down
/ nothing much" era, where the correction pulled the model off the majority.
The triple-barrier relabel (b4a704d) inverted the distribution. The barriers are
now the EA's own SL/TP, so ~89% of bars RESOLVE and only timeouts are Neutral.
Measured on SP500 H4, from the EA's own log:
measured priors Buy 48.26% Sell 41.13% Neutral 10.61%
log-prior spread 1.52 | tau 1.00 CAPPED to 0.79
Neutral became the RAREST class, so the correction started subsidising it - by
tau*(log pB - log pN) = 1.20 logits. With no directional edge to overcome that
(direction is closed at best-of-999, p=1.0000), the model took the free lunch:
OOS recall Buy:1% Sell:0% Neutral:100%
OOS raw out spread avg 0.9993 (softmax saturated, near one-hot)
dW/W bn1 0.000% bn3 2.0% bn5 6.5% (input weight block frozen; head twitching)
The anti-collapse mechanism was the collapse. The recall gate needs >=40% on all
three classes, so nothing could ever deploy and the plateau ladder burned eras.
Present in both runs today (b6b5 froze bn1 by era ~719, 17ae by ~169), so it
predates this week's work.
FIX: the correction now spans the DECIDABLE classes only, Buy against Sell,
centred on their midpoint, with Neutral pinned at offset 0. Neutral is the
ABSTAIN outcome and abstention already has a better owner - m_dirConfThreshold,
refitted every era on the held-out calibration band against a coverage floor and
the measured break-even. Subsidising the abstain class does that job twice and
spends the whole correction suppressing the only decisions that can pay.
What still gets corrected is real: a trending symbol resolves more long barriers
than short, and uncorrected the model inherits that as a standing directional
bias. Here it is log(0.4826)-log(0.4113) = 0.16, so the offsets are tiny - the
correct answer, not a broken one. The two traded classes were already balanced;
the old spread of 1.52 only ever described how rare a timeout is.
Everything is derived from the measured distribution, as requested - offsets from
the priors, cap from the resulting spread. tau itself is deliberately NOT fitted:
tuning it against the same data that selects the checkpoint would add another
search dimension to a project that has been burned by exactly that. tau=1 is the
theory value and the cap (now ~9.5x looser at spread 0.16) will rarely bind.
Log line now reports both spreads and, when the abstain class is the rarest, says
how much the old form would have boosted it. Fingerprint |LA:<tau> -> |LA:<tau>:BS
so models trained under the all-three form re-key instead of resuming.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b197999 gated the repaint on "this era beat the best checkpoint". That was not
what was asked for and it changes policy rather than fixing a bug, so it is
reverted - behaviour is now byte-identical to before b197999. Only a comment
recording what was verified remains.
What the check found: there is NO repaint or erase at the start of an era. With
NMS on, passes 1, 2.5 and 3 only RECORD predictions into m_arrowSignalCache -
pass 1's own comment says "NMS on: record only - the era-end sweep is the SOLE
renderer, so no raw (un-declustered) arrow is ever drawn mid-era" - and
PruneDirectionalClusters runs once, at the end of pass 3, which is the end of the
era. The requested behaviour was already the implemented behaviour.
So the arrows vanishing at the era boundary is not a timing fault. That sweep
DELETEs the arrow on any bar the era scored Neutral, and the model is currently
scoring Neutral on 98-100% of bars (see the logit-adjustment finding: Neutral is
the RAREST class at 10.6% and the imbalance correction is subsidising it by ~1.2
logits). The chart is reporting the model accurately; the model is the problem.
One thing that CAN clear arrows at era 0, and did on the first attach after the
|ALTW re-key: ClearPersistedChartSignals("fresh topology at era 0 - arrows belong
to a previous model"). That fires once per fresh model, not per era.
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
User report: "as soon as the next era training begins the chart signals are
erased, they should persist for as long as they are accurate."
Cause: PruneDirectionalClusters ran unconditionally at the end of pass 3, and it
DELETEs the arrow on any bar the CURRENT era scored Neutral. A model exploring
away from its best therefore wipes the chart every era even though the best
checkpoint still calls those turns. On the run that prompted this the model sat
at OOS recall Neutral:100% for 40+ consecutive eras, so essentially every arrow
was deleted at every era boundary.
The render is now deferred to the era-end block - the first point that knows
whether the era beat the best checkpoint - and only a new best repaints. On eras
that did not improve, the previous best's arrows stay untouched. Two exceptions
keep the chart from ever showing nothing: before the first checkpoint exists
there is no best to preserve, so early eras still paint; and a finishing run
repaints unconditionally, because FinalizeTrainRun is about to restore the
deployed weights and the chart must describe THOSE.
Recorded for ensemble members too. A member's own best era is not the deployable
one (the joint checkpoint decides that), but it is still the most accurate thing
that member has drawn, and the alternative is a chart that empties itself.
Also verified against the log, since two other symptoms were reported alongside:
era cadence PAI-b6b5 (before these changes) 3.76 s/era
PAI-17ae (after) 3.53 s/era
topology both "2 dense from 16 units | input 800 (16 bars x 50)"
calibration both fitted on 1699 held-out bars
So training speed is unchanged - the ~6% is the ranking slice removing 20% of
pass 3's bars. It only FEELS fast because this is a single PAI chart taking the
whole 120 ms budget, not four ensemble members sharing it behind an era barrier.
The Neutral collapse is also pre-existing, not new: b6b5 ran at Neutral 94-100%
with 1-5% directional calls for all 723 of its eras, before any of this work.
That is the known neutral-collapse/recall-gate failure mode, and it is what
"barely drawing signals" actually is. Worth watching, separately: b6b5 reached
best-bal 34.2% by era 723 while 17ae is at 13.6% after 77 - too early to read,
but it is the number to check once 17ae has run comparable eras.
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
User's call before deploy: "I would rather avoid lagging so the NN finds
accurate patterns." Correct instinct, and it picks the conservative variant.
110b384 deduplicated the WHOLE window, so every distinct reading survived at one
slot. The flaw is which slot: it depends on where the calendar-day boundary falls
inside that particular window, and on H4 that boundary cycles through ~6 phases.
A dense layer holds a separate weight per (slot, feature), so a given lag would
have landed on a different coordinate from one window to the next - turning a
stable lagged input into a moving one.
Now it blanks only bars carrying a BYTE-IDENTICAL copy of the anchor's reading
and stops at the first bar that differs. An as-of lookup into a daily file is a
step function in time, so those copies are exactly the contiguous run of bars
sharing the anchor's calendar day. Everything older keeps its natural replicated
run, in the same slots it always occupied - whatever the net learned to read
there, it still reads there.
Why the anchor's reading is the right one to isolate: the window's newest slot IS
the bar being predicted (BuildFeatureWindow's final iteration lands on r, and
pass 3 grades that same index), so it is the reading contemporaneous with the
decision - and the only one the alt screens ever validated. They measured the
CURRENT reading's MI against forward range and never tested lags, so the lagged
content is unproven, which is a reason to leave it undisturbed rather than a
licence to rearrange it.
What is still fixed: the anchor's reading reaches the first layer on one
coordinate instead of once per bar of its day, removing the ~16x gradient
upweight for the validated signal. And this is IDENTICAL to full dedup exactly
where replication was worst - on M15/H1 the whole window sits inside one calendar
day, so the anchor's run is the whole window - and a no-op on D1, where the bar
before the anchor is already a different day and the loop breaks immediately.
The two differ only on middle timeframes, and there this is the safe side.
Fingerprint |ALTW:1 -> |ALTW:2 so nothing trained under the hour-old full-dedup
semantics can silently resume under these.
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aba9bd2 kept only the newest slot's copy of the external block. That is right on
every intraday timeframe and WRONG on D1: there the 16 window bars are 16
distinct calendar days, the daily alt file returns a different row for each, and
blanking 15 of them destroyed real information instead of a copy of it. Caught
while extending the measurement to the other instruments.
Now compares values instead of slot positions: walk newest -> oldest, keep the
last DISTINCT reading, blank a slot only when it repeats one a newer slot
already carries. Exact on every timeframe with no timeframe test, and it also
handles weekends, holidays and publication gaps, where a window spans fewer
distinct rows than calendar days. How much collapses falls out of the data:
M15 x 16 bars = 0.17 calendar days -> 1 distinct row -> 15 of 16 blanked
H1 x 16 bars = 0.67 calendar days -> 1 distinct row -> 15 of 16 blanked
H4 x 16 bars = 2.67 calendar days -> ~3 rows -> ~13 of 16 blanked
D1 x 16 bars = 16 calendar days -> 16 rows -> NOTHING blanked
OTHER INSTRUMENTS - the question that prompted this. Per-symbol exports for
USDJPY/XAUUSD/EURUSD are not on disk (written only when that chart is attached),
but they are not needed: CAltData reads a DAILY file for every symbol, so bars
sharing a calendar day are byte-identical by construction everywhere. What
varies per symbol is only WHICH sources, and the catalog (AltDataFetch.mqh
AddSpec rows) gives:
SP500 13 = risk(3) + cot_spec_net(1) + eia(3) + mac(6)
EURUSD 15 = cot(3) + risk(3) + eia(3) + mac(6)
USDJPY 15 = cot(3) + risk(3) + eia(3) + mac(6)
XAUUSD 14 = risk(3) + ivol(2, GVZCLS) + eia(3) + mac(6)
Measured observation-date gaps in the raw sources on disk: VIX, USD index,
DGS10, T10Y2Y, T5YIE, DFF, ECBDFR all 1 day; COT and EIA 7 days; CPI and UNRATE
31 days. NO per-bar source exists anywhere in the catalog - the tick-activity
survivors from the flow screen are an in-terminal feature block, not alt data,
and are untouched by any of this. So the redundancy is universal across
instruments; only its magnitude varies, and by timeframe rather than by symbol.
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured on the live SP500 D1 export (6073 rows, 13 features, 5888 simulated
16-bar windows):
distinct values per feature per window : 1.7 - 2.7 of 16 slots
variance in the first 13 PCs : 96.5 - 97.0%
components for 95% / 99% : 12 / 17-19
effective rank (entropy) : ~11.5
208 inputs carrying about 12 dimensions. Only 6 of the 13 features move daily
(VIX complex, USD, the rates trio); 5 are weekly (COT, EIA, output gap) and 2
monthly (CPI, unemployment). The lookup is as-of by bar open time into a DAILY
file, so bars sharing a calendar day are byte-identical by construction.
The cost is NOT overfitting capacity - collinear copies span ~12 directions,
not 208, so an earlier claim that this wasted 26% of the model overstated it.
It is GRADIENT WEIGHTING. Batch norm standardizes each of the 208 coordinates
independently; that rescales the copies without decorrelating them, so one
factor arrives on 16 unit-variance coordinates, each weight takes a full-size
step, and the factor's aggregate coefficient moves ~16x faster than a per-bar
price feature's. The network was biased toward the external block by a factor
of the window length - and pointing the wrong way, since these features cleared
only a marginal incremental screen while price is the base signal.
Zeroed at WINDOW ASSEMBLY, not in BufferTempData: that output is cached PER BAR
and a bar sits at slot 15 of one window and slot 0 of the next, so a
slot-dependent value there would poison the cache or force a recompute per slot.
The cache keeps true values; only this window's copies are cleared. Width
contract untouched - same count, same positions - so conv/LSTM/HYBRID keep their
bar-major rectangle unchanged and the block arrives at the newest bar, which for
the LSTM is the final timestep. Zero-variance coordinates are safe through batch
norm (divisor is MathMax(MathSqrt(var + BN_EPSILON), BN_MIN_STD)).
Fingerprint gains |ALTW:1 when alt data is on. Same width and same .cfg, so
nothing else would have caught a model trained under the replicated layout
resuming under this one. Conditional append per the existing rule: configs
without alt data keep their fingerprints and their trained models.
NOT the concat branch. CNet is a strictly linear stack (CLayerDescription has no
input-source field; NetBuild wires i to i+1 and stores layer L's weights on
L-1), so a real two-tower model needs a new multi-input layer type across
WarriorCPU, WarriorDML and the OpenCL kernels plus an .nnw format change - the
highest-risk change in this repo, in the code that produced the transposed dense
gradient, the Adam second-moment bug and the reversed LSTM window. This captures
the part of that idea the measurement actually supports, at no engine risk.
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor.
The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS
first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This
codebase MEASURES that width, and on the live SP500 H4 config it is 16 units -
already floored, with the budget printing "11360 estimated in-sample bars
cannot support a 800-wide input ... roughly 1.1 weights per training bar -
expect overfitting". At 16 units a floor of 20 makes lastHidden >=
m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch
and the width taper - the only part derived from this symbol's data - became
dead code on all four ensemble members, with depth (2 -> 4) set entirely by
counting feature domains. ComputeLayerWidths had already rejected this exact
pair of constants in its own comment.
The causal floor's premise does not hold either: layers are not inference
steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is
Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko
1989 and Hornik 1991 give universal approximation from a single hidden layer.
Depth buys parameter efficiency for compositional functions, not reasoning
hops. ForceHiddenLayers remains for measuring depth directly.
RANKING SLICE - the backfill no longer reads the window it is judged on.
The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window,
so win rates measured back over it are selection-inflated, and the backfill
was writing exactly those into the table filter weights rank on: the
selection set consumed twice, beside a deploy gate that applies a Sidak
correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS
window, plus a label-horizon purge, is now reserved and graded by nothing -
not pass 3, not checkpoint selection, not the gate. The backfill reads only
that. The gate keeps ~80% of its measurement (power goes as the square root,
so ~10% of a sigma), and the slice is the newest data, which is the regime
about to be traded. RankSliceBars returns 0 when no honest slice fits and the
backfill then REFUSES and says so, rather than falling back to the scoring
window and looking like a success.
SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate
by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw
ratio at the minimum sample count carries a ~15pp standard error, so a tier
that went 8-2 was handed weight 80 and outranked a tier measured over
hundreds of calls at 55 - the ranking was being driven by which small tier got
lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour).
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four defects in 64c5dd5/1a05e63, found by review + a baseline compile.
Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable.
1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were
declared `virtual bool ... override`, but CAppDialog declares both as
`virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151
on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was
never a success flag to forward. Verified: 0 errors, 0 warnings.
2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual
simulation (that one has been dead since it was written). Both are armed
at the instant convergence is declared, and both advance only from inside
Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick
ArmStudyEvent site sits in the `else` of a branch taken whenever
m_trainingComplete is set and m_trainRunActive is clear - which is exactly
the state FinalizeTrainRun() leaves behind one line before they are armed.
Train() was never called again, so the walks sat at their start index
forever: no "simulation complete" line, and not one row written to the DB
this feature exists to fill. Only a manual Resume/Retrain unstuck them.
Both flags now keep the model schedulable.
3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed.
Ensemble members deploy at Train() ENTRY and return immediately (so no era
is wasted), which skips the era-end block the backfill was started from.
All four members were a no-op for a second, independent reason. Armed on
the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff.
4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key,
no duplicate check - and m_dbBackfillDone is in-memory, so every later
attach that retrained to convergence wrote a second full set of rows for
the same bars. The ranking would count one bar once per model that ever
deployed, weighting superseded opinions as heavily as the live one. A
.dbfill marker stamps the deployed era; written only on completion (an
interrupted walk redoes itself rather than ranking a partial window) and
deleted with the other sidecars on reset-weights.
Also: WarmBlocking's timeout was silent, which restored the exact silent
pin failure it was added to prevent - it now says so in the journal, and
returns true for "no reference pairs to wait for" so the warning stays rare
enough to be read.
Not addressed, needs a decision: the backfill scores the OOS window with the
checkpoint that was SELECTED as best on that same window, then writes those
win rates into the table filter weights rank on - the selection set consumed
twice, undiscounted, while the deploy gate right next to it applies a
family-wise correction for exactly that effect. The rows are also simulated
triple-barrier outcomes at today's spread sharing a table with realised
fills. The completion log line now states both plainly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ensemble chart UI had a shared-namespace defect that answered the user
question "what do the arrows represent?" with "a bug": all four members drew
arrows under the same WarSig_<bartime> object names, so the chart showed
whichever member rendered LAST, one member Neutral deleted another member Buy
at the same bar, each member init sweep wiped the arrows the previous member
had just restored, and SaveChartSignals - which rebuilds the sidecar by
SCANNING the chart - persisted every other member arrows into its own history
(the exact cross-model laundering its own header warns about, now happening
BETWEEN ensemble members).
Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_,
WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor
purge and the sidecar scan are all member-scoped, and the tooltip names the
model. Global purges keep matching the bare WarSig_ prefix, which covers all
member namespaces plus old-format leftovers from earlier builds.
Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is
one member; the header is the ensemble) and the CONVLSTM member displays as
ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the
model folder and changing it would orphan every model trained under that path.
Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins
WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now
cover it - it was in neither list, so a dialog starved of its own Destroy()
left its controls on the chart permanently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Log review of the 18:12 attach. The wiring works: VIX, dollar index, COT, all
seven macro series fetched and SP500_D1.csv rebuilt with its full 13 features
on the first pass. Three findings from the same log, fixed:
KEY LEAK: the EIA failure line echoed the first 80 chars of the URL, which
included most of the api_key. Every URL-echoing error path now goes through
MaskUrl(). The key itself is unchanged - it was printed to a local journal,
not transmitted - but rotate it if that log ever leaves the machine.
EIA HTTP 1003: an MT5 transport-layer code, not a server response. Requests
now carry a User-Agent (gateways reject empty-UA at the edge; the CBOE probe
showed no-UA is fine THERE, but EIA fronts differ) and 1xxx codes are
explained in the log line. Retries were already hourly.
UNBOUNDED BACKFILL: an empty cache fetched full series history - CPIAUCNS
goes back to 1913, whose pre-1970 dates are outside MQL5 datetime range and
whose 1913-era levels sat below the plausibility band, producing 157
scary-but-meaningless REJECTED lines. All FRED fetches now start at 2005
(5y of lookback margin ahead of the 2010 grid). DTWEXBGS staleness horizon
raised to 10 days to match its weekly H.10 publication lag.
Also confirmed from the log: the running build predates the H4 fallback, so
the H4 panels still show 0 features - resolved by the recompile this commit
requires anyway.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Systematic audit of the alt-data stack against "any symbol, any timeframe",
prompted by the H4 surprise. Findings, each fixed:
CAPACITY: ALTDATA_MAX_FEATURES was 16 with FX symbols already at 15 - the next
added column would have been silently truncated by a MathMin. Raised to 32,
pin-chars 512 -> 1024.
SYMBOL NAMES: the panel reload path re-derived symbol/timeframe by splitting
the file path on its FIRST underscore - mis-parsing every symbol containing
one (OANDA-style EUR_USD and US_500 are in our own alias lists) and knowing
only three timeframes. It now stores the (symbol, period) Load() was called
with and reuses them verbatim. Path-hostile characters in broker symbols
("EUR/USD") are sanitized by a shared AltDataFileSymbol() used by the panel,
the fetcher and TunedPeriods, so a slash cannot route a write into an
unintended subfolder.
DOWNLOAD VALIDATION: every FRED-family fetch now enforces a per-series
plausibility band (VIX 1-200, yields -5..30, CPI index 20-1000, ...) -
StringToDouble on transport garbage returns 0.0, and one absurd value poisons
every change/percentile feature computed across it (the BatchNorm NaN-latch
incident came from exactly one huge-but-finite input). Rejected rows are
counted and reported, never dropped silently.
LOUD EMPTINESS: a successful response with zero observations on an empty
cache now says so - naming the series (wrong id / format drift) or the COT
predicate (the unverified like-clauses) instead of leaving 0-filled features
unexplained. GEX gains a truncation guard: a day-over-day contract-count
collapse >50% is the fingerprint of a partial 13 MB download, not of markets,
and is skipped rather than recorded as a plausible-but-wrong number.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user attached H4 charts and the panel found no {SYM}_H4.csv - the fetcher
only writes _D1 files - so the run trained with ZERO alt features, silently.
The daily file is timeframe-agnostic by construction (rows are as-of daily
values, and Features() joins published <= bar open per bar), so the panel now
falls back to {SYM}_D1.csv on any timeframe, logging the substitution. A
per-TF file still takes precedence if one ever exists.
Per-symbol subfolders (the user suggestion) are NOT the fix for multi-chart
concerns: filenames are already symbol-keyed, and the shared caches
(raw_VIXCLS etc.) are shared deliberately - one download serves every chart.
The REAL races were: (1) two charts of one symbol (D1+H4) each caching their
own last-GEX date and double-appending the same day - UpdateGex now re-reads
the file date before spending the download; (2) whole-file rewrites were
truncate-then-write, so a concurrent reader could parse a torn file - SaveRaw
and RebuildFeatures now write a temp and FileMove-swap it into place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same harness as the D1 screens, forward 30 H4 bars (~5 days), 199 perms, on
the surviving htf mid bars (19.8k-36.4k bars per symbol). Question: does the
wired alt/volume information carry to H4, for the chart-timeframe decision.
Answer: the signal survives but is roughly halved, and the cost side worsens
2.6x per step down. SP500 vix_chg5 clears the family bar with MI|vol 0.0112
(vs 0.031 at D1); volLevel50 (the EA activity feature) is incremental on all
four symbols at H4 - USDJPY family-clean, and it is EURUSD strongest
non-control signal there too. Gold gvz_chg5 stays incremental (0.0038 vs
0.0197 at D1 - a fifth of the strength). COT is null at H4 on FX (weekly
cadence pasted across 30 bars/week dilutes it below detection on EURUSD/JPY;
survives conditionally on SP500).
Cost table (median spread/ATR; the 1.74xATR geometry in spread units):
SP500 138->52->25, EURUSD 282->113->57, USDJPY 230->87->44, XAUUSD 92->34->16
for D1->H4->H1. Every step down multiplies the cost share ~2.6x.
Also fixes the disaggregated-COT column name for gold/WTI in the screen
(M_Money vs Lev_Money - the same crash the EA-side catalog documents).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user whitelisted the hosts and still got the alert - because the alert
never said WHICH request failed. Two hosts (api.eia.gov, cdn.cboe.com) were
added to the EA after the original whitelist instruction, so any build newer
than the whitelist raises 4014 on the new hosts while the message implied the
old ones were the problem.
Three defects fixed: the popup and journal now print the exact blocked host as
a copy-paste whitelist line; the stale "three URLs" text is gone (the full
four-line reference prints once per session); and the backoff is per-host
instead of global - one missing entry no longer silences the whitelisted
sources for an hour per miss. Popup fires once per host per session; hourly
retries log one quiet line naming the host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POWER is the real find of the three: daily temperature -> degree days ->
natural-gas demand is the textbook gas fundamental, numeric and daily. But it
is point data needing construction into a national series (NOAA CPC ships that
ready-made), and its target symbol is not traded yet - fetch code written for
a chart nobody attaches first runs months later, unobserved, which is the
silent-FRED failure shape. Queued for the AvaTrade expansion, not refused.
FIRMS: re-raised, nothing changed since it was parked - point fire detections
behind the same unproven proxy chain. GIBS: imagery tiles, not numbers; our
CONV is 1D and NASA already sells the extracted products.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner decision (stated twice): available data gets wired; the networks judge
usefulness; the deploy gate remains the arbiter of what trades. Implemented:
MACRO block (6) on every symbol: 10y yield 20d change, curve slope, 5y
breakeven 20d change, Fed-ECB policy gap, CPI yoy, unemployment 12m change.
Screened null vs forward range on all four research symbols - recorded as
the honest prior in the catalog comment, wired regardless.
RISK block (3) extended to every symbol (FX majors, metals, energy, BTC all
now carry vix/vix_chg5/usd_chg5).
IVOL pair extended with the level alongside the change.
Vintage integrity kept where it is free: CPI is fetched as CPIAUCNS (NSA,
essentially never revised) so the plain-FRED backfill stays first-print-clean;
yields/curve/breakevens/policy rates are unrevised by nature. UNRATE is the
one exception (seasonal refits, ~0.1-0.2pp) - the EA cannot run the ALFRED
protocol, accepted and documented at the declaration site.
UpdateFred gains a staleDays parameter so the monthly series do not fire a
pointless fetch attempt every hour for three weeks after each print.
FeatureValue now takes the day and does its own as-of lookups - adding a
source no longer widens a parameter list. Feature counts: 12-15 per symbol;
symbol feature-order changed, safe only because no models exist yet.
export.py mirrors the new catalog for the five research symbols (13-15
features), smoke-tested: all five CSVs written, 6,072 daily rows each.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Screened yields (DGS2/DGS10), curve slope, inflation breakevens, Fed policy,
the Fed-ECB policy differential, and monthly US unemployment and CPI - all on
ALFRED first prints, 499 permutations, against forward 5-day range.
NOT ONE macro feature clears the family-wise bar on any symbol. The only thing
that clears anywhere is the trailing-range positive control, which is what it
is there to do. Best a-priori candidate, the Fed-ECB differential on EURUSD,
came in at MI 0.00170 p=0.088 - nothing. The two features flagged INCREMENTAL
(dgs2_chg5 on SP500) have null marginal MI and are isolated conditional cells
at the expected false-positive rate, not findings.
The `distinct` column quantifies the power argument instead of asserting it:
unemployment takes 51-66 distinct values across 3,745-6,159 bars, CPI 174-277,
against 6,159 for a continuous feature. A monthly series pasted onto daily bars
carries about 1% of the resolution, and it showed - the monthly features were
among the weakest in every table.
The contrast with the implied-vol screen is the useful part: the options
market FORWARD-LOOKING view of an instrument (gvz_chg5 on gold, MI|vol 0.0197)
carries real information about its range, while the economy BACKWARD-LOOKING
state carries none. Mismatched timescales - rate levels move over months,
5-day range moves daily.
Also makes load_bars fall back to htf/{SYM}_D1_mid.npz when the tick-derived
build is absent (the 2026-08-16 disk cleanup removed bars/ but htf/ survived),
with need_ticks=True turning that fallback into a loud failure for the
order-flow screen rather than silently testing flow features on OHLC data.
No EA change: nothing survived to wire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the screen_ivol survivors (41d726c). New per-symbol `ivolSeries` in the
catalog feeds a generic `ivol_chg5` feature from whichever CBOE vol index the
instrument owns, so one code path serves every symbol:
XAUUSD + ivol_chg5 (GVZ) - MI|vol 0.01971 p=0.002, 3.6x the positive
control and 4.6x the vix_chg5 gold had alone.
vix_chg5 KEPT: this appends, it does not replace.
EURUSD + vix_chg5 - screened, incremental p<=0.006, and its first
real feature ever (it had only exploratory EIA).
USDJPY + vix_chg5 - screened, incremental.
NAS100 / US30 / US2000 + ivol_chg5 (VXN / VXD / RVX) - exploratory by analogy.
XTIUSD / XBRUSD + ivol_chg5 (OVX) - exploratory, no oil bars to screen yet.
SP500 unchanged - its features already screened clean and VXN/VIX3M edging
out VIX is a correlated within-family best-of-N, not a real ranking.
On EURUSD/USDJPY the screen put VXD marginally above VIX, but they are
near-duplicates and the gap sits inside the noise, so the tie is broken by a
rule rather than by the number: take the series already in the fetch path.
Also fixes a real collector bug: fetch_vintaged built ALFRED realtime windows
out to 2028, and FRED rejects realtime_end after today - so every REVISED
series (unemployment, CPI, GDP: exactly the ones needing the vintage path) was
unreachable, while unrevised series never noticed because they bail earlier.
UNRATE and CPIAUCSL now return first prints correctly.
Adds screen_macro.py (rates, curve, breakevens, Fed/ECB policy differential,
plus monthly country stats) with a `distinct` column that reports the honest
effective sample size - a monthly series pasted onto D1 bars is a step
function, and that column is what decides whether it can clear a gate at all.
Not yet run: the Market Data bars directory is being regenerated right now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No free historical GEX exists: probed the CBOE chain endpoint with date/dt query
params (both silently ignored, returned today) and dated/historical paths (403),
and the CBOE index-history CSVs are 403 too. The forward recorder stays the only
path to GEX history.
But the options market publishes its per-instrument view of future range as the
CBOE vol indices, and FRED carries the whole family free with 15-25 years of
history - screenable today with the existing collector and harness. Fetched
GVZ (gold), OVX (oil), VXN, VXD, RVX, VIX3M.
HEADLINE - XAUUSD: gvz_chg5 (gold IV 5-day change) MI 0.02103, MI|vol 0.01971,
p=0.002. That is 3.6x the trailing-range positive control and 4.6x the vix_chg5
this project currently ships on gold - the second-largest incremental MI of the
whole campaign, on a symbol that carries exactly one screened feature today.
Vol-change is incremental on all four symbols: SP500 (known), USDJPY vxd_chg5
0.00492, and EURUSD vxd_chg5 0.00412 / vix_chg5 0.00379 - notable because
EURUSD has no screened features at all and its own trailing range is a weak
control there, so external vol carries information its own history does not.
Caveats recorded in the script and memory: SP500 within-family ordering
(VXN > VIX3M > VIX, all ~0.031-0.038 conditional) is a best-of-N artifact and
must not be cherry-picked; XAUUSD noise control misbehaved this run (MI|vol
0.00271 p=0.002), so anything under ~0.003 conditional on gold is unresolved -
gvz_chg5 at 7x that floor is unaffected; EVZ (euro IV) is DISCONTINUED since
2025-03 and must never be wired.
Nothing wired - the EA is mid-deploy and this would re-key every model again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spent 3 of 5 daily requests. All three were informative:
ETF data (SPY/QQQ/IWM) requires Basic - free tier is single stocks only.
Full-chain GEX (all expirations) requires Growth - free and Basic must
query one expiration per request, so even with history a full-chain
backfill would be 24-54 requests per day of history.
AAPL?expiration=2026-09-18 returned 200 with the right schema but a nearly
empty payload: 13 of 93 strikes carried any open interest, total call OI
4,296 against CBOE 373,253 for the same expiry, put OI zero, and every
near-the-money strike blank.
So the construction could not be validated - not because the math disagreed
but because there was nothing to compare against. From outside it is not
possible to tell free-tier degradation from their flow-signed methodology,
and finding out costs $1,499/month.
Verdict hardens: the free CBOE CDN is strictly better than Basic for this
project - complete chains, every expiry and strike, gamma and open interest
populated, unlimited, $0. Our own AAPL figures were internally coherent
(+0.929 Bn/1% total, Sep-18 expiry +0.154 Bn, near-money gammas 0.013-0.019).
GEX stays externally unvalidated; if that ever matters, use a different vendor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pricing checked: Free $0 (5/day), Basic $79 (250/day), Growth $299 (2,500/day),
Alpha $1,499 (unlimited) - and the Historical API is ALPHA-EXCLUSIVE. Basic and
Growth serve live data only.
The archive was the only thing worth buying from this vendor, so nothing in
budget helps: Basic would spend $79/month to make a once-a-day snapshot 15
seconds fresh instead of 15 minutes. Not subscribing.
The free key keeps one genuine use: a single live call to compare their GEX
against our CBOE-computed number, validating the recorder formula against a
commercial implementation (sign and magnitude only - they sign strikes from
classified tape, we use the standard open-interest assumption).
Recorded the EV argument for future sessions: the recorder banks this history
for free in ~12 months, and on this project base rate most alt-data families
die at the incremental gate. Paying four figures to test GEX a year early is a
poor trade. If revisited, price bulk ARCHIVE sellers, not analytics APIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Free tier is 5 requests per DAY, so the quota is reserved rather than spent:
one request answers whether a historical timestamp returns the whole chain or
one expiration, and that decides whether backfilling 2018->now is ~2,100
requests (GEX screenable now) or infeasible (compare bulk vendors instead).
Deliberately NOT an EA input, unlike FRED/EIA: the EA must never depend on a
paid, rate-limited vendor in its live path. Research-side backfill only, so the
key lives in the gitignored Market Data\altdata\keys.json plus this backup.
Auth is an X-Api-Key HEADER, not a query parameter - worth noting because the
EA's HttpGet currently sends no headers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Option open interest is a snapshot source - no free history exists anywhere -
so the series only accrues from the day recording starts. That is why this
ships BEFORE the redeploy: every day the EA is not running is a day of history
that cannot be recovered later.
Records one row per weekday after 21:00 UTC to gex_{CANONICAL}.csv: net/call/put
dollar GEX per 1% move, call and put OI, the three nearest expiries and the
front expiry code. Feeds NOTHING - wiring a feature that is missing across ~100%
of the training sample would waste input width and hand batch-norm a constant.
It becomes a screening candidate at ~250 rows, gated like every other feature.
Thesis: dealer gamma is a RANGE mechanism (long gamma -> hedging sells rallies
and buys dips, range compresses; short gamma amplifies both ways), and range is
this project's one proven channel.
Verified in situ against the live SPX chain before writing any MQL5: 29,362
contracts, 20,993 with nonzero gamma, 54 expiries, total +90.7 Bn/1% (calls
+305.7, puts -215.0), and 100% of net GEX inside 5% of spot. The CDN publishes
per-contract gamma directly, so no pricing model - and no model risk - enters
the recorded data. Also verified the CDN does NOT gate on User-Agent (the old
"CBOE is UA-gated" note in DESIGN.md was a different CBOE path), so plain
WebRequest reaches it.
Dropped a zero-gamma "flip level" field: the probe returned a crossing above
spot while total GEX was strongly positive, which is incoherent - a static
gamma snapshot cannot give a flip level without repricing. Recording a
plausible-looking wrong number is worse than recording nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The AltData folder in Common\Files gets wiped before every fresh test, and
keys.txt died with it (2026-08-16 silent-FRED incident). The credential now
travels with the EA: FredApiKey input, owner key as default; keys.txt demoted
to a fallback consulted only when the input is blanked. EiaApiKey stored the
same way - reserved, nothing consumes it since the WTI screen came back null.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-08-16 first live fetch looked complete but was not: COT (keyless)
downloaded 1053 reports, then FredKey() hit the absent keys.txt, returned ""
with no log line, UpdateFred bailed, and the SP500_D1.csv rebuild - gated on
all three raw series - never happened. The panel stayed at 0 features with
nothing in the journal explaining why.
FredKey() now logs loudly when keys.txt is missing, and only latches once a
key is actually FOUND: the file is re-read on each hourly-throttled attempt,
so dropping keys.txt in after attach recovers without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VPIN-style toxicity (|imbalance|): null-to-marginal everywhere. But tick
ACTIVITY (count vs 20d mean) clears the family bar on ALL FOUR symbols for
forward range AND survives conditioning on trailing realized range
(SP500 MI|vol 0.025, XAUUSD 0.0088, USDJPY 0.0069, EURUSD 0.0049, all
p<=0.006). On EURUSD it beats the trailing-range positive control itself -
resolving the void-control anomaly: EURUSD D1 range IS predictable, just
not by its own trailing range. USDJPY spread_stress (max/mean) also
family-clean + incremental. Direction: nothing beyond the known SP500
leverage effect. Validates the EA's volume feature block for the RANGE
objective the tuner now optimizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- PeriodMA/MA_Type/PeriodRSI: input -> const seeds (closing the set: every
indicator parameter is now tuner-owned)
- Variables\TunedPeriods.mqh: chart-level tuned-period state. A gated
install writes TunedPeriods_{SYM}_{TF}.cfg; next attach reads it BEFORE
the DB fingerprint and classic-signal config, so classic votes, DB key,
and tuner seeds always describe the same indicators regardless of
classic/AI/hybrid use. Restart-grained adoption by design (no mid-run
handle churn); new periods re-key the signal DB (semantics rule).
- EnableAltData input in AI Input Features (consumption gate only;
collection keeps running); |ALT DB-fingerprint token; opt-out on an
alt-trained model correctly starts fresh via the width compare.
- Defaults: all four classic votes OFF (AI-first; WARRIOR_MARKET_BUILD
branches collapsed with the marketplace pivot), order-flow/Wyckoff NN
features OFF (alt data is the default information diet; toggles stay).
Compiles 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 18 inputs added 2026-08-08 (when the tuner defaulted off and the
values needed an operator path) become compile-time aliases of their own
defaults - same names, zero consumer churn, byte-identical values. The
tuner is now the only path by which these values move: it defaults ON
(the 08-08 off-flip was measured against the direction target's flat
landscape; the objective is now RANGE, which has signal), searches from
the seeds under the Sidak family-wise gate, and persists winners in the
.nnw beside the weights. ADP fingerprint token retired (deviation now
impossible by construction; tuned values were never its job).
Menu shrinks 102 -> 84 inputs. Compiles 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- MI_TUNE_TARGET = MI_TARGET_EXC_RANGE: the coordinate sweep scored
candidates against the barrier label - measured noise - so it climbed a
flat landscape and the gate rightly rejected every winner. It now
selects indicator settings for MI vs realised RANGE (4x null, positive
control), the channel the excursion head consumes these features for.
Winner gate re-tests on the same target. Barrier-label report unchanged.
- AltDataFetch 4014 handling: Alert popup + once-per-session walkthrough
with the two whitelist URLs on their own journal lines (copy-paste
ready); hourly-backoff retry instead of a permanent latch, so the
whitelist fix takes effect without re-attaching.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
12 features (5 EIA petroleum + 3 COT managed-money + VIX/USD + controls)
vs forward 5-bar range and direction on 1,789 D1 bars resampled from the
decoded XTIUSD M1 file, 499 circular-shift perms. No feature clears the
family-wise bar on either target; every EIA fundamental is null even
marginally (best p=0.13). Sample is short (~7y) so a weak effect is not
excluded - but per the gate, no EIA feature ships. The EIA key stays in
keys.txt for future use (longer history / recorded surprise-vs-consensus).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
System\AltDataFetch.mqh: the EA backfills missing alt-data history at
attach and keeps appending forward while deployed - online learning never
depends on an external process. CFTC Socrata API (no key, 2006->now, one
GET per symbol; ES name variants verified, max-OI dedupe) + FRED (VIXCLS/
DTWEXBGS, key from AltData\keys.txt). Identical publication stamps and
fixed a-priori transforms as research/altdata/export.py; rebuilds the
same {SYM}_D1.csv files, so Python and EA interoperate on one format.
OnTimer hook (30-min staleness check, in-memory compares when current;
never in tester - cache files serve there) + AltDataReload() on signals.
Classic-signal removal CANCELLED per user (vote experiment later).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MI|vol column = I(X; target | trailing-range tercile), same circular-shift
null. Range target 499 perms: SP500 vix_chg5 survives conditioning at 0.031
(3x trailing range's own within-tercile residual); VIX LEVEL emerges
conditionally (variance-risk-premium structure); USDJPY COT family survives.
Direction target: SP500 vol/VIX-chg clear marginally (equity leverage
effect) but drop to p~0.05-0.06 conditional = redundant with price vol;
USDJPY/XAUUSD/EURUSD direction null across all alt features.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- fred.py: ALFRED output_type=4 first prints, chunked realtime windows
(2000-vintage cap), unrevised-series fallback (published=observed+1d);
NFCI excluded from features (revised, no vintage archive)
- eia.py: 4 weekly petroleum series on disk (1982->now)
- screen.py: as-of joined alt features vs forward 5-bar range/ATR on D1,
3x3 MI, circular-shift null, family-wise max bar, +/- controls
First readings (199 perms): SP500 vix_chg5 MI 0.047 (1.5x the positive
control) + usd_chg5 clear family bar; USDJPY 4 COT positioning features
clear family bar BEATING the positive control; XAUUSD vix_chg5 tops control
but sub-family-bar; EURUSD positive control FAILS -> table void per the
excursion-target rule, needs investigation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The derived stop/target were quantiles of EVERY bar''s excursions over a
fixed horizon - q75 adverse gave a 2.6-3.5*ATR stop against a ~1.7*ATR
target (user: "looks limiting"). That pooled measurement was correct
when direction was dead (any subset of bars had the same distribution)
and is provably mis-sized now that the gate certifies the label carries
information: the bars the model trades are the labeled bars, and their
excursions differ from the pool.
FractalDirectionLabel now records, for every Buy/Sell-labeled IS bar
during the prebuild, the favourable and adverse travel in ATR units
over exactly the LEG the label points at - entry close through the next
fractal extreme (user request: "from a fractal to the next for maximum
accuracy"). DeriveBarrierGeometry reads the same q75-adverse/q50-
favourable quantiles off that conditional sample instead of the pool,
with a logged fallback to pooled when fewer than the minimum legs
exist. Quantiles kept over averages deliberately: a mean MFE is
dominated by runaway legs and would set an unreachable target.
No circularity: the fractal label does not depend on SL/TP (the barrier
label does - this path must never feed it). Recording stops the moment
geometry is derived and pinned, so pass 2 relabels and later bars
cannot silently re-shape a certified pair.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two user-reported ensemble regressions, one cause each:
- "getting ready is very long": every member ran the full MI diagnostic
suite (headline MI, positive control, alignment, lag profile,
geometry scan + winner test - ~200 permuted draws per line) on
IDENTICAL features and labels, reporting the same numbers four times.
First member runs it, the rest adopt with one log line. Documented
caveat: if the geometry scan ever ADOPTS a winner under its gate
(it never has), the adoption becomes donor-only and the gate must be
revisited.
- "panel not responsive": four members chunks queue back-to-back on the
one chart thread - 4 x 120ms = 480ms worst-case click latency, the
exact regime the 200ms note in Training.mqh already documents as
broken. Ensemble members now use a 30ms chunk budget, restoring solo
UI latency at slightly higher dispatch overhead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All four ensemble members previously wrote their full multi-line panels
to the SAME global label objects - an ensemble chart would flicker
between four stacked panels covering the chart side (user request:
aggregate). Every AI-side SetStatusLabel call site now routes through
CExpertSignalAIBase::PublishStatus - solo charts draw the full panel
exactly as before; an ensemble member claims a slot and contributes
only its HEADLINE to one combined block ("HYBRID ensemble - N models",
then one line per model; the live line leads with the model current
signal). The combined render skips unchanged text and enforces its own
minimum redraw interval so four publishers cannot multiply
ChartRedraw() cost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user is right that no special combination logic is needed: the AI
signals are ordinary voting filters, and the aggregate already has
union semantics - abstaining filters do not dilute the average, so an
ensemble chart trades whenever ANY deployed member clears the vote
threshold and disagreeing members net out. What the ensemble preset
actually adds:
- AI_CHOICE value 4 renamed AI_CONVLSTM (the name says the front-end);
enum VALUES stable, CSignalHYBRID class and State\HYBRID\ folder kept,
so saved configs and trained models keep their identity.
- New AI_HYBRID = 6: enables PAI+CONV+LSTM+CONVLSTM together on one
chart - replaces four separate charts of the same symbol. Each member
trains and self-gates independently; only certified members ever vote.
- |ENS1 fingerprint token on every member, so an ensemble member's
weight files can never collide with a solo model of identical
settings on another chart of the same symbol (the duplicate-chart
guard would otherwise correctly fight over one .nnw).
- Private default AIType = AI_HYBRID: one D1 drop now yields every
topology's gate verdict for that symbol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first-ever family-wise gate pass (SP500 D1 PAI, +10.4pp, p=0.0081)
certifies a win rate measured on HOLD-TO-RESOLUTION outcomes: entry,
then the measured SL or TP decides. Live, three vote-driven exit routes
could close earlier - the averaged-vote close, the AI early-exit route
(both in CheckClosePosition), and CheckReverse - and the fractal
target's vote flips at swing-marker cadence (~3-5 bars), far inside the
barrier's typical travel time (median 7-8 D1 bars to target). The user
observed exactly this: an opposite arrow near an entry, trade cut,
price kept going.
On a fractal-target chart with a live direction model, all three routes
are now suppressed (m_holdToBarrier, set in InitializeSignal, loudly
logged): positions run to their broker SL/TP. Risk guards and trailing
are deliberately untouched - account protection is not signal opinion.
Barrier-target models keep the vote exits: their label is the vote's
own horizon, so for them the routes are semantically consistent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15, ~6 minutes after attach: PAI-8fea (fractal target, 37
features, D1) converged at era 299 and the plateau deploy CLEARED the
family-wise gate for the first time in project history: dir-precision
73.1% vs 63% break-even, +10.4pp on 350 test calls = 4.04 sigma,
p_family = 0.0081.
This script asks the first two hostile questions offline:
- DRIFT: always-long at the same 2.64/1.66 geometry scores 64.1% on the
last 15% of D1 history (66.7% on 30%) - drift alone clears BE by
~1-3pp, but the model is +9pp above ALWAYS-LONG, so the pass is
selection, not drift.
- SWAP EXPOSURE: median 7-8 bars to the long target = ~10 nights of
financing ~ 0.15-0.2% notional vs a ~1.7% target -> a ~1-1.5pp BE
haircut against a +10.4pp margin. Survives.
Remaining before belief: replication on other D1 symbols, and closing
the live-semantics gap (certified wins assume hold-to-barrier; live
exit paths can cut on vote flips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User direction (2026-08-15): back to predicting swing turns, D1 charts,
fractals over ZigZag pivots (their call - balances classes, matches the
reference library target, and a 5-bar fractal confirms 2 bars after its
extreme so labels resolve nearly to the present with no repaint embargo).
- TRAINING_TARGET enum + TrainingTarget input: TARGET_BARRIER (Market
default - existing models keep their meaning and fingerprints) or
TARGET_FRACTAL (private default).
- FractalDirectionLabel (Labels.mqh): per-bar 3-class label = direction
from the bar close to the next confirmed strict 5-bar fractal extreme,
costs charged in the same bid-series convention as the barrier label,
Neutral when the move cannot clear max(2 spreads, 0.10 ATR) or on an
outside bar (both-extreme bars are unorderable within OHLC).
- The barrier walk still runs in full: measured SL/TP geometry, the
expectancy scan, excursion caches and the era gate all keep scoring
what a trade at the EA's own stop/target actually collected - only the
TRAINING label changes. NOT the pre-b4a704d "is this bar the pivot"
form; that target's 31:1 imbalance stays retired.
- Fingerprint token |TGT:FRA1 so switching targets trains a separate
model; AI_META unaffected (guarded setter).
- Private defaults: AIType back to AI_HYBRID (direction topology needed)
+ TrainingTarget=TARGET_FRACTAL = drop-on-D1-chart workflow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User request: NNs predicting pivots (fractals for label density). Run on
the offline stack that demonstrably CAN learn (+2.6pp XAUUSD meta), free
of every historical in-EA training bug: pooled 4-symbol training,
per-symbol norm, scale-free causal features, target = side of current
price the next confirmed Bill Williams fractal lands on, real M1
ask/bid fills, threshold fitted on calib only, pre-registered 2-sigma
net-expectancy gate.
Result: train CE 0.682 (a whisper below the 0.693 coin), and on test no
symbol passes - EURUSD/USDJPY net zero, XAUUSD gross +0.22 pts vs a
larger spread (net -0.30), SP500 net +0.05 +/- 0.45. The gross-positive
tails are index drift plus sub-spread micro-reversion - the tick-flow
decay shape at swing scale. Seventh independent measurement of the same
fact: entry-time direction information is not in these features.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Faithful Python port of ADZigZag (stock MetaQuotes ZigZag 12/5/3,
verbatim rebrand) including the incremental prev_calculated branch, so
the indicator can be replayed bar by bar exactly as it draws live.
SP500 H1, 74,599 bars, fills at real M1 ask/bid:
- FINAL swings: 4,658 legs, mean 50.8 pts = 95 spreads. Perfect
foresight +50.2 pts/leg. The user premise (swings dwarf spread) is
fully confirmed.
- LIVE: 81% of drawn newest-pivots later repaint away entirely
(18,805 of 23,299). Holding the drawn direction at every bar close
grosses +0.38 pts/trade (t=0.8, zero cost charged) out of the
50.8-pt average swing - 0.7% of the line the chart ends up showing.
- LONG +1.66 gross / SHORT -0.90 = the index drift, nothing else;
long net +1.17 pts / 13-bar hold = ~1.5 bp, under one night financing.
The spread subtracts 0.49 pts of a swing that hands over 0.38: cost was
never the obstacle - pivot knowledge is.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
