TWO CHANGES, BOTH RETRAIN-FORCING BY INTENT.
1. SP500 was training alone, and one alt-data column was the reason.
The alt block's width joins the model fingerprint, and the pool reader only
adopts peer rows whose fingerprint and width match. The exporter gives each
instrument the series that apply to it - FX 15 columns, metals/oil 14, SP500
13 - so the fleet ran as three incompatible pools:
EURUSD/USDJPY/USDCAD adopt ~57-60k peer rows each
XAUUSD/XTIUSD adopt 6.4k / 20.3k
SP500 "EVERY peer file was REJECTED, so this chart is
training alone" - 0 rows
SP500 therefore trained on 2279 independent observations against a 600-wide
input with its first layer floored at 16, printing its own "expect
overfitting" warning. It is the one chart with no pool and the worst
capacity ratio in the fleet by a factor of three.
Fresh models now pin ALTDATA_FLEET_COLUMNS - the 12-column intersection -
instead of their own file header. An existing model still adopts its .cfg
pin, so this re-keys nothing that is already trained.
Intersection rather than union: filling an absent series with its median
makes that column constant per instrument, which lets a pooled model
identify the source instrument and stop learning the shared mechanism. It
is also 6 columns narrower. Cost is six columns whose retained information
is UNMEASURED - the keep-screen reports a bitmask nothing has mapped back
to names.
2. The MI keep-screen disabled itself for the whole run on any cold start.
ReportFeatureLabelInformation set m_miReportDone on ENTRY. On a cold start
the label cache is allocated before it is filled, so BuildMiSample finds no
row carrying a resolved label and returns 0 - a sixth exit, and the only
one the 8c1266d instrumentation did not cover, which is why it printed
nothing. observed then stayed -1, the permutation loop never iterated, and
the report emitted "-1.00000 nats over 0 permutations" beside a plausible
"strongest single feature 0.05979" that was a STALE m_miBestColumn from an
earlier scoring call. The first ensemble member propagated the latch to
g_ensembleChartMiReportDone and silenced every member on the chart.
The flag now latches only once a measurement exists. A short sample is
reported as a deferral naming the two numbers that identify it (cached bars
vs bars carrying a resolved label) and retried, up to
MI_REPORT_MAX_ATTEMPTS.
Build tag -> fleet-pool-v1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A DATA-INTEGRITY BUG, pre-existing, surfaced by the clearer failure message in
869cd1b putting two identical timestamps next to each other:
13:42:36.584 (EURUSD) AltRawSave: atomic rename raw_EIA_WPSR.csv.savetmp -> ... failed
13:42:36.584 (XTIUSD) AltRawSave: atomic rename raw_EIA_WPSR.csv.savetmp -> ... failed
Same file, same millisecond, two charts, a third winning the race. That is not
reader/writer contention - it is THREE WRITERS on one destination, and
AtomicWriteBegin derived the staging name from the destination alone:
tmpName = finalName + ".savetmp"
So all three opened the SAME temp with FILE_WRITE and wrote it from offset 0 at
once. The published file could be an interleaved mixture of two charts' output,
and the atomic rename publishes that mixture faithfully - the swap guarantees a
reader never sees a HALF-WRITTEN file, and does nothing about a HALF-CORRECT one.
Alt-data is the exposed case: several charts fetch the same series and write the
same Common file.
Keying the temp on symbol+period makes staging private. The rename stays the only
contended operation, and a rename IS atomic, so a loser now publishes nothing
rather than half of itself. It also makes deferred promotion sound for the first
time: the temp promoted later is THIS chart's complete content, never a fragment
of someone else's.
SharedFileCopy.mqh uses the same shape but its destination is agent/terminal-local
and keyed by symbol+fingerprint, so charts cannot collide there. Left alone.
Note the two bugs are independent and both fixes are real. Confirmed in situ at
13:45:45, on the reader/writer one:
CTrainPoolWriter::Publish: atomic rename TrainPool\USDCAD_16388.bin failed (5004)
Warrior: deferred promotion of TrainPool\USDCAD_16388.bin succeeded - the peer
chart that held it has closed it, and the content written earlier is now live
without rewriting the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
REPLACES the in-line retry from ad4ae58, which was the wrong shape and did not
work. Measured after deploying it:
atomic rename ... failed (error 5004) after 4 attempts
MQL5 exposes no FILE_SHARE_DELETE, so a rename CANNOT succeed while any reader
holds the destination open - it is not a lock that waiting longer wins. The
retry assumed a peer holds a pool file for "tens of ms"; USDCAD_16388.bin is
134 MB and a peer reading it holds the handle for SECONDS. The loop lost every
time and bought nothing but 75ms of tick latency on the failure path.
The content is already written and correct - only the SWAP is blocked. So try the
rename once, and on failure remember the temp and promote it from OnTimer, where
I/O belongs. Once the reader closes, a single FileMove lands it. That beats the
old fallback of waiting for the next full publish, which rewrites all 134 MB and
may be an era away.
* pending list is bounded (8) and deduplicated - AtomicWriteBegin reuses one
temp name per file, so a second failure for the same file must not take a
second slot. A full list falls back to the previous next-publish behaviour.
* a successful write FORGETS any queued promotion for that name, so a stale
temp can never overwrite fresher content.
* a vanished temp (a later publish succeeded outright) is dropped, not retried.
* a landed promotion is LOGGED. Silence is what made me misread the last
attempt as working when there had simply been no contention in the window.
Compiled clean; NOT yet run - and note that verification needs a collision to
occur, which happened ~27 times across a whole day. Absence of the message in any
one window is not evidence either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
THE EXIT KNOB. Exit_On_Reversal_Vote (default false) replaces the deleted
Signal_ThresholdClose with one boolean: false pins the close threshold to an
arithmetically unreachable 101, true pins it to the SAME threshold the entry
uses - the seed at first, then the derived value, republished together whenever
it moves. A second threshold was always redundant; "the bot now says the other
way" is one question.
It also arms CExpertSignalCustom::m_holdToBarrier, which was DEAD CODE:
HoldToBarrier(bool) had no caller anywhere in the build, so the flag had been
permanently false and the disabled close threshold was carrying the whole
hold-to-barrier policy alone. Both halves now move together.
Default stays false because the reason is statistical: the gate certifies
P(label agrees | vote fired) against a label that runs to the barrier, so an
early close trades something never measured. Turning it on is a different
strategy, not a tightening of this one.
THE PIN. The live threshold now moves only when an era's weights become the
checkpoint, and freezes once g_ensDeployApproved. Every era still derives its own
rung - that is how the best one is found - but the rung that TRADES belongs to
the checkpoint, exactly as the weights do. Two reasons, one measured and one
structural: the per-era rung moves on 6-34% of steps (the live run flapped
SP500 15 -> 10 -> 15 within a minute of starting), and without the pin a later
era's rung could end up applied to an earlier era's deployed model. A ladder
restart releases the pin, since clearing the checkpoint clears what it pinned.
The era line now prints the rung its own numbers came from, so it stays honest
when that differs from the pinned one.
THE ATOMIC RENAME retried zero times. Six charts share the TrainPool and AltData
directories, so a publish regularly lands while a peer chart holds the
destination open and FileMove returns 5004 - 27 times in one day on the live
fleet. Nothing was lost (the temp keeps the new content, the old file stays
intact) but the row did not update until the next publish. Now four attempts at
25ms, on the FAILURE PATH ONLY - a successful rename never sleeps - and skipped
in the tester, where the contention cannot happen and Sleep would distort a pass.
A rescued retry is logged, so worsening contention is visible.
Retrain-neutral. Compiled clean; NOT yet run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Added bulk read/write methods for feature caches in IFeaturesView and its implementations to optimize performance.
- Introduced LabelCacheInvalidateAll method to manage label cache invalidation alongside feature cache.
- Implemented PooledIndependentBars method in topology interfaces to account for additional independent observations.
- Enhanced risk budget management with throttling for peak-equity updates to reduce unnecessary file operations.
- Improved error handling and logging for ATR trailing stops to ensure better visibility of issues.
- Updated alt-data handling to prevent unnecessary operations during testing and optimization phases.
Four reported symptoms, three of them one root cause: the ensemble's
certified record was session-scoped and written ONLY at pass-3
completion. A deployed ensemble runs no further eras, so every restart
lost the aggregate win rate, the aggregate panel line and the overlay
snapshots - and could never regenerate them, because regeneration only
happens at an era end that will never come.
THE SELF-CONTRADICTION. Member rows read "Live - learning from new bars"
(from m_trainingComplete) while the line under them read "training, not
tradable yet" (from `prospective`, which means "this number came from
ProspectiveVote() rather than a real Direction() call" - what happens on
any bar where every member abstains, and which says nothing whatever
about training state). Both now resolve through one predicate:
WarriorChartModelsDeployed(), fed by members publishing their own state
on the same slot and cadence as their vote. Adds a third verdict word,
"armed (bar still open)", for a deployed model on a prospective
recompute - the case that used to claim it was training.
DEPLOYED PANEL. Once every published model is converged the per-member
rows are dropped: what ships is the aggregate vote win rate, the live
vote, and the verdict. While training the rows stay - they are the only
way a collapsed or lagging member is visible, since a collapsed member
abstains and so is invisible in the aggregate by construction.
ACCURACY NOW RESPECTS THE ENTRY THRESHOLD. The panel's "precision 65%"
came from m_cumOosCorrect/m_cumOosTotal, which counts every bar a model
called Buy or Sell - threshold-blind, and per-model rather than
per-vote. The correct number already existed (votePrecPct: bars where
|vote| >= threshold and the direction policy allows) and is now what the
panel shows, with the threshold named in the text because the number is
meaningless without it.
VOTE ARROWS PERSIST. With DrawUnfilteredSignals off - the default - the
chart shows SIG_VOTE_PREFIX arrows, and nothing saved them:
CChartUI's .arrows sidecar is member-scoped and never saw that layer.
New CVoteArrowStore mirrors them to a chart-keyed sidecar and restores
them progressively at init, on the same budgeted non-blocking path.
The header stores the open/close thresholds; a mismatch on load DISCARDS
the arrows rather than redrawing a picture of a strategy no longer
configured - stale arrows are worse than none, because none is visibly
empty and stale is confidently wrong.
Also: .stats bumped to WST7 carrying the ensemble record (guarded on
threshold match, most-complete-copy-wins), and the loader's version
tests collapsed from an or-chain to ">=" - the magics are ASCII 'WST1'..
'WST7' so they are already ordered, and a missed arm in that chain reads
the NEXT field's bytes into this one, which fails as plausible numbers
rather than as an error.
Compile-verified in _claude_stage: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three related changes, all aimed at work being repeated at a frequency
nobody chose.
1. OnDeinit gets a tester/optimizer fast path.
Everything in the live teardown exists to leave a CHART clean and a live
model's state on disk. An optimization agent has neither. It was still
running, on EVERY pass: a per-signal arrow-sidecar WRITE
(ShutdownChartCleanup -> PersistAndClearChartSignals) plus two full
chart-object scans plus a ChartRedraw. At optimization scale that is
hundreds of thousands of pointless file writes per agent, against a
~4,500 ms budget MetaTrader force-terminates on - the shape of thing
that stalls an agent rather than failing it.
The fast path keeps MarkShutdown() and FlushTrainRun() (so a killed pass
never leaves a half-written era) and still calls dbm.Deinit() and
Expert.Deinit() - leaking the signal tree or a handle across passes is
its own way to accumulate into a stall. The two now-unreachable
!isTesterRun guards further down are folded away.
2. All four tester handlers are present and documented by WHERE THEY RUN.
OnTesterInit/OnTesterPass/OnTesterDeinit run in the CONTROLLING TERMINAL
once per session; only OnTester runs on the agent, per pass. OnTesterPass
was missing entirely - added empty and deliberately so: it only fires for
passes that shipped FrameAdd() data, which this EA never sends, and
reading frames there would put per-pass work on the terminal's critical
path. Declared so that adding frame-sending later fails loudly instead of
silently dropping every frame.
3. Expert_EveryTick is now actually enforced.
It was passed to Expert.Init() and only ever reached StartIndex() - which
bar a signal READS. The whole pipeline still ran on every quote. It now
gates m_signal.SetDirection() in CExpertCustom::Processing(): that call
drives Direction(), which is a TRANSACTION (NN forward passes, DB rows,
chart arrows, one-shot vote state), and re-running it on every tick of a
4-hour bar repeats all of it.
Scoped deliberately. Everything after that line still runs per tick -
CheckReverse/CheckClose/CheckTrailingStop and pending-order maintenance
are risk management, and a stop that only trails at bar boundaries is a
different strategy, not a faster one. The scheduled close-all in OnTick()
matches a +-1 MINUTE window, so bar-gating it on H4 would step straight
over the thing 100% of label timeouts already resolve against.
g_riskBudget.Update() also stays at quote frequency, by design.
System/NewBar.mqh becomes CNewBar, a class. The free function it replaced
had zero callers and kept its watermark in a `static`: ONE watermark
shared by every caller, so the first caller each tick consumed the
transition and every other caller was told "no new bar" for a bar that
had just opened. Per-instance state fixes that; first observation counts
as new, so a fresh attach acts immediately instead of idling up to a full
bar.
Compile-verified in _claude_stage: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two chart-display fixes reported after watching a converged 4-model
ensemble: the ensemble panel's trailing "(era 69, 4 models,
DEPLOYING)" was frozen at whatever era the ensemble happened to
deploy on, and the separate top-right HUD (one line per model, raw
B/S/N + weight + era + error) was clutter once the vote itself is
what matters.
Root cause of the freeze: g_ensembleVoteLine is written once per era,
at pass-3 completion. A deployed/converged ensemble runs no further
eras (ScheduleTrainingIfNeeded's trainingComplete branch skips
Train() entirely), so that line could never update again - the era
count and "DEPLOYING" marker were permanent set-dressing from the
deploying era, not a live reading.
- EnsembleScoreCombinedVote() drops the era/DEPLOYING tail once
g_ensDeployApproved - nothing left there worth freezing.
- UpdateVoteReadout() (the aggregate "VOTE ..." line, previously its
own top-right chart object) now writes g_liveVoteLine instead of
drawing anything. Both status-label builders - PublishEnsembleStatus
for the ensemble panel, PublishStatus's choke point for the solo
panel - append it as one line, refreshed every tick/timer exactly
as the old HUD was, so the live vote replaces the frozen era tail
in the same visual slot.
- RefreshVoteReadout()'s per-member loop (DisplayHudLine, one
ObjectLabel per model) is deleted outright rather than folded in -
the operator asked for the aggregate only, "without telling me each
individual network".
Follow-on dead-code removal, since DisplayHudLine was the only
caller: the DispProb/DispSignal/MetaGateArmedNow/MetaHasScore/
MetaLastP/MetaLastBe/MetaApproved/MetaVetoed leg of IChartView (and
its AIBaseChartView/AIBaseChartViewImpl/ExpertSignalAIBase forwards)
had no other reader. The underlying data survives untouched -
m_metaTelemetry is still populated live by SignalMETA.mqh,
m_dispSignal still feeds ProspectiveVote - only the chart-view
forwarding that existed solely to reach the deleted HUD is gone.
Compile: 0 errors, 0 warnings (stage).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FeatureScreen.mqh's MI/permutation-null diagnostics (mean/best-col
report, excursion report, lag-profile family-wise test, barrier-
geometry scan) and AutoTune.mqh's TuneIndicatorsByFilter install gate
each spelled out the add-one-smoothed Monte-Carlo p-value
(1+atLeast)/(draws+1) independently. Added PermutationPValue(atLeast,
draws) to System/BinomialStats.mqh (returns 1.0 for draws<=0, matching
every existing call site's own guard) and replaced all six inline
expressions with a call to it. Pure arithmetic substitution, no
control-flow change.
CAltDataFetch (System/AltDataFetch.mqh) mixed six unrelated concerns in one
1406-line class. This is plain composition, not the Expert/AIBase view+adapter
pattern - CAltDataFetch has no single-inheritance parent forcing an adapter,
same shape as Database/DatabaseManager.mqh composing its four Database*
managers.
Extracted, grep-confirmed zero external callers of any moved method (only
Warrior_EA.mq5's already-public Update/NeedsMapping/CatalogCount/CatalogName/
CatalogLabel/SaveUserMapping surface, unchanged):
- CAltDataHttpClient (System/AltDataHttpClient.mqh): HttpGet/HostOf/MaskUrl/
BlockedIndex (4014 per-host backoff), JsonField, UrlEncodePart. Owns the
m_blockedHost/m_blockedUntil/m_urlAlerted state - STATEFUL, moved verbatim.
- CAltDataCatalog (System/AltDataCatalog.mqh): the SAltSymbolSpec catalog
(AddSpec/BuildCatalog/AliasMatches/FindSpec) plus symbol_map.cfg user-mapping
persistence (LoadUserMap/SaveUserMapping) and the public
NeedsMapping/CatalogCount/CatalogName/CatalogLabel surface. Owns
m_specs/m_userFrom/m_userTo/m_userMapLoaded - STATEFUL, moved verbatim. Added
one new Spec(i) getter (by value - 4 strings + 1 double, cheap) so
CAltDataFetch::Update()/RebuildFeatures() can read a resolved catalog row
without reaching into the collaborator's array.
- LoadRaw/SaveRaw (raw-series CSV <-> file mapping) had zero member-state
dependency - turned into free functions AltRawLoad/AltRawSave, matching the
file's own existing AltSeriesAppend precedent, instead of a needless class.
CAltDataFetch itself keeps three concerns as a deliberate partial, same
judgment already applied to Topology's boot sequence / Features' shared-
indicator lifecycle elsewhere in this campaign: the four per-source fetch
pipelines (UpdateFred/UpdateCot/UpdateEia/UpdateGex, including FredKey/EiaKey/
LoadCommonKey/ShouldAttemptFetch/the GEX CBOE helpers) and the feature-CSV
builder (RebuildFeatures/FeatureValue/RollingPctRank/AsOf) both read/write the
9 SAltRawSeries caches kept resident on the orchestrator between timer ticks -
splitting them out would mean either relocating that cache's ownership or a
9-13 parameter signature per method, a larger design decision better made as
its own pass rather than forced through unattended given the finding's own
"high risk" estimate.
Every moved method body is copied verbatim (statement-by-statement diffed
against git show HEAD~1:System/AltDataFetch.mqh) with only the mechanical
substitution HttpGet/JsonField/UrlEncodePart -> m_http.*,
LoadRaw/SaveRaw -> AltRawLoad/AltRawSave, FindSpec -> m_catalog.FindSpec, and
m_specs[si].X -> spec.X (spec = m_catalog.Spec(si), resolved once per Update()
call instead of re-indexing). ALTFETCH_TIMEOUT_MS/ALTFETCH_GEX_TIMEOUT_MS
macros moved into AltDataHttpClient.mqh (their logical owner); ALTFETCH_DIR
stays in AltDataFetch.mqh, defined before both new #includes since
CAltDataCatalog's ctor path (symbol_map.cfg) and CAltDataHttpClient's HttpGet
default param reference it.
Self-compiled 0 errors, 0 warnings (_claude_stage, MetaEditor64 /compile),
twice - once before and once after a stale doc-comment fix (a leftover
"same reasoning as SaveRaw" mention updated to AltRawSave).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CopyFileWithRetry (System/SharedFileCopy.mqh) and CModelPersistence::
LoadNetWithRetry independently implemented the identical 5-attempt
Sleep-doubled-and-capped retry shape around a different single
operation, with a comment on the latter pointing at the former as the
"same reasoning" instead of sharing code. Added System/RetryWithBackoff.mqh:
an IRetryableOp interface (one bool TryOnce(bool quiet) method, MQL5 has
no closures/function pointers that bind per-call-site arguments) plus the
RetryWithBackoff(op, attempts, initialDelayMs, delayCapMs) loop. Each call
site now defines a tiny local operand class (CCopySharedFileOp,
CLoadNetOnceOp) and keeps its own tuning constants (150ms/1000ms cap vs
200ms/2000ms cap) unchanged - pure mechanical relocation, no behavior
change. CModelPersistence stays stateless (grep-verified in the prior
Persistence extraction): CLoadNetOnceOp is a separate local class, not a
new member on CModelPersistence itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SaveRaw() and RebuildFeatures() hand-rolled the same FileOpen(tmp)->write
->FileClose->FileMove(FILE_REWRITE) swap AtomicWriteBegin/AtomicWriteEnd
already generalize. AtomicWriteBegin hardcoded FILE_BIN (every prior
caller wrote binary payloads); these two write plain ANSI CSV lines via
FileWriteString, so AtomicWriteBegin now takes an optional modeFlags
param (default FILE_BIN, unchanged for the 4 existing callers) and the
two AltDataFetch sites pass FILE_TXT|FILE_ANSI.
SetStatusLabel's trailing-line trim and ClearStatusLabel's full wipe
had byte-identical bg/txt ObjectFind+ObjectDelete loop bodies, differing
only in start bound. Extract DeleteStatusLines(fromIdx, toIdx); both
call sites keep their own counter reset/redraw. Pure UI, no behavior
change.
CExpertCustom and CExpertSignalCustom each defined their own
stops-level order-type decision against the same TCStopsLevel()
helper, identical except CExpertSignalCustom's guard skipped the
EMPTY_VALUE check (a huge finite double would fall through into the
ask/bid comparison and misclassify as a pending order instead of a
market order). Moved the logic into TCResolveOrderType() in
System/TradeChecks.mqh, keeping the more defensive guard; both
classes now delegate. ask/bid are still passed in from each class's
own m_symbol so routing keeps using whatever RefreshRates() snapshot
that class already had - only the duplicated arithmetic moved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TunedPeriodsFileName() inlined the same 9-line StringReplace sanitizer
AltData.mqh already defines once as AltDataFileSymbol(), whose own
comment said it was meant for exactly this. Gave AltData.mqh a proper
include guard (it had none) and included it directly from
TunedPeriods.mqh so the call is safe regardless of include order.
Compile-verified 0 errors/0 warnings.
CCrossAssetPanel::Warm() and WarmBlocking() opened with the identical
pinned-vs-discovered pair-set resolution block. Extracted a private
ResolvePairSet(pairs) returning whether a usable set was resolved; each
caller keeps its own early-return semantics (void vs bool-true) around
the call. Pure mechanical relocation, no arithmetic/behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Added private CAltDataFetch::ShouldAttemptFetch(s, staleDays, throttleSlot, now)
combining the staleness check (parameterized by staleDays) and the byte-for-byte
identical throttle-check-and-stamp that opened UpdateFred/UpdateCot/UpdateEia.
Each call site drops from 5-6 lines to one guard call; UpdateGex is untouched
(its day/hour gate is unrelated). Same dedup doctrine as c214d3e (FredKey/EiaKey)
and defab3b (identifier-validation guard), just one level up the call chain.
FredKey() and EiaKey() implemented the same contract (input, then cache,
then a keys.txt "<prefix>=" line, cache + warn once) differing only in
which input/cache/warned-flag/prefix/messages they used. Both are now
one-line callers of a shared private helper taking those as parameters
(cache and warned-flag by reference). Pure relocation, no behavior
change - verified the warning text is character-identical at both call
sites and that the one flag-timing difference (LoadCommonKey only sets
warnedFlag inside the branch that actually prints, matching EiaKey's
original shape, vs FredKey's unconditional set) is unobservable since
a found key always short-circuits on the cache check before the flag
is ever read again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Neither ever touched a CExpertSignalAIBase member - both take everything as
parameters. Moved to System\SharedFileCopy.mqh (free functions, same doctrine as
TradeChecks.mqh's TC*), matching what they actually are instead of carrying them as
methods on a class they don't depend on. Topology.mqh's call sites are unchanged -
unqualified calls from within a class method resolve to the free function exactly
the same way. Compiled clean.
Correction to project_oop_module_pattern's Persistence(696) note: LoadNetWithRetry
is NOT a third pure utility alongside these two - it touches Net/dError/dUndefine/
dForecast/dtStudied/m_activeFileName/m_activeFileCommon/m_eraCount/m_trainingComplete.
The rest of Persistence.mqh (EnforceTopologyContract, Save/LoadModelStats,
ValidateCpuInference, Save/LoadAndCompareTopologyConfiguration, ReadAltDataPinFromCfg)
is heavily coupled to signal state - a real view+adapter extraction on the scale of
ChartUI's, not attempted here.
Two literal duplications the scan found, both of the kind where a divergence is
silent:
- Training.mqh stashed the era-loop resume context at FOUR yield points, seven
identical assignments each (pass 1 differing only in i-1). A field missed at
one of them resumes the next chunk against a different era than the one that
yielded, and nothing reports it until the numbers drift. Now StashEraResume().
- AltDataFetch grew its five parallel arrays inline in three places. They are
one record split across five buffers, so a resize missed on any one reads out
of range on the NEXT append, not at the site of the mistake. Now
AltSeriesAppend(), which returns the new index and zero-fills; callers set
only the columns their source has.
Braces balance across every in-scope file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The //| box blocks were excluded from 0b06f8e and 5efdb48 and were what
remained: 160 of them ran to 10+ lines, the longest to 88. Compressed to their
leading topic sentences - 5 lines for a function header, 8 for a file header -
keeping the box format and the standard MQL5 name/author lines verbatim.
Verified at the BYTE level this time, across every in-scope file: the list of
non-comment lines is byte-identical to HEAD and braces balance. The first check
compared a locale-decoded 'git show' against a UTF-8 read and flagged 25 files
that had not changed at all - every BOM and every non-ASCII line mismatched.
47,696 -> 40,665 lines in scope; comment share 38% -> 26%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same pass as 0b06f8e, applied file by file: comment runs of 4+ lines compressed
to their leading topic sentences, capped at 4 lines, whole sentences only.
Warning sentences (NEVER / MUST / trap / would-have) survive the budget.
Every file was checked the same way before committing: the list of non-comment
lines is byte-identical to HEAD, and braces balance. No code was touched.
Panel/, Enumerations/ and the already-terse System headers needed little or
nothing - PooledGate, TradeChecks, BinomialStats and Random came through with
no blocks over the threshold at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Features() binary-searches m_rowTime (StringToTime of a naive "2026.08.21" = 00:00) against
m_Time.GetData(idx), which is BROKER time. A row dated D carries values public from D 00:00 UTC,
so every bar in [D 00:00 broker, D 00:00 UTC) - the first H4 bar of each broker day - reads that
row 2-3 hours before it was public.
Not a leak today: every collector stamps with a conservative buffer on top of the real release,
and the overshoot fits inside all of them. Tightest is the FRED daily series - VIX prints 16:15 ET
and is stamped D+1 00:00 UTC, leaving >=1h45 in the worst DST alignment. COT (Friday 15:30 ET ->
Saturday 00:00 UTC) has no bars to read it before Monday at all; EIA has ~8h.
Recording it because nothing in the file said so, and the safety lives in a number nobody would
think to check before shortening a stamp.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MQL5's MathRand() is the 15-bit MSVC LCG - 32768 distinct values and
the lattice structure that shape of generator has. Two places here
actually lean on randomness and both were hurt by it:
WEIGHT INIT. Six He/LeCun-uniform sites drew
((MathRand()+1)/32768.0 - 0.5) * 2 * scale, so a first dense layer of
~250k weights had only 32768 possible values and thousands of
connections started byte-identical. Breaking that symmetry is the whole
job of random init.
SHUFFLING. ShuffleRandomIndex() already had to splice TWO MathRand()
draws to reach 30 bits, and its own comment documented the residual
modulo bias it still carried. HQRndUniformI() is rejection-sampled and
exactly uniform, so the splice and the bias note both go.
CHighQualityRand is L'Ecuyer's combined multiplicative congruential
generator - two differenced streams, 31-bit output, period ~2.3e18 -
and it ships with the terminal.
AND A BUG THE MIGRATION EXPOSED. The three MathSrand(GetTickCount())
calls sit immediately before "build a fresh topology", once per model.
GetTickCount() steps in ~15.6 ms on Windows and an ensemble builds every
member inside one OnInit, so members could be handed the SAME seed and
draw the SAME weights wherever their shapes coincide - and members that
start identical are not an ensemble. WarriorRandSeed() takes a salt (the
model id) plus a never-reset call counter, so a collision is impossible
rather than merely unlikely, while the tick keeps the run itself
genuinely unrepeatable the way those call sites asked for.
Seeds are masked positive rather than trusted: HQRndSeed computes
s % (M-1) + 1 and MQL5's % keeps the sign, so a negative seed leaves the
generator in a state its own assertions reject. GetTickCount() is a uint
and goes negative as an int after ~24 days of uptime - a fault that
would surface as "training is broken" on a long-running terminal and
nowhere else.
The indicator tuner's 52 draws move across too: its random search is
where sample quality earns its keep.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Beta-prior arithmetic that turns counts into a ranking weight was
written twice, term for term: WinRateFromCounts() for the classic
pattern ladders and RankTiersFromOos() for the AI confidence tiers.
Same formula, two transcriptions, and the same class of duplication the
binomial SE consolidation removed a few commits ago.
ShrunkRatePct() in System\BinomialStats.mqh is now the only copy. The
two call sites keep what genuinely differs - the classic path passes RAW
trade counts with a prior of MIN_TRADES_FOR_WIN_RATE, the AI path passes
OVERLAP-CORRECTED effective counts with TIER_PRIOR_EFF_N, which is far
smaller precisely because effective counts are - and that contract is
now stated once, in the function, instead of being implied by two
comments that could drift apart.
Also fixes a difference the consolidation exposed: with an empty sample
and a prior present, the posterior mean IS the prior, and returning 0
there would have handed a tier a vote weight of zero on no evidence.
The AI path could reach that (effN can round to 0 when labels overlap
heavily); the classic path cannot, since it returns NO_DATA_WIN_RATE
first.
Corrects a stale note of my own in passing: this ranking was recorded as
a "raw win rate behind a MIN_TRADES cutoff heuristic". It is not, and
has not been for some time - it is already a proper empirical-Bayes
estimator with a per-filter pooled prior. Replacing it with a
significance test, as that note implied, would have swapped the
estimator the weight needs for a gate answering a different question.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The codebase had THREE conventions for the same statistic. AltData took a
true median; the barrier horizon and the derived input window took the
upper of the two middle values; the MI terciles and the barrier stop
ladder used nearest-rank indexing. All four now go through MathMedian /
MathQuantile, which is R's type 7 and the library's one answer.
System\AltData.mqh column median -> MathMedian (exact, no change)
AIBase\Labels.mqh swing median -> MathMedian
leg-range med -> MathMedian
stop ladder -> MathQuantile, read in one call
AIBase\Topology.mqh window median -> MathMedian
AIBase\AutoTune.mqh MI terciles -> MathQuantile + MathMin/MathMax
Signals\SignalSessionFilter DST last Sunday-> CDateTime::DaysInMonth()
gaps[]/legs[] change from int to double so MathMedian can read them; the
values are bar counts either way.
VALUES MOVE. Even-sample medians shift by half a bin and the quantile
reads interpolate, so the barrier geometry and the derived input window
can land on different rungs - re-keying fingerprints and forcing a
retrain. Accepted deliberately: stdlib consistency was the ask, and three
private conventions for one statistic is what it buys out.
Two YAGNI finds fell out of the ladder rewrite. MathQuantile sorts its own
copy, so DeriveBarrierGeometry no longer sorts up[]/dn[] in place - which
means upUnsorted[], a full array copy kept only to undo that sort, is
gone. ArraySort(up) had no consumer needing order at all; it was pure
work. The library call also gets a failure guard the hand-rolled indexing
never needed but the ladder read does.
Verified while here: Math\Stat\Math.mqh's MathAbs/MathMax/MathSqrt/MathPow
and friends are ARRAY overloads, not scalar redefinitions, so pulling it
into the translation unit shadows no builtin.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The formula p(1-p)/n was transcribed nine times across six files - the two
deploy gates, the two edge floors, the collapse recall floor, the barrier
rung ladder, the inference bin SE, the pooled inverse-variance weights and
both detectability reports. System\BinomialStats.mqh now holds it once, as
free functions with no class dependency, so the god-class declaration does
not grow to host pure math.
BinomialVar(p, n) p(1-p)/n
BinomialSEPct(p, n) 100*sqrt(p(1-p)/n)
BinomialCallsForEdge(p, edge, sigmas) the same, solved for n
NormalUpperTailQ(z) Q(z), via Math\Stat\Normal.mqh
SidakFamilyP(z, N) 1-(1-Q(z))^N
Value-preserving by construction: rates go in as probabilities so no call
site gained a *100/100 round-trip, and BinomialSEPct is written through
BinomialVar so the multiply order is the one it replaced. Every degenerate
guard each site carried (p<=0, p>=1, n<=0) now lives in one place and
returns the 0 those sites already treated as "no bar to clear".
CExpertSignalAIBase::NormalUpperTail is gone; NormalUpperTailQ replaces it.
What consolidating SURFACED, and is deliberately NOT changed here: the two
Sidak selection gates compute their SE on the RAW call count, while every
other SE in the project deflates by EffectiveSampleSize() for triple-
barrier label overlap. That makes them the most permissive test in the
codebase, by ~sqrt(mean label lifespan). Correcting it tightens a live
deploy bar, which is a policy decision, not a refactor - flagged in the
code at both sites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six checks each fetched bid/ask and rejected a non-positive pair with their
own wording. TCLiveQuote() now owns that rule, so what counts as a usable
quote is defined once and every rejection reads the same way. The one
message that said only "no live quote for <symbol>" now names what it was
about to do, like the other five.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate's NormalUpperTail was a hand-rolled Abramowitz & Stegun 26.2.17
approximation. Its own comment gave the reason - "drags a chain of headers
behind it" - and that turned out to be one file: Math\Stat\Normal.mqh
includes only Math.mqh, which includes nothing. Swapped for Cody's rational
approximation in the library (~18 significant digits vs |error| < 7.5e-8).
No past verdict changes: at the z the gate operates on, the difference is
orders of magnitude below DEPLOY_FAMILY_WISE_ALPHA.
Adopting it needed the four bare macros in AI\Network.mqh gone first.
"#define b1 AdamBeta1" collides with an identifier in Math.mqh, so the
include would have macro-expanded the library's own local and failed to
compile - the same landmine that made the original author rename the
approximation's coefficients to ntB1..ntB5 rather than use the reference's
b1..b5. lr, b2 and momentum are the same class of hazard: single-token
global macros in a 52k-line codebase. All four now resolve to the input
names they always aliased, which is a pure textual identity - verified zero
bare occurrences remain.
Also:
- SelectionSort over the buffered signals was O(n^2) with an O(n^2) count of
StructToTime calls, because the comparison rebuilt both datetimes from the
six int date fields every time. Now materialises the keys once and does an
insertion sort; ArraySort cannot permute a struct array. IsEarlier goes
with it, MakeDateTime becomes SignalTime.
- Seven FileOpen sites lacked FILE_SHARE_READ|FILE_SHARE_WRITE, including
AtomicWriteBegin, which stages every model save. All 43 sites now carry
them - an exclusive open fails outright when another process holds the
path, which here has meant a silently skipped save.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified dead by grep across all first-party sources (references/, Scripts/,
research/ excluded): EraCount, HiddenLayersCount, LstmHiddenSize, ConvFilterCount,
HistoryBars and MinTrainYear setters, PendingBatchSamples, getPrevOutIndex,
BaseCurrency, QuoteCurrency, CurrencyCount, IsLoaded, LastFiredDirection,
DBConfidence, SpecIndex, and the conv Step/WindowOut shape accessors. Every
backing member stays - each is still read internally and several are pinned by
the positional .cfg layout - so this removes surface, not behaviour.
Two comments were asserting the opposite of the code and are now true: the
"No setter: the taper's endpoints are derived" note was directly above three
setters, and the conv shape block claimed EnforceTopologyContract reads all
three accessors when CNet::FirstConvWindow only ever calls Window().
CTrailingATR::CheckTrailingStopLong/Short were byte-identical but for Bid vs Ask
and the isLong flag; both now delegate to one CheckTrailingStop body.
Deliberately NOT removed: the fractal-target branch (TrainTargetFractal,
IsFractalTarget and their label machinery). It reads as dead because the
TrainingTarget input was withdrawn, but Warrior_EA.mq5:836 documents it as a
parked option with a three-line restore path - that is a product call, not a
refactor.
Not compiled - MetaEditor compile pending.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
User design (2026-08-19): 'remove the enum menu that selects neural networks... individual
inputs for every NN just like classic signals... the META NN should be integrated into the
voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.'
- AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/
Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to
any subset because the consensus divisor is the enabled capable weight. Two or more
enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so
existing weight files keep loading); one = the old solo preset; none = classic-only.
- Use_MetaLabeling un-couples META from the direction NNs (the old selector made them
mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry
(shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR;
pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and
vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly.
- COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the
meta target - solo-only until today, a trained META would otherwise sit in the consensus
divisor as a permanent abstainer and shrink every vote by its module weight.
- CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same
g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own
time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K
unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class,
calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not
model the veto - the standing solo-gate caveat, documented at the input.
- DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType;
DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member
subsets get 100+bitmask, outside the legacy range) so no existing database re-keys.
filterID becomes the enabled roster via one EnabledNNSummary().
- HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the
armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not
only when an entry happens to be proposed.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two things, one of which was a compile error.
1. ExitPolicy() was declared in the protected block but is pushed in from
Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters.
2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured
from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot
begin until whatever is in flight returns - so a scan still running after
_StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge
never gets its turn.
New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress.
Deliberately NOT m_trainingStopRequested: that latches, and a latched flag
would permanently disable scans that must run again on the next Start.
Guarded, longest first:
- TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings
on the way out (best[] is mutated in place; the tuner otherwise keeps the
last trial's parameters, which nothing chose).
- ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so
m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar
read the last candidate's multiples as the configured geometry).
- ReportFeatureLabelInformation / ReportExcursionInformation / lag profile -
nulls ABANDON rather than truncate: fewer draws is not a smaller null, it
is a wrong one, and p shifts toward significance. m_dirEvidence staying
false is the safe direction.
- SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line
is dropped instead of latching a partial expectancy as the run's only report.
- ReportGeometryExpectancyScan - per ladder rung.
- HttpGet - one choke point for up to a dozen blocking WebRequests per
first-pass Update(). An in-flight request cannot be cancelled; refusing to
start another is the whole remedy.
- PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain -
entry points, so a queued event cannot open an era during teardown.
TuneIndicatorsAndTrain's guard is the first statement, ahead of the
m_tuneFilterDone / g_ensembleChartTuneDone latches.
- OnTick / OnTimer / OnChartEvent.
Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3
yield on a 120 ms budget); the warm-up scans did not, and they are the longest
uninterruptible stretches the EA has.
StopTraining() is unchanged: the operator's Stop still finalises synchronously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
THE ANSWER, off the instrumentation added in be39674, first run:
ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable
indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ...
Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982
MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not
short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem;
the previous session's five theories were all answering the wrong question.
And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on
that same chart at 15:24:18 and went on to train normally (feature health, 51
features, excursion head) reading the same indicator. Only ConvLSTM's handle -
the last member constructed - was dead. WHY is still not established. All four
members request ADMovingAverage with identical params, so MT5 hands them the SAME
refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over
exactly that shared handle; that is the obvious suspect and it is NOT yet proven,
so this commit does not act on it.
1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths.
"MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982
(h12)" says it is the SAME handle and someone released it; "(h-1)" says it was
never created. That is the difference between a refcount bug and a creation
failure and it is one field. This is the measurement the shared-handle suspicion
needs before anyone acts on it.
2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT.
A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles()
re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD
indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does
NOT release first: -1 means the terminal no longer knows the handle, so there is
nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could
decrement whatever now owns that number. 30s cooldown, because every ServableBars()
consumer reaches it including live inference on every tick. The feature cache is
dropped with it, and the log names before/after depths.
Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars
against a buffer that answers EMPTY_VALUE at every index - then discarding the era
and doing it again - is not a recovery.
3. THE SWEEP NOW HOLDS ON A DEAD HANDLE.
ServableBars() keeps answering `want` (its contract; live inference and online
learning have their own refusal paths and a 0 there reads as "no history at all").
SettledBars() - the training sweep's entry, the one caller that can afford to wait -
returns 0 instead, so Train() holds and reports rather than burning a full-history
pass it is guaranteed to throw away. A recreated handle is cold, so it primes
through the existing settle path on the next call. If the repair fails the member
holds indefinitely and says so every minute, and be39674's barrier liveness escape
releases the rest of the ensemble after 12 minutes - which is the correct
degradation and is exactly what the log shows happening.
4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST.
Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of
Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the
next free row on each member's FIRST PublishStatus() call, so the order was a race -
the members busy sweeping published before the ones sitting idle at the era barrier,
and be39674 sharpened it by (correctly) making a held member stop writing the terse
line. Rows are now keyed to m_ensembleIndex, the registration/construction order,
which is fixed for the life of the chart. Claimed on every publish rather than once,
so it is idempotent and refreshes the tag for a member whose ID was not final when it
first published (the config-tag suffix is appended during InitIndicators, after
EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded
from the model count, so a member that has not published yet leaves no gap and shifts
nobody.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ALT-DATA AUDIT. The files themselves are healthy - all six symbols, 6,073 daily
rows, 2010-01-01 to 2026-08-17, no constant or degenerate columns, sane tails
(mac_cpi/mac_unemp flat ~47d is monthly data behaving correctly). The problem is
not the data, it is what happens where the data ISN'T.
CAltDataPanel::Features() returned an all-ZERO vector for any bar older than the
file's first row, and left blank cells at 0 too. Both were deliberate ('the block
is additive context and must degrade, never reject the bar') and that reasoning
holds for the CHANGE columns - but half these features are LEVELS: vix, ivol,
mac_y10, mac_cpi, mac_unemp, eia_util. For a level, 0 is not a missing reading,
it is an impossible one far outside the series' range. VIX does not visit zero.
And the spike lands in exactly the wrong place. Every alt file starts 2010-01-01
while the charts run far deeper - USDJPY H4 reaches ~1994, roughly HALF its
history - so 'alt block is all zeros' is precisely the predicate 'this bar is
older than 2010'. The IS/OOS split is chronological, so that predicate covers
~half of IS and none of OOS: an in-sample feature guaranteed to be useless
out-of-sample, and a bimodal input for the first BatchNorm to normalise. Not a
lookahead leak - a distribution corruption, which is quieter and was never
reported anywhere.
Now filled with the column MEDIAN over the covered range. A constant cannot leak
whatever its source - it takes the same value on every pre-coverage bar, so it
carries no information about which of those bars won - which is what makes a
median computed over later data legitimate here. Median not mean because the
series are skewed. Blank cells get the same treatment (eia_stk_idx1y alone has
181 blanks in 6,073 rows) and the count is now logged at load.
THE BACKOFF WAS ALREADY THERE AND WAS DEAD. Training.mqh arms m_coldSweepTick on
m_featureFailTransient, but only the open/ATR guards ever set that flag, so
f0cf659's cold ADMovingAverage looked PERMANENT and the sweep re-ran at full
speed forever. Setting the flag in the indicator guards revives the mechanism
that was already designed for this; no second backoff was needed and the one I
first wrote has been removed in favour of it.
SELF-HEALING, as asked. ReportFeatureHealth() runs once, the first time pass 1
produces usable windows, samples ~400 bars spread across the whole training range
and names every feature slot that is CONSTANT or mostly-zero, tagging alt-block
slots as alt[i]. Both of today's failures were the same shape - a block silently
produces nothing while every downstream number stays plausible - and neither an
accuracy figure nor a model can tell 'this feature is always 0' from 'this
feature is genuinely 0 here'. Evenly spaced sampling so a block that dies only in
deep history is caught as surely as one dead everywhere. A report, not a gate:
a rare-flag feature can be legitimately constant, and refusing to train would
turn a diagnostic into an outage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four defects in 64c5dd5/1a05e63, found by review + a baseline compile.
Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable.
1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were
declared `virtual bool ... override`, but CAppDialog declares both as
`virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151
on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was
never a success flag to forward. Verified: 0 errors, 0 warnings.
2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual
simulation (that one has been dead since it was written). Both are armed
at the instant convergence is declared, and both advance only from inside
Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick
ArmStudyEvent site sits in the `else` of a branch taken whenever
m_trainingComplete is set and m_trainRunActive is clear - which is exactly
the state FinalizeTrainRun() leaves behind one line before they are armed.
Train() was never called again, so the walks sat at their start index
forever: no "simulation complete" line, and not one row written to the DB
this feature exists to fill. Only a manual Resume/Retrain unstuck them.
Both flags now keep the model schedulable.
3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed.
Ensemble members deploy at Train() ENTRY and return immediately (so no era
is wasted), which skips the era-end block the backfill was started from.
All four members were a no-op for a second, independent reason. Armed on
the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff.
4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key,
no duplicate check - and m_dbBackfillDone is in-memory, so every later
attach that retrained to convergence wrote a second full set of rows for
the same bars. The ranking would count one bar once per model that ever
deployed, weighting superseded opinions as heavily as the live one. A
.dbfill marker stamps the deployed era; written only on completion (an
interrupted walk redoes itself rather than ranking a partial window) and
deleted with the other sidecars on reset-weights.
Also: WarmBlocking's timeout was silent, which restored the exact silent
pin failure it was added to prevent - it now says so in the journal, and
returns true for "no reference pairs to wait for" so the warning stays rare
enough to be read.
Not addressed, needs a decision: the backfill scores the OOS window with the
checkpoint that was SELECTED as best on that same window, then writes those
win rates into the table filter weights rank on - the selection set consumed
twice, undiscounted, while the deploy gate right next to it applies a
family-wise correction for exactly that effect. The rows are also simulated
triple-barrier outcomes at today's spread sharing a table with realised
fills. The completion log line now states both plainly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ensemble chart UI had a shared-namespace defect that answered the user
question "what do the arrows represent?" with "a bug": all four members drew
arrows under the same WarSig_<bartime> object names, so the chart showed
whichever member rendered LAST, one member Neutral deleted another member Buy
at the same bar, each member init sweep wiped the arrows the previous member
had just restored, and SaveChartSignals - which rebuilds the sidecar by
SCANNING the chart - persisted every other member arrows into its own history
(the exact cross-model laundering its own header warns about, now happening
BETWEEN ensemble members).
Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_,
WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor
purge and the sidecar scan are all member-scoped, and the tooltip names the
model. Global purges keep matching the bare WarSig_ prefix, which covers all
member namespaces plus old-format leftovers from earlier builds.
Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is
one member; the header is the ensemble) and the CONVLSTM member displays as
ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the
model folder and changing it would orphan every model trained under that path.
Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins
WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now
cover it - it was in neither list, so a dialog starved of its own Destroy()
left its controls on the chart permanently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Log review of the 18:12 attach. The wiring works: VIX, dollar index, COT, all
seven macro series fetched and SP500_D1.csv rebuilt with its full 13 features
on the first pass. Three findings from the same log, fixed:
KEY LEAK: the EIA failure line echoed the first 80 chars of the URL, which
included most of the api_key. Every URL-echoing error path now goes through
MaskUrl(). The key itself is unchanged - it was printed to a local journal,
not transmitted - but rotate it if that log ever leaves the machine.
EIA HTTP 1003: an MT5 transport-layer code, not a server response. Requests
now carry a User-Agent (gateways reject empty-UA at the edge; the CBOE probe
showed no-UA is fine THERE, but EIA fronts differ) and 1xxx codes are
explained in the log line. Retries were already hourly.
UNBOUNDED BACKFILL: an empty cache fetched full series history - CPIAUCNS
goes back to 1913, whose pre-1970 dates are outside MQL5 datetime range and
whose 1913-era levels sat below the plausibility band, producing 157
scary-but-meaningless REJECTED lines. All FRED fetches now start at 2005
(5y of lookback margin ahead of the 2010 grid). DTWEXBGS staleness horizon
raised to 10 days to match its weekly H.10 publication lag.
Also confirmed from the log: the running build predates the H4 fallback, so
the H4 panels still show 0 features - resolved by the recompile this commit
requires anyway.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Systematic audit of the alt-data stack against "any symbol, any timeframe",
prompted by the H4 surprise. Findings, each fixed:
CAPACITY: ALTDATA_MAX_FEATURES was 16 with FX symbols already at 15 - the next
added column would have been silently truncated by a MathMin. Raised to 32,
pin-chars 512 -> 1024.
SYMBOL NAMES: the panel reload path re-derived symbol/timeframe by splitting
the file path on its FIRST underscore - mis-parsing every symbol containing
one (OANDA-style EUR_USD and US_500 are in our own alias lists) and knowing
only three timeframes. It now stores the (symbol, period) Load() was called
with and reuses them verbatim. Path-hostile characters in broker symbols
("EUR/USD") are sanitized by a shared AltDataFileSymbol() used by the panel,
the fetcher and TunedPeriods, so a slash cannot route a write into an
unintended subfolder.
DOWNLOAD VALIDATION: every FRED-family fetch now enforces a per-series
plausibility band (VIX 1-200, yields -5..30, CPI index 20-1000, ...) -
StringToDouble on transport garbage returns 0.0, and one absurd value poisons
every change/percentile feature computed across it (the BatchNorm NaN-latch
incident came from exactly one huge-but-finite input). Rejected rows are
counted and reported, never dropped silently.
LOUD EMPTINESS: a successful response with zero observations on an empty
cache now says so - naming the series (wrong id / format drift) or the COT
predicate (the unverified like-clauses) instead of leaving 0-filled features
unexplained. GEX gains a truncation guard: a day-over-day contract-count
collapse >50% is the fingerprint of a partial 13 MB download, not of markets,
and is skipped rather than recorded as a plausible-but-wrong number.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user attached H4 charts and the panel found no {SYM}_H4.csv - the fetcher
only writes _D1 files - so the run trained with ZERO alt features, silently.
The daily file is timeframe-agnostic by construction (rows are as-of daily
values, and Features() joins published <= bar open per bar), so the panel now
falls back to {SYM}_D1.csv on any timeframe, logging the substitution. A
per-TF file still takes precedence if one ever exists.
Per-symbol subfolders (the user suggestion) are NOT the fix for multi-chart
concerns: filenames are already symbol-keyed, and the shared caches
(raw_VIXCLS etc.) are shared deliberately - one download serves every chart.
The REAL races were: (1) two charts of one symbol (D1+H4) each caching their
own last-GEX date and double-appending the same day - UpdateGex now re-reads
the file date before spending the download; (2) whole-file rewrites were
truncate-then-write, so a concurrent reader could parse a torn file - SaveRaw
and RebuildFeatures now write a temp and FileMove-swap it into place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user whitelisted the hosts and still got the alert - because the alert
never said WHICH request failed. Two hosts (api.eia.gov, cdn.cboe.com) were
added to the EA after the original whitelist instruction, so any build newer
than the whitelist raises 4014 on the new hosts while the message implied the
old ones were the problem.
Three defects fixed: the popup and journal now print the exact blocked host as
a copy-paste whitelist line; the stale "three URLs" text is gone (the full
four-line reference prints once per session); and the backoff is per-host
instead of global - one missing entry no longer silences the whitelisted
sources for an hour per miss. Popup fires once per host per session; hourly
retries log one quiet line naming the host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner decision (stated twice): available data gets wired; the networks judge
usefulness; the deploy gate remains the arbiter of what trades. Implemented:
MACRO block (6) on every symbol: 10y yield 20d change, curve slope, 5y
breakeven 20d change, Fed-ECB policy gap, CPI yoy, unemployment 12m change.
Screened null vs forward range on all four research symbols - recorded as
the honest prior in the catalog comment, wired regardless.
RISK block (3) extended to every symbol (FX majors, metals, energy, BTC all
now carry vix/vix_chg5/usd_chg5).
IVOL pair extended with the level alongside the change.
Vintage integrity kept where it is free: CPI is fetched as CPIAUCNS (NSA,
essentially never revised) so the plain-FRED backfill stays first-print-clean;
yields/curve/breakevens/policy rates are unrevised by nature. UNRATE is the
one exception (seasonal refits, ~0.1-0.2pp) - the EA cannot run the ALFRED
protocol, accepted and documented at the declaration site.
UpdateFred gains a staleDays parameter so the monthly series do not fire a
pointless fetch attempt every hour for three weeks after each print.
FeatureValue now takes the day and does its own as-of lookups - adding a
source no longer widens a parameter list. Feature counts: 12-15 per symbol;
symbol feature-order changed, safe only because no models exist yet.
export.py mirrors the new catalog for the five research symbols (13-15
features), smoke-tested: all five CSVs written, 6,072 daily rows each.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the screen_ivol survivors (41d726c). New per-symbol `ivolSeries` in the
catalog feeds a generic `ivol_chg5` feature from whichever CBOE vol index the
instrument owns, so one code path serves every symbol:
XAUUSD + ivol_chg5 (GVZ) - MI|vol 0.01971 p=0.002, 3.6x the positive
control and 4.6x the vix_chg5 gold had alone.
vix_chg5 KEPT: this appends, it does not replace.
EURUSD + vix_chg5 - screened, incremental p<=0.006, and its first
real feature ever (it had only exploratory EIA).
USDJPY + vix_chg5 - screened, incremental.
NAS100 / US30 / US2000 + ivol_chg5 (VXN / VXD / RVX) - exploratory by analogy.
XTIUSD / XBRUSD + ivol_chg5 (OVX) - exploratory, no oil bars to screen yet.
SP500 unchanged - its features already screened clean and VXN/VIX3M edging
out VIX is a correlated within-family best-of-N, not a real ranking.
On EURUSD/USDJPY the screen put VXD marginally above VIX, but they are
near-duplicates and the gap sits inside the noise, so the tie is broken by a
rule rather than by the number: take the series already in the fetch path.
Also fixes a real collector bug: fetch_vintaged built ALFRED realtime windows
out to 2028, and FRED rejects realtime_end after today - so every REVISED
series (unemployment, CPI, GDP: exactly the ones needing the vintage path) was
unreachable, while unrevised series never noticed because they bail earlier.
UNRATE and CPIAUCSL now return first prints correctly.
Adds screen_macro.py (rates, curve, breakevens, Fed/ECB policy differential,
plus monthly country stats) with a `distinct` column that reports the honest
effective sample size - a monthly series pasted onto D1 bars is a step
function, and that column is what decides whether it can clear a gate at all.
Not yet run: the Market Data bars directory is being regenerated right now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Option open interest is a snapshot source - no free history exists anywhere -
so the series only accrues from the day recording starts. That is why this
ships BEFORE the redeploy: every day the EA is not running is a day of history
that cannot be recovered later.
Records one row per weekday after 21:00 UTC to gex_{CANONICAL}.csv: net/call/put
dollar GEX per 1% move, call and put OI, the three nearest expiries and the
front expiry code. Feeds NOTHING - wiring a feature that is missing across ~100%
of the training sample would waste input width and hand batch-norm a constant.
It becomes a screening candidate at ~250 rows, gated like every other feature.
Thesis: dealer gamma is a RANGE mechanism (long gamma -> hedging sells rallies
and buys dips, range compresses; short gamma amplifies both ways), and range is
this project's one proven channel.
Verified in situ against the live SPX chain before writing any MQL5: 29,362
contracts, 20,993 with nonzero gamma, 54 expiries, total +90.7 Bn/1% (calls
+305.7, puts -215.0), and 100% of net GEX inside 5% of spot. The CDN publishes
per-contract gamma directly, so no pricing model - and no model risk - enters
the recorded data. Also verified the CDN does NOT gate on User-Agent (the old
"CBOE is UA-gated" note in DESIGN.md was a different CBOE path), so plain
WebRequest reaches it.
Dropped a zero-gamma "flip level" field: the probe returned a crossing above
spot while total GEX was strongly positive, which is incoherent - a static
gamma snapshot cannot give a flip level without repricing. Recording a
plausible-looking wrong number is worse than recording nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The AltData folder in Common\Files gets wiped before every fresh test, and
keys.txt died with it (2026-08-16 silent-FRED incident). The credential now
travels with the EA: FredApiKey input, owner key as default; keys.txt demoted
to a fallback consulted only when the input is blanked. EiaApiKey stored the
same way - reserved, nothing consumes it since the WTI screen came back null.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-08-16 first live fetch looked complete but was not: COT (keyless)
downloaded 1053 reports, then FredKey() hit the absent keys.txt, returned ""
with no log line, UpdateFred bailed, and the SP500_D1.csv rebuild - gated on
all three raw series - never happened. The panel stayed at 0 features with
nothing in the journal explaining why.
FredKey() now logs loudly when keys.txt is missing, and only latches once a
key is actually FOUND: the file is re-read on each hourly-throttled attempt,
so dropping keys.txt in after attach recovers without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
