feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//+------------------------------------------------------------------+
2026-07-14 22:36:27 -04:00
//| Warrior_EA |
//| AnimateDread |
//| |
//+------------------------------------------------------------------+
# include "ExpertSignalCustom.mqh"
# include "..\AI\Network.mqh"
feat: add unified MA type support to indicator tuner
Add `MA_TYPE_PRESETS` enum covering advanced (ALMA, DEMA, ZLEMA, T3, Kalman) and standard (SMA, EMA, SMMA, LWMA) moving averages. Integrate `maType` and `bestMaType` into `CADIndicatorTuner` struct, update flatten/unflatten routines, and bump `AD_TUNE_PARAM_COUNT` to 33. This allows the auto-tuner to search over MA type alongside period, improving feature discovery.
2026-07-23 15:02:09 -04:00
# include "..\Variables\IndicatorResources.mqh"
2026-07-14 22:36:27 -04:00
# include "..\Variables\IndicatorTuneRanges.mqh"
2026-07-17 21:28:59 -04:00
# include "..\System\StatusLabel.mqh"
feat: add configurable news event proximity/impact as an NN input feature
Price, time, volume, and volatility were already trained-model input
features; the real economic calendar (already used for the live
NewsFilter veto) is now an optional one too, reusing
System/NewsRelevance.mqh's symbol-relevance logic from the prior fix.
New EnableNews/NewsFeatureWindowMinutes inputs gate two features per
bar: minutes-since and minutes-until the nearest symbol-relevant
calendar event, impact-weighted. Deliberately limited to proximity +
impact, not actual-vs-forecast deviation - release schedules are
public knowledge ahead of time (not lookahead bias to use for a
historical training bar), but a release's actual outcome is not.
Wired identically to the existing EnableVolume/EnableTime/EnableATR
toggles: InitIndicators() accounts for the +2 neuron count,
BufferTempDataCompute() appends the two feature values, PAI/CONV/LSTM
all wired in Warrior_EA.mq5. Compiled clean (MetaEditor, 0 errors/0
warnings).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 17:26:04 -04:00
# include "..\System\NewsRelevance.mqh"
fix(signals): revive a dead MA model, and demote Sanyaku from state to event
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
# include "..\System\CrossAsset.mqh"
2026-08-16 13:39:00 -04:00
# include "..\System\AltData.mqh"
refactor(ExpertSignalAIBase): extract AutoTune param state into CADIndicatorTuner
CExpertSignalAIBase (4,326 lines, one class) carried 5 struct
definitions, 10 member fields, and 3 methods (Flatten/Unflatten/
PerturbRandom) purely for the AutoTuneIndicators search-space state -
entirely self-contained (never touches Net, Train()'s resumable state
machine, or anything else in the class). Extracted into a new
Expert/ADIndicatorTuner.mqh (CADIndicatorTuner), held as a single
m_indicatorTuner member.
TuneIndicatorsAndTrain() itself - the outer loop that actually
orchestrates Train()/Net/checkpointing around this tuner - turned out
to be exactly as tightly coupled to Train()'s resumable state machine
as Train() itself, so per the same caution already applied to Train()
in this refactor pass, it stays in CExpertSignalAIBase rather than
being pulled into the collaborator; it now calls the tuner's public
Flatten()/Unflatten()/PerturbRandom()/SaveAsBest()/RestoreBest()
instead of manipulating the structs inline.
All internal field-access renames (m_adCumDeltaParams.lookback ->
m_indicatorTuner.adCumDelta.lookback, etc., ~40 sites across the 5
InitAD*() indicator-setup methods) verified against a full grep sweep
- no leftover references to the old field/method names. Compiled
clean (MetaEditor, 0 errors/0 warnings).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 16:22:09 -04:00
# include "ADIndicatorTuner.mqh"
2026-07-18 14:56:41 -04:00
//--- Hard 0/1 one-hot targets for the 3-neuron classification head - matches the book's
//--- (nnbook.txt sec. 1.4 "Cross-entropy") cross-entropy formulation, which defines the reference
//--- distribution's occurring-event probability as exactly 1.0 (missing event exactly 0.0), no
//--- smoothing. This used to be softened to 0.9/0.1 (see git history) as a workaround for the OLD
//--- per-neuron independent-sigmoid backward gradient: since that gradient was each neuron's own
//--- raw (target-sigmoid_output) delta with nothing else driving it to zero, a literal 1.0/0.0
//--- target was only ever approached asymptotically, so weights kept growing (up to the MAX_WEIGHT
//--- clamp) chasing it forever. Now that CNet::backProp()/backPropOCL() (AI\Network.mqh) compute a
//--- joint softmax+categorical-cross-entropy gradient (softmax_i - target_i) across all 3 neurons
//--- instead, the winning class's SOFTMAX probability - not any one neuron's raw sigmoid value -
//--- is what needs to approach the target, and softmax can reach very close to 1.0 for the winning
//--- class from ordinary (non-saturated) logit separation, so the old runaway-weight failure mode
//--- is no longer expected to require smoothed targets to avoid. If it resurfaces in practice (the
//--- per-neuron SIGMOID forward pass can still individually saturate before softmax normalizes),
//--- that's the first thing to check before reintroducing smoothing.
2026-07-28 10:49:53 -04:00
// Soft labels keep the classification head from overfitting to hard one-hot targets while still
// preserving a clear target for the true class. The true class gets 0.9 and the others 0.05 each.
# define LABEL_SMOOTH_HIGH 0.9
# define LABEL_SMOOTH_LOW 0.05
feat: add unified MA type support to indicator tuner
Add `MA_TYPE_PRESETS` enum covering advanced (ALMA, DEMA, ZLEMA, T3, Kalman) and standard (SMA, EMA, SMMA, LWMA) moving averages. Integrate `maType` and `bestMaType` into `CADIndicatorTuner` struct, update flatten/unflatten routines, and bump `AD_TUNE_PARAM_COUNT` to 33. This allows the auto-tuner to search over MA type alongside period, improving feature discovery.
2026-07-23 15:02:09 -04:00
//--- Namespace prefix for the directional signal arrows this class draws (DrawObject/DeleteObject). Two
//--- reasons it exists: (1) so PurgeChart() can delete ONLY our arrows and never the user's own manual
//--- chart drawings (a full ObjectsDeleteAll(0) on a client's chart is not acceptable for a commercial
//--- product), and (2) so arrows can survive an EA re-init instead of being wiped on every InitIndicators
//--- - the whole point of keeping them on screen (the panel has a hide toggle if the user wants them gone).
feat(chart): filtered view - one arrow per trade the bot would actually take
Adds DrawUnfilteredSignals (default OFF) and, with it off, replaces the
per-model arrow layer with the decision the EA would really have made.
THE FILTERED ARROW IS DRAWN AT THE ORDER, NOT AT THE THRESHOLD. Clearing
Min_Vote_Open is not the same as trading: a setup can pass the vote and still
never reach the broker (invalid SL/TP, stops-level, ATR warm-up, unsynced
swing history), and every one of those lands in OpenParams' failure branch.
So DrawVoteArrow() fires only after the order parameters validate, and the
failure branch withdraws any arrow already standing on that bar. One arrow is
one entry the EA would have placed - carrying the vote, the threshold it
cleared, and the SL/TP the order would have had.
Classic signals now draw too, under their own name and weight, so a chart
running MA/RSI/MACD/Ichimoku alongside the nets reads the same way an
ensemble chart does. They can only be drawn from the aggregate's once-per-bar
pass, because unlike the AI members they have no cached per-bar scan.
Two subtleties that would each have produced a quietly wrong chart:
- The raw classic draw sits AFTER filter.Direction(), not beside the
journaling block. GetActivePattern*() are CONSUMING reads holding the
PREVIOUS evaluation - "one tick later", which at Expert_EveryTick=false is
one BAR later. Keyed off those and placed at StartIndex(), every classic
arrow would have been drawn one bar early, which on a chart is
indistinguishable from a model that genuinely leads. Peek*() accessors
(non-consuming) let pattern, weight and bar come from one evaluation.
- CExpertSignalAIBase::DrawObject() early-returns instead of gating its five
call sites, so the switch cannot be honoured in three passes and missed in
the fourth. Its delete counterparts stay ungated so flipping the input off
and rescanning clears the raw layer rather than stranding it.
SIG_ARROW_PREFIX and g_signalsVisible move from ExpertSignalAIBase.mqh down
to ExpertSignalCustom.mqh - the nearest common ancestor - because the classic
signals cannot see the AI header (it is included later in Warrior_EA.mq5).
The vote layer gets SIG_VOTE_PREFIX under the same bare prefix, so
WarriorChartPrefixes()' purge still reaches every arrow without knowing they
exist.
NOT YET BUILT: the reconstructed history behind attach. Filtered arrows
currently start where the EA starts. See the next commit.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:41:28 -04:00
//--- MOVED to Expert\ExpertSignalCustom.mqh (the nearest common ancestor) when the classic signals
//--- started drawing too - this header is included too late for them to see it. Left as a comment
//--- rather than deleted because every reference below still names it and the reader lands here.
fix: purge every EA object namespace on init and after deinit teardown
Leftover objects survived deinit because the cleanup list had drifted.
PurgeChart()'s own comment said it removed "our namespaced signal arrows
plus the status-label objects" while the code removed arrows ONLY, and
the panel prefix was swept at OnInit and nowhere else - so an ordinary
deinit left the status line, and any panel straggler, on the chart.
Three scattered call sites and a comment cannot be kept in step. There is
now ONE list - WarriorChartPrefixes() - covering arrows, status label and
panel, and one sweep, WarriorPurgeChartObjects(), used by every path.
Add a prefix there when a new object family appears and every cleanup
picks it up.
Two call sites added:
OnInit, before ANYTHING is drawn (including the status label it would
otherwise delete). Chart objects live in the chart PROFILE, not in the
EA, so they outlive the process: a deinit force-terminated at
MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5
replaced while attached all strand objects no later deinit will ever
own - and deleting the EA's files does not remove them, which is why
they read as corruption. Arrows are included: LoadChartSignals restores
them from their sidecar moments later and already opens with its own
arrow sweep, so this only removes orphans the sidecar does not account
for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds
that sidecar by scanning the chart.
OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control
tree and ClearStatusLabel clears text rather than guaranteeing object
removal; either can leave a straggler and nothing looked afterwards.
Bounded work - three prefix deletes and one object-list scan - so it
respects the ordering rule that keeps the cheap visible cleanup ahead
of the heavy save. Arrows excluded: ShutdownChartCleanup already
persisted and removed them and re-deleting would race that write.
The two are complementary: the deinit sweep closes the ordinary case, the
OnInit purge closes the case where MetaTrader never let us finish. Only
the second can help after a starved shutdown.
Both sweeps rescan by name across EVERY object type and delete what the
bulk call missed. ObjectsDeleteAll's return has already been observed
disagreeing with a by-name scan of the same chart microseconds apart, and
object commands are queued on the chart rather than applied inline, so a
returned count is not evidence the objects are gone.
Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so
the name cannot drift away from the list that cleans it up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
//--- Control-panel object namespace. CAppDialog names every control it owns from the dialog name it is
//--- created with, so one prefix covers the whole tree. Declared HERE rather than left as a string
//--- literal at the ExtPanel.Create call site so it can appear in the sweep list below - a prefix that
//--- lives only at its creation site is a prefix nothing cleans up.
# define WARRIOR_PANEL_PREFIX " WarriorCP "
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- Custom chart-event id space for the training self-event ("study" event). Every instance used to
//--- post id 1 and handle id 1001, which had two consequences on an ensemble chart: (a) CExpertCustom
//--- broadcasts chart events to EVERY filter, so each member's event ran a train chunk in ALL N
//--- members - N events per round times N chunks each = N*N chunks, the chart thread never went idle,
//--- and the control panel's clicks/drags starved behind the queue (observed 2026-08-16 as a completely
//--- dead panel in AI_HYBRID ensemble mode: no buttons, no dragging, no minimize); and (b) id 1 IS the
//--- Standard Library Controls ON_DBL_CLICK code, so a double-click on any panel control also fired a
//--- training chunk. Ids are now assigned per instance from this base, chosen far above the Controls
//--- library's ON_* event range (0..~100) so neither side can ever misread the other's events.
# define STUDY_EVENT_ID_BASE 500
//--- Lost-event watchdog (see ScheduleTrainingIfNeeded): before the per-instance ids, a member whose
//--- armed event vanished (queue overflow) was accidentally rescued by any SIBLING's event clearing its
//--- bEventStudy. Member-scoped ids remove that rescue, so the liveness guarantee has to be explicit:
//--- an event armed this long ago that never arrived is declared lost and re-armed. Generous on purpose
//--- - a legitimately queued event can wait tens of seconds behind a sibling's unchunked warm-up
//--- diagnostics, and while the thread is busy the watchdog cannot run either, so a false trip would
//--- need the thread idle AND the event still missing, which is exactly the failure being covered.
# define STUDY_EVENT_LOST_MS 60000
//--- Next unassigned study-event id, claimed in the constructor. File-scope because each chart runs its
//--- own copy of the program (globals are per-instance in MQL5), so this simply numbers the ensemble
//--- members 0..N-1 in construction order on this chart.
int g_warriorStudyEventSeq = 0 ;
//+------------------------------------------------------------------+
//| ENSEMBLE CHART-LEVEL SHARED STATE (AIType == AI_HYBRID preset). |
//| |
//| The registry lists every AI signal running as an ensemble member |
//| on this chart (registered in EnsembleMember()). It exists because |
//| three user-requested behaviours (2026-08-16) are chart-level, not |
//| member-level: |
//| 1. ERA BARRIER - members advance era by era TOGETHER. A member |
//| ahead of the slowest still-training member declines its |
//| Train() calls (EnsembleEraBarrierHolds), and its chunk budget |
//| flows to the laggards (EnsembleActiveTrainers scales |
//| TRAIN_TIME_BUDGET_MS), so wall-clock throughput is conserved. |
//| 2. COMBINED-VOTE OOS SCORE - what the ensemble panel reports as |
//| THE ensemble's accuracy. Each member's pass-3 OOS scan drops |
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//| the per-bar VOTE it would have cast (LiveVoteContribution(), |
//| not its raw confidence) into the row buffer below |
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//| (EnsembleOosContribute); when every still-training member has |
//| finished its scan for the era (EnsembleOosPassComplete), the |
//| last one scores the AVERAGED vote against the same |
//| win-long/win-short outcomes the members grade themselves on |
//| (EnsembleScoreCombinedVote) - the panel line answers "how |
//| often is the vote that actually trades right", which no |
//| per-member number can. |
//| 3. The auto-tune/MI warm-up sharing (see AutoTune.mqh's |
//| g_ensembleChartTuneDone) rides the same identical-inputs |
//| argument. |
//| Solo charts register nothing and none of this runs. |
//+------------------------------------------------------------------+
class CExpertSignalAIBase ;
CExpertSignalAIBase * g_warriorEnsemble [ ] ;
//--- Combined-vote OOS row buffer, one row per scored OOS bar of the CURRENT era (g_ensVoteEra; the
//--- era barrier keeps members on the same era, so one stamp suffices - a stale-era contribution
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- resets the buffer). Sum + bitmasks rather than per-member columns: the vote is an average, so
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- only the sum and who contributed are needed. Win outcomes are label-cache-derived and identical
//--- across members; the first member to touch a row writes them.
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//---
//--- TWO masks, not one, and the difference is the whole union-semantics rule:
//--- g_ensVoteMask - who EVALUATED this bar (contributed a row at all). Gates the "shared bars
//--- only" test: a bar one member skipped is a different population.
//--- g_ensVoteVoterMask - who actually VOTED (contributed a NON-ZERO number). This is the divisor,
//--- because CExpertSignalCustom::Direction() skips abstentions in BOTH the sum
//--- and the count (`if(direction == 0) continue;` before `number++`), so the
//--- live vote is a mean over VOTERS, not over members. The gate used to divide
//--- by the member count, which made an abstention dilute here and not there -
//--- a strictly different fire set from the one that trades.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
datetime g_ensVoteTime [ ] ;
double g_ensVoteSum [ ] ;
int g_ensVoteMask [ ] ;
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
int g_ensVoteVoterMask [ ] ;
feat(vote): thresholds become confidence percentages, on ONE scale everywhere
User request: "the entry/exit thresholds are manual numbers, I would like
them to be confidence percentages, so the current 20 would be only 20%
confidence in a profitable trade."
WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's
contribution are win rates: the pattern weight is that pattern's measured win
rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's
average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT
therefore produced a mean of PRODUCTS of two win rates - a genuinely
60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was
never on a probability scale, so its magnitude meant nothing on its own.
Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which
is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60;
MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight
stops being a discount on the probability and becomes how much a filter's
opinion COUNTS - which is what a module weight should always have been.
Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed.
ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three
other places compared against a 0..1 softmax confidence and would each have
become a fresh currency mismatch the moment the input changed meaning:
* the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now
reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the
classic side, which is the only reason that route exists - against the
same m_threshold_close the averaged vote uses. m_ai_exit_threshold is
retired rather than left dangling.
* m_oosDecisionSeries now carries the vote, not the confidence, so the exit
SIMULATION stops modelling a close rule the EA does not run.
* ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input
through that would have silently switched vote exits off in the
simulation while live went on running them - found before it shipped;
the bound now tracks the scale.
LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing,
SL/TP scaling and the intelligent trailing want a model confidence, not a win
rate.
CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a
real probability to the extent the pattern weights are. A pattern with fewer
than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a
designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the
signal DB fills, "60" means "the designed conviction of the patterns that
fired". Closing that gap is the next commit.
Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales
this removes.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
//--- Sum of the module weights behind g_ensVoteSum, i.e. the weighted mean's DIVISOR. Carried per
//--- row rather than recomputed at verdict time because a member's m_weight can be rewritten by
//--- UpdateSignalsWeights() between the scan and the verdict, and the divisor has to be the one
//--- that was in force when the numerator was accumulated - otherwise the ratio is of two
//--- different moments. See CExpertSignalCustom::Direction()'s normalization comment.
double g_ensVoteWeightSum [ ] ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
bool g_ensVoteWinLong [ ] ;
bool g_ensVoteWinShort [ ] ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
bool g_ensVoteDirLabel [ ] ; // bar carried a Buy/Sell label - the coverage floor's base rate
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
int g_ensVoteRows = 0 ;
long g_ensVoteEra = -1 ;
int g_ensVoteDoneMask = 0 ;
int g_ensVoteCursor [ 8 ] ; // per-member monotonic row cursor (members scan bars in the same order)
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- Population count over the member masks above. Bounded by the 8-slot ensemble, so a plain loop is
//--- both the clearest and the fastest thing here; it runs once per shared OOS row per era.
int EnsembleBitCount ( const int mask )
{
int n = 0 ;
for ( int b = 0 ; b < 8 ; b + + )
if ( ( mask & ( 1 < < b ) ) ! = 0 )
n + + ;
return n ;
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//+------------------------------------------------------------------+
//| ENSEMBLE DEPLOY GATE (user request 2026-08-16: "an ensemble mode |
//| for the pass gates - at the end of the day they vote together"). |
//| |
//| In ensemble mode the UNIT OF EVALUATION IS THE VOTE. Members keep |
//| their own per-era statistics and their own learning-rate dynamics |
//| (regression restore, eta decay - those are per-net training |
//| mechanics), but four decisions move here, because all four are |
//| really questions about what gets TRADED: |
//| * which era is "best" -> best COMBINED-vote era |
//| * what gets checkpointed -> a JOINT snapshot: every member's |
//| weights at that one era |
//| * when the run gives up -> one shared plateau ladder |
//| * whether it may deploy -> family-wise gate on the VOTE |
//| |
//| WHY A JOINT CHECKPOINT IS THE WHOLE POINT: per-member selection |
//| picks each net's own best era, and those eras are different. The |
//| resulting quartet was never measured together at any instant, so |
//| the vote it casts live is a configuration no OOS number ever |
//| described. Capturing all four at the era whose VOTE scored best |
//| makes the deployed ensemble exactly the one that was measured. |
//| |
//| SAFE BY CONSTRUCTION, and it depends on the era barrier: Train() |
//| runs at most one era per call and a member that finished era N |
//| is held at the barrier until every member reaches N, so when the |
//| last member scores the vote, no member's weights have advanced |
//| past the end of era N. That is what makes a deferred simultaneous |
//| capture correct rather than a race. |
//| |
//| The statistics mirror the per-member gate one for one (see |
//| tradeableOK / BestCheckpointSurvivesSelection in Training.mqh): |
//| same coverage floor, same EDGE_MIN_SIGMAS margin over the same |
//| always-call-one-direction chance reference, same Sidak correction |
//| over the eras ranked. Only the population changes - the bars the |
//| VOTE fired on, rather than the bars one member called. |
//+------------------------------------------------------------------+
double g_ensBestScore = -1.0 ; // best combined-vote selection score (precision x coverage credit)
bool g_ensBestTradeable = false ; // did that era clear the vote's own deployability floor
bool g_ensBestTwoSided = false ; // did it fire both long and short
double g_ensBestPrecPct = -1.0 ; // the winning era's vote win rate, for the family-wise test
double g_ensBestChancePct = -1.0 ; // and its chance reference
int g_ensBestCalls = 0 ; // and the n that sets the standard error
long g_ensBestEra = -1 ;
int g_ensCandidateEras = 0 ; // N for the family-wise correction: eras that COULD have won
int g_ensErasSinceBest = 0 ; // shared plateau counter
int g_ensPlateauStage = 0 ; // shared plateau stage
fix(plateau): the IS-error early stop was inert for every ensemble member
15 hours of training, and the stop that exists to END a run announced itself
1,299 consecutive times without ending anything:
SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299
eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398
SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras
XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras
CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors
the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` -
on EVERY era, purely so each member's status line shows the collective stage. A
display mirror was silently overwriting a decision, so the stop re-armed and
re-fired the next era, forever.
This is the worst possible direction for this particular bug. Every one of those
1,299 eras was scored out of sample and joined the family the deploy gate
corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make
that family SMALLER; instead the run spent fifteen hours raising its own bar.
- m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run.
Nothing in the ladder may reset it. The stop condition and the two solo deploy
conditions read the latch, not the mirrored stage.
- The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across
participating members (same participation test the era barrier uses, so an
excluded or finished member cannot veto). One member still learning can still
move the combined vote, and the vote is what the gate certifies.
- Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage.
The block that actually ends the run sits under `dueStage > g_ensPlateauStage`,
so assigning the stage directly makes that test false and the deploy never
happens - the same inert-write shape as the bug being fixed. Caught before
committing; raising dueStage carries it through the ladder's own path (warm
restarts skipped, family-wise vote test, measurement screen, joint checkpoint)
unchanged.
- g_ensIsPlateauAnnounced: announce once per run, not once per era.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
//--- One-shot latch so the collective IS-error plateau announces itself once per run instead of once
//--- per era. Reset with the rest of the shared gate state at the start of a run.
bool g_ensIsPlateauAnnounced = false ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
bool g_ensDeployApproved = false ; // stage 3 reached AND the vote cleared the family-wise gate
long g_ensLastVerdictEra = -1 ; // guards against scoring one era twice
2026-08-16 21:08:41 -04:00
//--- Lifetime (compounded, never reset per era) combined-vote win-rate over every bar the VOTE
//--- actually fired on, in the SAME shape as a solo model's m_cumOosCorrect/m_cumOosTotal - so the
//--- ensemble panel reads with the identical label and the identical measurement (a persisted OOS
//--- win-rate over called bars) instead of a single era's fired-bar percentage, which is a different,
//--- noisier quantity and used a different label ("Ensemble vote: win X% on Y fired bars"). Not
//--- persisted to disk - like every other g_ens* search-state global, it is chart-session-scoped and
//--- resets on a fresh attach, same as the rest of the ensemble ladder state.
long g_ensCumOosCorrect = 0 ;
long g_ensCumOosTotal = 0 ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- Mirror of the Min_Vote_Open input (the CExpert open threshold), pushed in at registration so the
//--- combined-vote scorer fires on the same criterion the live trade does. Signals themselves only see
//--- their own confidence; the threshold is an EA-level input this header cannot read directly.
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//---
//--- UNITS: the 0..100 VOTE scale - the same numbers m_pattern_0..3 carry and the same scale
//--- CExpertSignalCustom::CheckOpenPosition compares m_direction against. It is NOT a confidence
//--- percentage; the scorer used to multiply a 0..1 confidence by 100 to meet it here, which put a head
//--- output and a DB-ranked win-rate weight on the same axis with nothing relating them. See
//--- LiveVoteContribution(). The seed below is only ever read before registration overwrites it, which
//--- happens for every ensemble member (Warrior_EA.mq5) before any verdict can run.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
double g_ensembleVoteThreshold = 60.0 ;
2026-08-16 21:08:41 -04:00
//--- Set true by AdvancePatternDatabaseBackfill() on completion; consumed by Warrior_EA.mq5's OnTimer
//--- to bypass the hourly DB-ranking throttle ONCE, so filter weights refresh from the freshly
//--- backfilled rows immediately instead of waiting up to an hour to reflect a model that just deployed.
bool g_forcePatternWeightsRefresh = false ;
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era
Full-pipeline analysis after "threshold 30, attained often, nothing drawn,
still glued to buy". The log falsified the premise before any code did:
21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0%
21:42:07 swept 4999, 0 voters, drew 0
21:51:30 swept 4999, 0 voters, drew 0
21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0%
The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root
cause, three symptoms: every display path read m_arrowSignalCache, which is
wiped to sentinel at each era start and only complete again when pass 3
finishes. With eras at ~30s and a sweep at ~17s:
* ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its
else-branch deleted the arrow on every voteless bar - erasing the previous
sweep's entire output. The chart cycled populated -> blank -> populated;
the user kept catching the blank phase.
* READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every
era and fell through to dPrevSignal - the frozen purge-band edge bar that
reads Buy. 659638e fixed which bar was frozen, not the freezing.
* VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a
different fraction of half-rebuilt caches.
THE FIX, structural rather than another patch:
1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one
moment the cache is complete - and now copies it (raw signals, newest
LOOKBACK+16 bars) into member-owned snapshot state, unconditionally,
BEFORE its early return: an all-Neutral era is a snapshot worth showing,
not an absence of one. Raw signals rather than votes, so a tier re-rank
between eras reprices them at read time via LiveVoteContribution for free.
2. The sweep (SnapshotVoteAt) and the prospective readout both read
snapshots; the readout's fallback chain is live-cache -> snapshot ->
dPrevSignal, and the snapshot leg is the one that fires most of the time.
3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual
sub-threshold vote takes an arrow down. This alone ends the wipe half of
the flicker even where snapshots are missing (before the first era).
4. Arming moved from an era-counter diff (which fires at era BOUNDARIES,
i.e. precisely when caches are about to be wiped) to
g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's
snapshot just got fresher", the only event a redraw can act on. 60s rate
limit collapses the four members' burst into one sweep. Classic-only
charts arm once at start.
5. Census now reports the direction split - "922 had a voter (610 buy / 312
sell)" - so "the vote leans buy" is checkable from the log instead of
inferred from arrow colours.
Also visible in the log and worth knowing: the threshold flip-flopped
30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51
ran at 40), so part of the observed blankness was configuration, not code.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
//--- Set true by RankTiersFromOos() at every pass-3 completion (any member, solo or ensemble);
//--- consumed by Warrior_EA.mq5's timer to (re)arm the filtered-view overlay sweep. A flag rather
//--- than an era-sum comparison because what the sweep needs is "a member's snapshot just got
//--- fresher", which era counters only approximate - and approximating it is how the sweep used to
//--- re-arm against half-built caches.
bool g_warriorOverlayArmRequest = false ;
fix: purge every EA object namespace on init and after deinit teardown
Leftover objects survived deinit because the cleanup list had drifted.
PurgeChart()'s own comment said it removed "our namespaced signal arrows
plus the status-label objects" while the code removed arrows ONLY, and
the panel prefix was swept at OnInit and nowhere else - so an ordinary
deinit left the status line, and any panel straggler, on the chart.
Three scattered call sites and a comment cannot be kept in step. There is
now ONE list - WarriorChartPrefixes() - covering arrows, status label and
panel, and one sweep, WarriorPurgeChartObjects(), used by every path.
Add a prefix there when a new object family appears and every cleanup
picks it up.
Two call sites added:
OnInit, before ANYTHING is drawn (including the status label it would
otherwise delete). Chart objects live in the chart PROFILE, not in the
EA, so they outlive the process: a deinit force-terminated at
MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5
replaced while attached all strand objects no later deinit will ever
own - and deleting the EA's files does not remove them, which is why
they read as corruption. Arrows are included: LoadChartSignals restores
them from their sidecar moments later and already opens with its own
arrow sweep, so this only removes orphans the sidecar does not account
for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds
that sidecar by scanning the chart.
OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control
tree and ClearStatusLabel clears text rather than guaranteeing object
removal; either can leave a straggler and nothing looked afterwards.
Bounded work - three prefix deletes and one object-list scan - so it
respects the ordering rule that keeps the cheap visible cleanup ahead
of the heavy save. Arrows excluded: ShutdownChartCleanup already
persisted and removed them and re-deleting would race that write.
The two are complementary: the deinit sweep closes the ordinary case, the
OnInit purge closes the case where MetaTrader never let us finish. Only
the second can help after a starved shutdown.
Both sweeps rescan by name across EVERY object type and delete what the
bulk call missed. ObjectsDeleteAll's return has already been observed
disagreeing with a by-name scan of the same chart microseconds apart, and
object commands are queued on the chart rather than applied inline, so a
returned count is not evidence the objects are gone.
Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so
the name cannot drift away from the list that cleans it up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
//+------------------------------------------------------------------+
//| EVERY chart-object namespace this EA creates, in ONE list. |
//| |
//| This exists because the list drifted. PurgeChart()'s own comment |
//| said it removed "our namespaced signal arrows plus the status- |
//| label objects" while the code deleted arrows only, and the panel |
//| prefix was swept at OnInit and nowhere else - so a deinit left |
//| the status line and any panel straggler on the chart, which is |
//| exactly the reported symptom. A comment cannot be kept in step |
//| with three scattered call sites; one array can. |
//| |
//| Add a prefix here the moment a new object family is introduced. |
//| Deleting by prefix and never by ObjectsDeleteAll(chart) is |
//| deliberate: a blanket wipe also removes the user's own drawings |
//| and other indicators' objects, which is not acceptable on a |
//| client's chart. |
//+------------------------------------------------------------------+
int WarriorChartPrefixes ( string & out [ ] )
{
feat(chart): on-chart vote readout, and Min_Vote_Open 50 -> 40
THRESHOLD. 40 is a measured correction, not a preference. Once
RankTiersFromOos() replaced the designed tier priors with each model's real
held-out win rate, the vote converges on that win rate - logged 2026-08-18 as
pooled 23-36% across four members on three symbols - so a 50% bar could not be
reached by anything on offer and the ensemble gate fired on 0 of 4,865 OOS
bars. 40 clears the ~34% break-even those same lines report without being
unreachable. The comment says plainly not to copy the number: break-even is a
function of the barrier geometry, so read the gate's own "needs >N%" for the
config in front of you.
READOUT. One line, top-right:
VOTE SELL 37.2% peak 44.1% need 40% 3 voter(s) -> no trade
Every other number on the chart is downstream of the weighted mean the open
threshold is compared against, and that was the one quantity never displayed.
A chart with no arrows could mean the models abstained, the vote was diluted,
or the threshold is unreachable - and telling those apart meant waiting for an
era to end and reading the gate line, which is how the last two sessions went.
PEAK is the part that earns its space. A threshold above what the vote ever
attains can never fire, and that is not knowable from a single bar - it is
precisely the "unreachable gate vs merely unmet gate" confusion this project
has paid for twice. Colour carries the verdict rather than the direction:
green/red ONLY when the vote would actually place an order, grey otherwise.
Green-for-buy would make a below-threshold buy look like a trade, which is the
specific misreading the display exists to prevent.
Guarded on `total > 0` for the same reason the normalization is: Direction()
is inherited as-is by every leaf filter, so without it each filter would write
its own opinion into the one shared label and the last to run would win - the
reader would be looking at an arbitrary member's number believing it was the
vote. Drawn after the +-100 range check, so it shows what the threshold is
actually tested against.
CORNER_RIGHT_UPPER: the status lines, control panel and ensemble panel all
live on the left. Registered in WarriorChartPrefixes() explicitly even though
the "Warrior" catch-all already reaches it - that catch-all exists because the
list has drifted twice, not to make entries optional.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:22:45 -04:00
ArrayResize ( out , 7 ) ;
fix(ensemble): per-member arrow namespaces; ConvLSTM rename; dialog in purge list
The ensemble chart UI had a shared-namespace defect that answered the user
question "what do the arrows represent?" with "a bug": all four members drew
arrows under the same WarSig_<bartime> object names, so the chart showed
whichever member rendered LAST, one member Neutral deleted another member Buy
at the same bar, each member init sweep wiped the arrows the previous member
had just restored, and SaveChartSignals - which rebuilds the sidecar by
SCANNING the chart - persisted every other member arrows into its own history
(the exact cross-model laundering its own header warns about, now happening
BETWEEN ensemble members).
Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_,
WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor
purge and the sidecar scan are all member-scoped, and the tooltip names the
model. Global purges keep matching the bare WarSig_ prefix, which covers all
member namespaces plus old-format leftovers from earlier builds.
Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is
one member; the header is the ensemble) and the CONVLSTM member displays as
ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the
model folder and changing it would orphan every model trained under that path.
Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins
WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now
cover it - it was in neither list, so a dialog starved of its own Destroy()
left its controls on the chart permanently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 18:26:55 -04:00
out [ 0 ] = SIG_ARROW_PREFIX ; // directional signal arrows - bare prefix, so it also
// matches every per-member namespace (WarSig_PAI_ ...)
fix: purge every EA object namespace on init and after deinit teardown
Leftover objects survived deinit because the cleanup list had drifted.
PurgeChart()'s own comment said it removed "our namespaced signal arrows
plus the status-label objects" while the code removed arrows ONLY, and
the panel prefix was swept at OnInit and nowhere else - so an ordinary
deinit left the status line, and any panel straggler, on the chart.
Three scattered call sites and a comment cannot be kept in step. There is
now ONE list - WarriorChartPrefixes() - covering arrows, status label and
panel, and one sweep, WarriorPurgeChartObjects(), used by every path.
Add a prefix there when a new object family appears and every cleanup
picks it up.
Two call sites added:
OnInit, before ANYTHING is drawn (including the status label it would
otherwise delete). Chart objects live in the chart PROFILE, not in the
EA, so they outlive the process: a deinit force-terminated at
MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5
replaced while attached all strand objects no later deinit will ever
own - and deleting the EA's files does not remove them, which is why
they read as corruption. Arrows are included: LoadChartSignals restores
them from their sidecar moments later and already opens with its own
arrow sweep, so this only removes orphans the sidecar does not account
for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds
that sidecar by scanning the chart.
OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control
tree and ClearStatusLabel clears text rather than guaranteeing object
removal; either can leave a straggler and nothing looked afterwards.
Bounded work - three prefix deletes and one object-list scan - so it
respects the ordering rule that keeps the cheap visible cleanup ahead
of the heavy save. Arrows excluded: ShutdownChartCleanup already
persisted and removed them and re-deleting would race that write.
The two are complementary: the deinit sweep closes the ordinary case, the
OnInit purge closes the case where MetaTrader never let us finish. Only
the second can help after a starved shutdown.
Both sweeps rescan by name across EVERY object type and delete what the
bulk call missed. ObjectsDeleteAll's return has already been observed
disagreeing with a by-name scan of the same chart microseconds apart, and
object commands are queued on the chart rather than applied inline, so a
returned count is not evidence the objects are gone.
Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so
the name cannot drift away from the list that cleans it up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
out [ 1 ] = STATUS_LABEL_PREFIX ; // status line background + text (System\StatusLabel.mqh)
out [ 2 ] = WARRIOR_PANEL_PREFIX ; // control panel and its whole control tree
fix(ensemble): per-member arrow namespaces; ConvLSTM rename; dialog in purge list
The ensemble chart UI had a shared-namespace defect that answered the user
question "what do the arrows represent?" with "a bug": all four members drew
arrows under the same WarSig_<bartime> object names, so the chart showed
whichever member rendered LAST, one member Neutral deleted another member Buy
at the same bar, each member init sweep wiped the arrows the previous member
had just restored, and SaveChartSignals - which rebuilds the sidecar by
SCANNING the chart - persisted every other member arrows into its own history
(the exact cross-model laundering its own header warns about, now happening
BETWEEN ensemble members).
Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_,
WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor
purge and the sidecar scan are all member-scoped, and the tooltip names the
model. Global purges keep matching the bare WarSig_ prefix, which covers all
member namespaces plus old-format leftovers from earlier builds.
Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is
one member; the header is the ensemble) and the CONVLSTM member displays as
ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the
model folder and changing it would orphan every model trained under that path.
Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins
WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now
cover it - it was in neither list, so a dialog starved of its own Destroy()
left its controls on the chart permanently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 18:26:55 -04:00
out [ 3 ] = " WarriorAltMap_ " ; // alt-data symbol-mapping dialog (ADM_PREFIX in
// Panel\AltDataMapDialog.mqh - literal here because that
// header is included later in the build order)
fix(chart): a purge that reports "zero leftovers" was only ever checking its own list
2026-08-17 21:58: all three charts hit "Abnormal termination" ~5.3 s into
OnDeinit with NO cleanup-timings line - the teardown was starved again. The
22:00 init purge then removed 993 / 1373 / 1557 stranded objects and reported
ZERO by-name leftovers on every chart, and the charts still came up with
duplicated panels. "Nothing matching our prefixes remains" and "the chart is
clean" are different statements and only the first was being made.
Three changes, in the order they matter:
1. WHY the teardown starved, and it is a gap in ad80e0b. StartLabelCachePrebuild
runs ResizeBuffers + RefreshData over the FULL study window (33,984 bars on
XAUUSD), unchunked, and OnDeinit cannot begin until it returns. Normally a
once-per-run cost. That night SP500 and XAUUSD LSTM were wedged in the "cache
invalidated at era start" loop, which calls it on EVERY Train() call - two
members re-preparing tens of thousands of bars indefinitely. The terminal
closed into that. Guarded now, plus a resumable guard in the prebuild chunk
loop (the tally pass after it is not chunked).
2. Catch-all "Warrior" prefix in WarriorChartPrefixes. Every family this EA
creates is named Warrior* except the arrows (WarSig_), so one bare prefix
covers the three named entries AND anything a rename or a stale .ex5 left
under a name nobody remembers. Still a prefix delete, never
ObjectsDeleteAll(chart) - the user's own drawings are not ours to remove. Does
not defeat skipArrows: "WarSig_" does not start with "Warrior".
3. The init purge now REPORTS the residue it did not claim, by name (up to 12).
Not deleted - an unmatched object may belong to the user or another indicator.
If a Warrior panel is visible and appears in neither the removed count nor
this list, the prefix list has drifted a third time and the name is in the
journal instead of being inferred from a screenshot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 22:07:09 -04:00
//--- CATCH-ALL, and the reason it is here rather than replacing the four above: the list drifted once
//--- (see the header) and on 2026-08-17 22:00 it evidently drifted again - all three charts were
//--- starved on teardown at 21:58 (Abnormal termination, no cleanup-timings line), the init purge then
//--- removed 993/1373/1557 objects and reported ZERO by-name leftovers, and the charts still came up
//--- with duplicated panels. "Nothing left that matches our prefixes" and "the chart is clean" are not
//--- the same statement, and only the first one was ever being checked.
//--- Every family this EA has ever created is named "Warrior..." except the arrows ("WarSig_"), so one
//--- bare prefix covers the three named above AND anything a future rename or a stale .ex5 left behind
//--- under a name nobody remembers. It stays a PREFIX delete, never ObjectsDeleteAll(chart): the
//--- user's own drawings and other indicators' objects are not ours to remove.
//--- Harmless to overlap - deleting an already-deleted name is a no-op - and it does NOT defeat
//--- skipArrows, because "WarSig_" does not start with "Warrior".
out [ 4 ] = " Warrior " ;
fix(deinit): vote arrows survived the cheap sweep, and 5 long loops ignored the stop
Leftover chart objects on long-history charts. Two causes, one of them
introduced by 07aa017.
THE ONE I ADDED. The filtered view's overlay draws up to
SIGNAL_RESCAN_LOOKBACK_BARS vote arrows. OnDeinit's EARLY VISIBLE-UI SWEEP
runs with skipArrows=true, which skips any prefix equal to SIG_ARROW_PREFIX -
and "WarSig_VOTE_..." starts with "WarSig_", so every one of them was skipped
by the one sweep that is cheap enough to always complete. They then sat in the
object list while the two expensive scans that follow walked it: a per-member
SaveChartSignals O(total) scan, then the by-name rescan. On a chart with years
of history that is thousands of extra objects walked twice, inside a teardown
budget measured from the stop REQUEST rather than from OnDeinit's first line.
skipArrows exists because the per-model arrows' sidecar is rebuilt by SCANNING
them off the chart, so they cannot be deleted before that write. Vote arrows
have no sidecar - they are a reconstruction, rebuilt on the next attach - so
nothing is preserving them and they now get their own prefix slot, deleted by
one native call in the first few milliseconds.
THE FIVE LOOPS. A time budget bounds THROUGHPUT, not latency to an unload, and
OnDeinit cannot begin until whatever is in flight returns. These all scaled
with history and none of them checked:
* Training passes 2, 2.5 and 3 yielded only on TRAIN_TIME_BUDGET_MS. Pass 1
has checked IsStopped() all along; the other three never have, and they
are the ones that grow with the bar count. Free to fix - the resume state
is written either way, so a stopped chunk simply is not re-entered.
* PruneDirectionalClusters: the one UNCHUNKED sweep left, once per era over
every bar, with its own header noting that raising the training budget
cannot help its cost. Now bails outright.
* AdvanceChartSignalRestore / AdvanceChartSignalRescan: chunked, but the
rescan runs a full feedForward per bar over up to 5000 bars and the
restore can hold MAX_RESTORED_ARROWS entries. Checked on the same
64-object stride as the clock read, since the check is not free either.
* AdvanceFilteredOverlay (mine, 07aa017) replays Direction() on every
classic filter per bar and had no check at all. Now per bar.
ChartUI.mqh had ZERO shutdown checks across six loops before this.
Nothing was added to the purge path itself: that is the work that must
complete, and an IsStopped() check inside it would abort unconditionally -
IsStopped() is already true by the time OnDeinit runs.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 17:13:50 -04:00
//--- THE FILTERED VIEW's vote arrows, listed SEPARATELY from SIG_ARROW_PREFIX even though the
//--- name matches it, and that separation is the whole point: skipArrows protects the per-model
//--- arrows at deinit because their sidecar is rebuilt by SCANNING them off the chart, so they
//--- cannot be deleted until that write has run. Vote arrows have NO sidecar - they are a
//--- reconstruction, rebuilt from scratch on the next attach - so there is nothing to preserve
//--- them for, and holding them through the early sweep only inflates ObjectsTotal for the two
//--- expensive scans that follow it (SaveChartSignals per member, then the by-name rescan).
//--- On a long-history chart that is thousands of extra objects walked, twice, inside the
//--- teardown budget - which is exactly when the chart cannot afford it. Deleting them here is
//--- one native prefix call in the first few milliseconds.
out [ 5 ] = SIG_VOTE_PREFIX ;
feat(chart): on-chart vote readout, and Min_Vote_Open 50 -> 40
THRESHOLD. 40 is a measured correction, not a preference. Once
RankTiersFromOos() replaced the designed tier priors with each model's real
held-out win rate, the vote converges on that win rate - logged 2026-08-18 as
pooled 23-36% across four members on three symbols - so a 50% bar could not be
reached by anything on offer and the ensemble gate fired on 0 of 4,865 OOS
bars. 40 clears the ~34% break-even those same lines report without being
unreachable. The comment says plainly not to copy the number: break-even is a
function of the barrier geometry, so read the gate's own "needs >N%" for the
config in front of you.
READOUT. One line, top-right:
VOTE SELL 37.2% peak 44.1% need 40% 3 voter(s) -> no trade
Every other number on the chart is downstream of the weighted mean the open
threshold is compared against, and that was the one quantity never displayed.
A chart with no arrows could mean the models abstained, the vote was diluted,
or the threshold is unreachable - and telling those apart meant waiting for an
era to end and reading the gate line, which is how the last two sessions went.
PEAK is the part that earns its space. A threshold above what the vote ever
attains can never fire, and that is not knowable from a single bar - it is
precisely the "unreachable gate vs merely unmet gate" confusion this project
has paid for twice. Colour carries the verdict rather than the direction:
green/red ONLY when the vote would actually place an order, grey otherwise.
Green-for-buy would make a below-threshold buy look like a trade, which is the
specific misreading the display exists to prevent.
Guarded on `total > 0` for the same reason the normalization is: Direction()
is inherited as-is by every leaf filter, so without it each filter would write
its own opinion into the one shared label and the last to run would win - the
reader would be looking at an arbitrary member's number believing it was the
vote. Drawn after the +-100 range check, so it shows what the threshold is
actually tested against.
CORNER_RIGHT_UPPER: the status lines, control panel and ensemble panel all
live on the left. Registered in WarriorChartPrefixes() explicitly even though
the "Warrior" catch-all already reaches it - that catch-all exists because the
list has drifted twice, not to make entries optional.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:22:45 -04:00
//--- The vote readout (CExpertSignalCustom::UpdateVoteReadout). Already covered by the "Warrior"
//--- catch-all above, and listed anyway: the catch-all exists because this list has drifted
//--- twice, not to make entries optional.
out [ 6 ] = VOTE_HUD_PREFIX ;
return 7 ;
fix: purge every EA object namespace on init and after deinit teardown
Leftover objects survived deinit because the cleanup list had drifted.
PurgeChart()'s own comment said it removed "our namespaced signal arrows
plus the status-label objects" while the code removed arrows ONLY, and
the panel prefix was swept at OnInit and nowhere else - so an ordinary
deinit left the status line, and any panel straggler, on the chart.
Three scattered call sites and a comment cannot be kept in step. There is
now ONE list - WarriorChartPrefixes() - covering arrows, status label and
panel, and one sweep, WarriorPurgeChartObjects(), used by every path.
Add a prefix there when a new object family appears and every cleanup
picks it up.
Two call sites added:
OnInit, before ANYTHING is drawn (including the status label it would
otherwise delete). Chart objects live in the chart PROFILE, not in the
EA, so they outlive the process: a deinit force-terminated at
MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5
replaced while attached all strand objects no later deinit will ever
own - and deleting the EA's files does not remove them, which is why
they read as corruption. Arrows are included: LoadChartSignals restores
them from their sidecar moments later and already opens with its own
arrow sweep, so this only removes orphans the sidecar does not account
for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds
that sidecar by scanning the chart.
OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control
tree and ClearStatusLabel clears text rather than guaranteeing object
removal; either can leave a straggler and nothing looked afterwards.
Bounded work - three prefix deletes and one object-list scan - so it
respects the ordering rule that keeps the cheap visible cleanup ahead
of the heavy save. Arrows excluded: ShutdownChartCleanup already
persisted and removed them and re-deleting would race that write.
The two are complementary: the deinit sweep closes the ordinary case, the
OnInit purge closes the case where MetaTrader never let us finish. Only
the second can help after a starved shutdown.
Both sweeps rescan by name across EVERY object type and delete what the
bulk call missed. ObjectsDeleteAll's return has already been observed
disagreeing with a by-name scan of the same chart microseconds apart, and
object commands are queued on the chart rather than applied inline, so a
returned count is not evidence the objects are gone.
Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so
the name cannot drift away from the list that cleans it up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
}
//+------------------------------------------------------------------+
//| Delete every object in those namespaces from a chart, and verify. |
//| |
//| skipArrows leaves the signal arrows alone, for the one caller |
//| that must: a re-init restores arrows from their sidecar and |
//| wiping them here would make them flicker off and back on. |
//| |
//| The rescan is not paranoia. ObjectsDeleteAll's return value was |
//| already observed disagreeing with a by-name scan of the same |
//| chart microseconds apart, and object commands are QUEUED on the |
//| chart rather than applied inline - so "the bulk call returned a |
//| number" is not evidence the objects are gone. Names are collected |
//| before any deletion because deleting while enumerating by index |
//| renumbers the list being walked. |
//+------------------------------------------------------------------+
int WarriorPurgeChartObjects ( long chartID , bool skipArrows , int & leftoverCount )
{
string prefixes [ ] ;
int n = WarriorChartPrefixes ( prefixes ) ;
int removed = 0 ;
leftoverCount = 0 ;
for ( int p = 0 ; p < n ; p + + )
{
if ( skipArrows & & prefixes [ p ] = = SIG_ARROW_PREFIX )
continue ;
int r = ObjectsDeleteAll ( chartID , prefixes [ p ] ) ;
if ( r > 0 )
removed + = r ;
}
//--- Typed-blind rescan across EVERY object type: an earlier version filtered on OBJ_ARROW and was
//--- therefore blind in the same way the bulk delete was, which is how two scans of one chart
//--- disagreed for three sessions.
int total = ObjectsTotal ( chartID , -1 , -1 ) ;
string leftovers [ ] ;
int found = 0 ;
if ( total > 0 )
{
ArrayResize ( leftovers , total ) ;
for ( int i = 0 ; i < total ; i + + )
{
string nm = ObjectName ( chartID , i , -1 , -1 ) ;
for ( int p = 0 ; p < n ; p + + )
{
if ( skipArrows & & prefixes [ p ] = = SIG_ARROW_PREFIX )
continue ;
if ( StringFind ( nm , prefixes [ p ] ) = = 0 )
{
leftovers [ found + + ] = nm ;
break ;
}
}
}
}
for ( int i = 0 ; i < found ; i + + )
ObjectDelete ( chartID , leftovers [ i ] ) ;
leftoverCount = found ;
return removed + found ;
}
2026-07-26 11:17:55 -04:00
//--- Upper bound on arrows restored from a .arrows file (see LoadChartSignals). Purely a guard against a
//--- corrupt header declaring a garbage count - a long converged run legitimately accumulates a few
//--- thousand, so this sits well above that. Restoring is chunked across timer calls regardless, so a
//--- large-but-valid count costs progressive fill-in, never a frozen OnInit.
# define MAX_RESTORED_ARROWS 50000
//--- How many of the MOST RECENT arrows are kept on the chart and in the .arrows sidecar. Arrows would
//--- otherwise accumulate for the life of the model (a converged SP500 H1 run had reached 2896), which
//--- clutters the chart, slows every save/restore, and preserves history nobody scrolls back to. Both
//--- SaveChartSignals (which also DELETES the pruned objects from the chart) and LoadChartSignals select
//--- by TIME, not by scan order - ObjectsTotal() order is arbitrary, so "the last N scanned" would keep a
//--- random subset rather than the newest.
# define MAX_PERSISTED_ARROWS 1000
2026-07-26 12:36:56 -04:00
//--- Manual "rescan" window (see RescanChartSignals): how many of the MOST RECENT bars get re-inferred
//--- from the currently deployed weights when the operator asks for a fresh signal set. Bounded well
//--- below a full StudyPeriods re-render (which can be years of bars and would freeze the one MQL5 chart
//--- thread, same doctrine as MAX_RESTORED_ARROWS/ARROW_RESTORE_BUDGET_MS) - a rescan is meant to replace
//--- stale old arrows with what the model calls on RECENT history, not reproduce the whole training run.
feat(chart): reconstruct the filtered view behind the handover point
Completes the filtered view from 282b535, which only reached forward of
attach. On a multi-hour training run that is the entire time you are looking
at the chart, so the answer to "how would the whole bot have traded" was
blank exactly when it was wanted.
The sweep lives on the AGGREGATE signal, which is the only object holding
every filter. AI members contribute their CACHED per-bar decision from the
era scan - no inference re-runs, the cache already spans the chart - and the
classic ladders are replayed with EvalShift(i), the same mechanism
CSignalMETA's candidate sweep uses and exact because every classic pattern
condition anchors on StartIndex(). Combination is the live one: weighted mean
over voting filters, abstentions out of both sums, against Min_Vote_Open.
THE REPLAY CORRUPTS LIVE JOURNALING IF LEFT UNGUARDED, and this is the part
that is not obvious. Live journaling reads m_active_pattern_long/short from
the PREVIOUS Direction() call. Replaying hundreds of past bars between two
live bars leaves those slots holding whichever bar the sweep stopped on, so
the next live bar journals that pattern under the current timestamp - a
corrupted row in the very table pattern win rates are computed from, which is
now also where vote weights come from. Save/RestoreVoteState() brackets every
replayed call. CSignalMETA gets away without it only because its sweep runs
once, at the first era, before any of that state matters.
TWO SOURCES OF TRUTH, KEPT APART. A reconstruction cannot know the broker
rejected an order - it has no stops level, ATR warm-up or swing-history sync
as they were at that moment - so it is an upper bound: honest about the vote,
optimistic about placement. It therefore stops dead at the handover bar,
which is latched ONCE so later rebuilds cannot creep it forward and start
overwriting real decisions with guesses, and its arrows say "reconstructed
(vote only - order validation not replayed)" in the tooltip. Someone
comparing two arrows either side of that line has to be able to tell which is
a record and which is a replay, and the chart is the only place they look.
Re-armed on any era boundary (summed era counters), because that is when the
answer changes - RankTiersFromOos has just re-derived every tier's vote weight
- and only between sweeps, so a restart cannot leave the previous pass's tail
undrawn. Chunked at 150 bars per timer slice: each bar replays Direction() on
every classic filter, which is real indicator work on the chart thread, and an
unchunked sweep here is the 2026-07-26 arrow-restore freeze waiting to happen.
SIGNAL_RESCAN_LOOKBACK_BARS moves to ExpertSignalCustom.mqh alongside
SIG_ARROW_PREFIX - same include-order reason, and the two rebuilds should
reach the same distance or the raw and filtered views are not comparable.
Known gap: on a classic-only chart the reconstruction is built once and not
refreshed when the hourly DB ranking moves the classic weights.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:05:13 -04:00
//--- MOVED to Expert\ExpertSignalCustom.mqh with the historical filtered overlay, which needs the
//--- same bound and lives in the common base (this header is included later - see SIG_ARROW_PREFIX).
feat(rank): AI models rank their own confidence tiers from held-out outcomes
Closes the caveat 4858507 shipped with: the vote is a confidence percentage,
but only to the extent the pattern weights are measured. AI tier weights sat
at their designed defaults (25/50/75/100) because AI rows only ever arrive
from LIVE journaling, of which a training run produces almost none.
AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed
every scanned bar by ConfidenceTier(), which reads dPrevSignal - and
dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an
entire era's fires were bucketed by one stale, unrelated bar's confidence and
landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0)
T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the
calibration clamp. The clamp was real and was fixed then; this is a second,
independent cause of the identical output that survived that fix untouched -
which is why the log kept reading the same afterwards. Two causes, one symptom.
Now ConfidenceTierFor(adjSig): the bar this iteration actually scored.
WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of
"fill the database during training". The user's own observation is the reason:
a classic Pattern_2 is a fixed geometric condition, so its win rate is
legitimately accumulated over years, but an AI Pattern_2 means "confidence
landed in tier 2" and tier 2 under era 100's weights is a different statement
from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation
is exactly what is wrong here - it would average together models that no
longer exist, while colliding with the per-table row cap and mixing
measured-on-holdout outcomes into the live ledger's own tables. What the DB
actually supplies is a measured win rate per pattern, and pass 3 already
computes that on held-out bars, thousands at a time. So the model ranks itself
once per era, REPLACING rather than accumulating, which makes the weights
describe the current weights by construction.
ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades
BEFORE shrinking, which here would fire on every tier every era and hand all
four the pooled rate - the tiers could never separate and the mechanism would
be inert. Shrinkage is the answer to a small sample; a floor in front of it
means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N
pseudo-observations centred on the model's pooled holdout rate, counted in
EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw
fires can be worth ~12 independent ones. Rounded to the integer, not to the
decade NormalizeWinRate() uses, which would collapse the shrunk tiers back
into one number.
NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard:
weights are computed at the END of era N, so the vote scored during era N was
cast with era N-1's weights. The deploy gate never grades a vote whose weights
were fitted on the bars it is scoring. Residual leakage remains - the same OOS
bars each era under a different model - and is stated in the code rather than
papered over.
Both DB clobber paths are closed: ApplyPatternWeight() declines once
self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by
SelfRanked() - guarding only the tiers would have let the hourly ranking pass
undo half the self-ranking.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- Strength of the Beta prior RankTiersFromOos() shrinks each confidence tier toward the model's
//--- pooled holdout win rate, counted in EFFECTIVE observations (see EffectiveSampleSize). A tier
//--- carries roughly 8-15 of those out of one era's OOS window, so at 10 a tier sits about half on
//--- its own evidence and half on the pool. NOT MIN_TRADES_FOR_WIN_RATE (100): that is the classic
//--- ladders' prior, counted in RAW live trades, and applying it to effective counts would pin all
//--- four tiers to the pooled rate forever.
# define TIER_PRIOR_EFF_N 10.0
2026-07-26 11:17:55 -04:00
//--- Wall-clock budget per chunk of the deferred arrow restore. Same doctrine as TRAIN_TIME_BUDGET_MS:
//--- MQL5 gives a chart ONE thread, so "async" here means small time-boxed slices between which the
//--- terminal can service the panel, the chart and the journal - never a single long blocking pass.
//--- 50ms sits between the training chunk (120ms, already tolerated) and the 500ms timer period, so the
//--- restore completes in a handful of slices while leaving ~90% of each timer window free for the UI.
# define ARROW_RESTORE_BUDGET_MS 50
2026-07-24 11:52:19 -04:00
//--- Max allowed |Δ| between the compute backend's and the pure-MQL5 path's outputs for a model to be
//--- marked MQL5-inference-safe (see ValidateCpuInference). The outputs are bounded sigmoid values;
//--- backend-vs-double summation-order noise is ~1e-6 on the CPU-DLL path (double) and stays well under
//--- this even on a float32 OpenCL backend, while a genuine math/port bug shows up as >0.01. Fails safe.
# define CPU_INFERENCE_MAX_DIFF 1.0e-3
feat(chart): filtered view - one arrow per trade the bot would actually take
Adds DrawUnfilteredSignals (default OFF) and, with it off, replaces the
per-model arrow layer with the decision the EA would really have made.
THE FILTERED ARROW IS DRAWN AT THE ORDER, NOT AT THE THRESHOLD. Clearing
Min_Vote_Open is not the same as trading: a setup can pass the vote and still
never reach the broker (invalid SL/TP, stops-level, ATR warm-up, unsynced
swing history), and every one of those lands in OpenParams' failure branch.
So DrawVoteArrow() fires only after the order parameters validate, and the
failure branch withdraws any arrow already standing on that bar. One arrow is
one entry the EA would have placed - carrying the vote, the threshold it
cleared, and the SL/TP the order would have had.
Classic signals now draw too, under their own name and weight, so a chart
running MA/RSI/MACD/Ichimoku alongside the nets reads the same way an
ensemble chart does. They can only be drawn from the aggregate's once-per-bar
pass, because unlike the AI members they have no cached per-bar scan.
Two subtleties that would each have produced a quietly wrong chart:
- The raw classic draw sits AFTER filter.Direction(), not beside the
journaling block. GetActivePattern*() are CONSUMING reads holding the
PREVIOUS evaluation - "one tick later", which at Expert_EveryTick=false is
one BAR later. Keyed off those and placed at StartIndex(), every classic
arrow would have been drawn one bar early, which on a chart is
indistinguishable from a model that genuinely leads. Peek*() accessors
(non-consuming) let pattern, weight and bar come from one evaluation.
- CExpertSignalAIBase::DrawObject() early-returns instead of gating its five
call sites, so the switch cannot be honoured in three passes and missed in
the fourth. Its delete counterparts stay ungated so flipping the input off
and rescanning clears the raw layer rather than stranding it.
SIG_ARROW_PREFIX and g_signalsVisible move from ExpertSignalAIBase.mqh down
to ExpertSignalCustom.mqh - the nearest common ancestor - because the classic
signals cannot see the AI header (it is included later in Warrior_EA.mq5).
The vote layer gets SIG_VOTE_PREFIX under the same bare prefix, so
WarriorChartPrefixes()' purge still reaches every arrow without knowing they
exist.
NOT YET BUILT: the reconstructed history behind attach. Filtered arrows
currently start where the EA starts. See the next commit.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:41:28 -04:00
//--- g_signalsVisible is declared in ExpertSignalCustom.mqh now, alongside SIG_ARROW_PREFIX.
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- FILTER-BASED auto-tuner constants (see TuneIndicatorsByFilter). The GA_* / TUNE_POP_* knobs that
//--- lived here are gone with the genetic search they configured.
//--- MI_BINS: equal-frequency bins the feature column is discretised into before the joint histogram.
//--- Mutual information is biased upward as bins increase (each bin holds fewer samples, so noise looks
//--- like structure); 8 bins against MI_SAMPLE_BARS samples keeps ~250 samples per bin per class, which
//--- is comfortably in the regime where that bias is small and equal across candidates - and equal is
//--- what matters, since this score is only ever used to RANK.
# define MI_BINS 8
//--- Bars sampled (evenly spaced across the in-sample window) per candidate evaluation. The whole cost of
//--- tuning is candidates x this x features, so it is the one number that trades accuracy for time.
# define MI_SAMPLE_BARS 2000
# define MI_MIN_SAMPLES 200
2026-08-02 12:25:20 -04:00
//--- Eras the MI diagnostics may wait for the cross-asset panel before reporting without it. Three is
//--- enough for the terminal to finish synchronising auxiliary symbols on a live chart, and short
//--- enough that a tester run - where an un-downloaded reference symbol is permanently absent - is not
//--- left without diagnostics at all.
# define MI_REPORT_MAX_DEFERRALS 3
2026-08-02 08:12:47 -04:00
//--- Largest |k| the label-alignment scan uses. Every MI sample is padded by MiShiftPad() - which is at
//--- least this - at BOTH ends REGARDLESS of the offset being requested, so that every build enumerates
//--- the IDENTICAL bar set with the IDENTICAL stride. That is what makes two builds comparable row by
//--- row. Padding by |offset| instead (as this did until 2026-08-02) shifted the offset build's starting
//--- bar, so the positive control paired rows that were offset+pad apart rather than offset apart: it
//--- reported 0.00307 nats for a pair it called "24 bars apart", which is the value for 48 bars, failed
//--- its own 5x gate, and printed "every mutual-information figure above is void" over perfectly sound
//--- measurements. Verified against an independent computation in research/test_mi_control.py.
# define MI_ALIGN_MAX_SHIFT 5
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Coordinate-descent passes. The second pass lets a parameter re-optimise against what the others
//--- moved to; the loop breaks early as soon as a pass changes nothing, so this is a ceiling, not a cost.
# define MI_TUNE_PASSES 2
diag(autotune): five permutations was still a coin flip - use a real test
The 5-draw z-score shipped an hour ago disproved itself on its first run.
All four charts scored the IDENTICAL 0.00401 nats on identical features
and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two
"AT THE NOISE FLOOR", two "a real association", same data. The entire
swing came from estimating the null's spread from five draws, where the
standard deviation of the standard-deviation estimate is ~35%: the
denominator was noisier than the effect it was judging.
Replaced with an empirical permutation test. 200 draws, p counted by rank
with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never
reported as exactly zero - no normality assumption and no spread to
estimate. The strongest single column is tested against the null
distribution OF THE MAXIMUM, which corrects for scoring 26 features at
once by construction and is far less conservative than Bonferroni.
Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI
and runs ONCE for the whole test - every draw reuses that sample and costs
a relabel plus 26 histogram passes, not 2000 feature extractions. The
coordinate sweep still calls the combined form, which is correct there:
each candidate changes the indicator settings, so its features really do
have to be re-extracted.
The verdict line keeps both questions apart and prints both answers: the
p-value for "is it real", the excess as a percentage of H(Y) for "is it
big enough to trade". At n=2000 those can disagree, and collapsing them
into one word is how a worthless effect gets called a discovery.
Compiles 0 errors / 0 warnings. Build tag permtest-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
//--- MI_NOISE_PERMUTATIONS: draws from the null distribution used to test the observed score. Sets the
//--- resolution of the empirical p-value, which can never go below 1/(B+1) - so 200 draws can report
//--- "p<=0.005" and no finer, which is ample for a yes/no on whether a feature set carries signal.
//--- Cheap because BuildMiSample runs ONCE and every draw reuses it (see ScoreMiSample); the cost is a
//--- relabel and 26 histogram passes, not 2000 feature extractions.
//--- Two earlier values were wrong and both are instructive. ONE shuffle answers "is this above a coin
//--- flip's worth of noise" rather than "is this above noise". FIVE looked principled but was not: on
//--- 2026-08-01 all four charts scored the identical 0.00401 nats on identical features and identical
//--- labels, and reported z of +1.3, +2.0, +4.0 and +4.7 - two "noise floor", two "real". With five
//--- draws the standard deviation of the standard-deviation estimate is ~35%, so the denominator of that
//--- z was noisier than the effect it was judging. Counting ranks avoids estimating a spread at all.
# define MI_NOISE_PERMUTATIONS 200
diag: MI feature-lag profile - close the blind spot in every MI verdict so far
BuildMiSample samples features from ONE bar. So every "MI is at the noise
floor" result this codebase has produced - including yesterday's p=0.18 on
SP500 H1 - described the ENTRY BAR's 31 features only, while the network is
fed 20 bars of them. If information lived at lag 7 and not lag 0, the report
would have said "no signal" while the model could still learn. The diagnostic
we have been making decisions on had a blind spot exactly the width of the
input vector.
Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as
the existing labelBarOffset and is not interchangeable with it. Shifting the
LABEL changes which trade is predicted, so at any non-zero offset the
features sit inside the labelled window and the score is lookahead - that is
precisely what the alignment scan measures and correctly reports (4.7x more
knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label
pinned to the entry bar, so every row stays causal.
ReportFeatureLagProfile() then scores k = 0..historyBars against the same
block-permutation null and reports the deepest lag that clears it - the
lookback the data supports, versus the 20 that was picked by hand and never
measured. The null is redrawn PER LAG: finite-sample MI bias moves with the
realised class counts and bin occupancy, and different rows survive the
validity checks at each lag, so one shared floor would be right for lag 0 and
wrong everywhere else. Draw count is reduced accordingly (40, not 200) since
cost is draws x historyBars; this figure decides a lookback, never a trade.
MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that
makes two builds comparable row by row.
Read-only - no input, topology or label change, so no retrain. Both builds
0/0. Build tag lag-profile-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
//--- Lag-profile draws, deliberately far below MI_NOISE_PERMUTATIONS: the profile redraws its null at
//--- EVERY lag (see ReportFeatureLagProfile for why a shared floor would be wrong), so the cost is
//--- draws x historyBars, not draws. 40 resolves a p of 0.05 to within about one draw, which is all a
//--- lookback decision needs - this figure never gates a trade.
# define MI_LAG_PERMUTATIONS 40
fix: correct the lag profile across lags too - it contradicted itself
3271f1e tested each of ~21 lags against its OWN null at alpha 0.05 and starred
whatever cleared. That is about one false positive per run before any signal
exists, and because neighbouring lags share nearly their entire feature window
the false positives arrive in CLUSTERS that read like a hump.
It did exactly that on SP500 H1, twice in one afternoon on identical data:
13:55 nothing clears at any lag headline MI p=0.4478
16:22 k6/k10/k12/k16 starred, headline MI p=0.8756, observed
"information survives to lag 16" BELOW its own null mean
Same 31 features, same 2009 samples, same 287 blocks, cross-asset absent in
both - so this was not two different measurements. Non-replication on identical
data is the signature of an uncorrected multiple comparison, and acting on the
second run would have pinned the lookback to 17 off noise.
Galling detail: 04ee2e1 had just added exactly this correction to the
barrier-geometry scan one function below. The rigorous bar went on the report
with 6 candidates and the naive one stayed on the report with 21.
So the lag profile now uses the same construction as the geometry winner test:
one draw from every lag, keep the largest, repeat; a lag clears only by beating
that distribution. Draws centred leave-one-out to match how the observed excess
is centred. Independence across lags overstates the spread of the maximum
(neighbours share their window), so it errs toward rejecting.
Also: the positive branch now says to re-run before acting, because one run of
this report has demonstrably not been a result; and MI_LAG_MAX_PROFILE caps the
retained-draw matrix rather than trusting a derived m_historyBars.
Read-only diagnostic. No input, topology or label change: no retrain, and a
training run already in flight stays valid.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:28:57 -04:00
//--- Per-lag significance, applied against the null of the MAXIMUM over lags rather than against each
//--- lag's own null. The first version did the latter and it was wrong: ~21 lags at 0.05 stars one lag
//--- per run before any signal exists, and on SP500 H1 that produced two opposite verdicts on identical
//--- data hours apart. The alpha stays 0.05; what changed is the null it is measured against.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far
BuildMiSample samples features from ONE bar. So every "MI is at the noise
floor" result this codebase has produced - including yesterday's p=0.18 on
SP500 H1 - described the ENTRY BAR's 31 features only, while the network is
fed 20 bars of them. If information lived at lag 7 and not lag 0, the report
would have said "no signal" while the model could still learn. The diagnostic
we have been making decisions on had a blind spot exactly the width of the
input vector.
Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as
the existing labelBarOffset and is not interchangeable with it. Shifting the
LABEL changes which trade is predicted, so at any non-zero offset the
features sit inside the labelled window and the score is lookahead - that is
precisely what the alignment scan measures and correctly reports (4.7x more
knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label
pinned to the entry bar, so every row stays causal.
ReportFeatureLagProfile() then scores k = 0..historyBars against the same
block-permutation null and reports the deepest lag that clears it - the
lookback the data supports, versus the 20 that was picked by hand and never
measured. The null is redrawn PER LAG: finite-sample MI bias moves with the
realised class counts and bin occupancy, and different rows survive the
validity checks at each lag, so one shared floor would be right for lag 0 and
wrong everywhere else. Draw count is reduced accordingly (40, not 200) since
cost is draws x historyBars; this figure decides a lookback, never a trade.
MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that
makes two builds comparable row by row.
Read-only - no input, topology or label change, so no retrain. Both builds
0/0. Build tag lag-profile-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
# define MI_LAG_ALPHA 0.05
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- WHICH TARGET BuildMiSample() scores the features against. The barrier class is the shipped training
//--- target; the excursion targets exist to answer a question the barrier label cannot, and that every MI
//--- verdict in this project so far has silently conflated.
//---
//--- "Optimal SL/TP" decomposes into two predictions that behave nothing alike:
//--- HOW FAR price travels (the excursions) - essentially a volatility question, and volatility
//--- clustering is about the most robust regularity there is, so expect this to be predictable.
//--- WHICH barrier is reached first (the asymmetry) - direction, which is what the barrier label
//--- measures and what has come back at the noise floor every time.
//--- Expectancy comes ONLY from the second. The first buys position sizing and drawdown control, which
//--- is worth having under prop-firm limits but is not an edge - exit management tested against RANDOM
//--- entries moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT.
//--- Measuring them separately is the point: if size clears and asymmetry does not, the deliverable is
//--- risk control and we should stop looking for edge in the exit.
fix: normalise the asymmetry target - the raw one is confounded by volatility
Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three;
raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500
(p=0.1045). That looked like the first directional signal this project has
found. It probably is not, and the test as built could not tell.
(up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its
null on every instrument - and the directional part is symmetric noise eps, then
up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A
pure volatility predictor scores positive MI against a 3-bin (up-dn) while
carrying no directional information at all. Crucially that confound REPLICATES,
so reproducing on two instruments is not evidence against it - and the effect
sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries
~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a
leaked fraction of the volatility signal, not an independent one.
So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only
target a directional claim may rest on. The verdict now separates the cases and
NAMES the confound when raw clears while normalised does not, instead of
reporting the raw line as a finding.
Two bugs of mine in the same block, both caught by output rather than review:
- The derived-geometry line had a MISORDERED argument list: it printed
"stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the
multiple and the multiple as the quantile. Real values were 2.61 stop /
8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen.
- THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25
"so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the
stop - hit three times in four. The printed reachability said exactly that
("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate.
This is the entire reason reachability is measured and printed rather than
assumed.
Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop
still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And
bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the
end of the sorted array.
The geometry from the previous run is NOT usable and the asymmetry result is
unresolved, not established. Both are decided by the next run.
FORCES A FULL RETRAIN (the stop quantile changes every label).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
//--- BARRIER DERIVATION. The stop sits at a HIGH quantile of adverse travel, so only the minority of bars
//--- whose adverse excursion exceeds it ever reach it; the target at the MEDIAN of favourable travel, so
//--- it is reached about half the time inside the horizon by construction. Neither number creates
//--- expectancy (chance precision equals break-even at every geometry) - they make the target reachable
//--- and the stop survivable, which the enum grid {2,3} x {2,3,4,6,8,10} could only do by luck.
//---
//--- THE STOP QUANTILE WAS 0.25 AND THAT WAS BACKWARDS. q25 means 75% of bars exceed the stop, i.e. it is
//--- hit three times in four - the opposite of "ordinary noise does not reach it". Caught by the measured
//--- reachability line the derivation prints ("stop on 75.0% of bars"), which is the entire reason that
//--- figure is reported instead of assumed. A quantile is a threshold, not a rate: to be rarely reached a
//--- stop must sit ABOVE most of the distribution.
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed
The derivation read the stop from q75 of ADVERSE travel and the target from q50
of FAVOURABLE travel. Over one horizon those distributions are broadly the same
shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no
matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1
payoff needing 64.3%. That was never a measurement, it was two mismatched
constants.
The reachability line printed beside it - "target on 50.0% of bars, stop on
25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm
anything, and it read as validation.
WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width;
ratio is EV-neutral (a driftless walk reaches +m before -k with probability
k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread
is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per
unit of travel. So:
RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%.
SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST,
taking the first rung whose implied 2x target is still reached often
enough to be a trainable class.
That last clause is the difference from the min-reward:risk raise removed in
2026-08-09, which forced target = 2 x stop with NO reachability test, landed on
6.66*ATR reachable on 3.3% of bars, and trained the model to predict something
that essentially never happened. Same ratio; the scale now retreats until the
data says the target is attainable. Every rung is logged.
LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's
"best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it
pinned to the top rung. A recommendation landing exactly on the edge of its own
search space is a boundary, not a finding: it cannot tell "5 ATR is optimal"
from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else
needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and
the horizon constraints (decided >= 60%, reachability floor) now bind instead of
a constant.
THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked
the configured pair up in its integer grid, and DeriveBarrierGeometry produces
CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so
cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2
scores -1.00000", which reads as a catastrophic score and actually means "never
evaluated". Worse, the grid skipped target<stop entirely because it "inverts the
trade's whole premise" - while the derivation was shipping exactly that. The
incumbent is now always scored as a peer (never crowned; it is already in force
and is not an enum pairing the scan could adopt).
BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless
SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was
62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp
better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and
a loss at (SL + spread), matching the expectancy scan's convention exactly so
the two reports cannot disagree.
It also feeds FitDirConfThreshold, which is the correctness half: the operating
point subtracts break-even from precision, so the frictionless figure made every
candidate threshold look better by the width of the spread - 2.2pp against a
measured edge of 2.3pp, i.e. very nearly all of it.
Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD".
Forces a full relabel and retrain. Requested.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
//--- 2026-08-17 REPLACED. The pair above read the stop from q75 of ADVERSE travel and the target from
//--- q50 of FAVOURABLE travel. Over one horizon those two distributions are broadly the same shape, so
//--- q75 > q50 MECHANICALLY - the target came out smaller than the stop no matter what the market did.
//--- SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1 payoff needing a 64.3% win rate. That was never
//--- a measurement, it was two mismatched constants, and the reachability line printed beside it
//--- ("target on 50.0% of bars, stop on 25.0%") is exactly 1-q50 and 1-q75 - tautological, and unable to
//--- disconfirm anything. ReportBarrierGeometryScan will not even SCORE a pairing with the target inside
//--- the stop ("inverts the trade's whole premise"), so the one check that could have caught it was
//--- structurally blind to the geometry that shipped.
//---
//--- WIDTH AND RATIO ARE INDEPENDENT, and only one of them pays.
//--- EV per trade = edge x width, width = stop + target. Ratio is EV-NEUTRAL: chance precision equals
//--- break-even at every ratio (a driftless walk reaches +m before -k with probability k/(k+m), which
//--- IS the break-even rate). What the ratio DOES buy is the shape of the loss distribution, and what
//--- WIDTH buys is cost efficiency - the spread is a fixed 0.047*ATR on this instrument, so a 4.77*ATR
//--- width pays it 21 times per unit of travel where a 9.2*ATR width pays it 11.
//--- So the ratio is a POLICY choice and the scale is a MEASUREMENT. They are set separately below.
//---
//--- RATIO: user policy, 1 risk : 2 reward. Break-even 33.3%.
# define BARRIER_TARGET_RR 2.0
//--- SCALE: the stop is still read off the ADVERSE distribution, so it stays measured and survivable.
//--- The quantile is chosen from this ladder, WIDEST FIRST, taking the first one whose implied target
//--- (RR x stop) is still reached often enough to be a trainable class. That is the honest version of
//--- what the removed min-reward:risk raise got wrong in 2026-08-09: it forced target = 2 x stop with no
//--- reachability test at all, landed on 6.66*ATR reachable on 3.3% of bars, and trained the model to
//--- predict something that essentially never happened. Same ratio here, but the SCALE now retreats
//--- until the data says the target is attainable, instead of the ratio overriding the data.
# define BARRIER_SL_QUANTILE_COUNT 7
const double BARRIER_SL_QUANTILE_LADDER [ BARRIER_SL_QUANTILE_COUNT ] =
{ 0.90 , 0.85 , 0.80 , 0.75 , 0.70 , 0.60 , 0.50 } ;
//--- Retained as the fallback quantile when no rung clears the reachability floor.
fix: normalise the asymmetry target - the raw one is confounded by volatility
Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three;
raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500
(p=0.1045). That looked like the first directional signal this project has
found. It probably is not, and the test as built could not tell.
(up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its
null on every instrument - and the directional part is symmetric noise eps, then
up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A
pure volatility predictor scores positive MI against a 3-bin (up-dn) while
carrying no directional information at all. Crucially that confound REPLICATES,
so reproducing on two instruments is not evidence against it - and the effect
sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries
~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a
leaked fraction of the volatility signal, not an independent one.
So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only
target a directional claim may rest on. The verdict now separates the cases and
NAMES the confound when raw clears while normalised does not, instead of
reporting the raw line as a finding.
Two bugs of mine in the same block, both caught by output rather than review:
- The derived-geometry line had a MISORDERED argument list: it printed
"stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the
multiple and the multiple as the quantile. Real values were 2.61 stop /
8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen.
- THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25
"so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the
stop - hit three times in four. The printed reachability said exactly that
("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate.
This is the entire reason reachability is measured and printed rather than
assumed.
Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop
still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And
bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the
end of the sorted array.
The geometry from the previous run is NOT usable and the asymmetry result is
unresolved, not established. Both are decided by the next run.
FORCES A FULL RETRAIN (the stop quantile changes every label).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
# define BARRIER_SL_QUANTILE 0.75
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in
3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever
chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of
eleven guesses is not deriving anything.
WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It
ranks pairings by how predictable their OUTCOME is - a question about direction.
The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing
absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE
scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x.
Hence the scan failing its own gate on every run, and its "winner" wandering
2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is
strongly measurable, so derive the geometry from that instead.
stop = q25 of measured ADVERSE travel (ordinary noise does not reach it)
target = q50 of measured FAVOURABLE travel (reached ~half the time, by
construction, inside the horizon)
Continuous, in ATR units, superseding the enum multiples. Reachability ("target
on X% of bars, stop on Y%") and the implied break-even are printed so the choice
is auditable rather than trusted.
FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the
horizon with the target (first-passage time grows with the band) and the
excursions are measured OVER the horizon, so target -> horizon -> excursions ->
target is a real loop - deriving once sizes the target from travel measured
under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at
3 passes, and says so if it does not settle.
Does NOT create expectancy, and the log says as much: chance precision equals
break-even at every geometry (m/(m+k) on both sides). It buys a target the
market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio
forces a target the market rarely reaches, it WARNS rather than overriding -
the ratio is the user's risk policy, so the honest move is to state its cost.
That is the collision that once rejected 100% of setups.
Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg
files written earlier today still load (their length guard finds no doubles) and
a model that carries them was trained on them and never re-derives.
Also fixes a message from e5ceed6 that claimed "this model resumed from disk"
unconditionally - it printed above a "seeding era 0" line on a brand-new model,
because the branch fires whenever the cache is not built, which is equally true
before a fresh model's first prebuild. A diagnostic that misreports its own
trigger is worse than one that says nothing: it gets quoted back as evidence.
FORCES A FULL RETRAIN (labels change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- Below this many resolved excursions the quantiles are too noisy to key a training target on, and the
//--- configured multiples stand.
# define BARRIER_DERIVE_MIN_SAMPLES 500
feat: first-passage ladder + expectancy scan - price every geometry, not just the chosen one
Corrects the premise of the previous plan. Break-even is NOT a ceiling.
If the model shifts the win probability on the bars it selects from
p0 = m/(m+k) to p0 + d, then
EV = (p0+d)*k - (1-p0-d)*m = d*(k+m)
because p0*k - (1-p0)*m is zero by construction. The stop:target RATIO
is expectancy-neutral - a punishing break-even is exactly repaid by the
payoff - and only the real edge d and the TOTAL WIDTH (k+m) move EV.
Width matters because the spread is charged once per trade however wide
the barriers are, so a narrow barrier spends much of its own range on
costs. DeriveBarrierGeometry's own comment already said the ratio buys
nothing; the objective just never followed from it.
Blocker this had to solve first: m_excUpCache/m_excDownCache hold only
MAXIMUM travel each way, and a maximum cannot say which side was
reached FIRST - so any geometry other than the walked one was
undecidable on precisely the bars where both barriers were touched,
~28% of the sample.
- BARRIER_LADDER: per bar, the first-touch AGE for 8 travel distances
in each direction, filled during the walk the labels already run.
Cursors keep it O(1) amortised per walked bar rather than 16
comparisons. Levels are travel FROM ENTRY, not barrier prices, so one
ladder serves both directions and the spread is applied analytically
when a level converts back to an SL/TP multiple - storing prices
would need four ladders and bake today's spread into the cache.
Sized, invalidated and validity-gated with the label caches.
- ReportGeometryExpectancyScan: every ladder pair priced exactly off
that cache - width in ATR and in SPREADS (cost efficiency, knowable
without knowing d), break-even, both base rates, the share of bars
resolved inside the horizon, and EV per unit of edge. Compares the
widest resolvable pair against the quantile rule's pick.
MEASUREMENT ONLY - the quantile rule still chooses. Nothing here can
measure d, and width buys nothing if the wider target is less
predictable. Base rates are printed beside each break-even because a
persistent gap is DRIFT and must not be credited to the model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 12:59:18 -04:00
//--- FIRST-PASSAGE LADDER. m_excUpCache/m_excDownCache hold only the MAXIMUM travel each way, and a
//--- maximum cannot say which side was reached FIRST - so a candidate geometry other than the one the
//--- walk actually used is undecidable on exactly the bars where both barriers were touched, which is
//--- ~28% of the sample. Recording the first-touch AGE for a ladder of travel distances makes any pair
//--- of ladder levels evaluable exactly, with no re-walk: long wins iff its target was touched and the
//--- stop either never was or was touched later.
//--- Levels are TRAVEL FROM ENTRY (the bar's close), not barrier prices, so one ladder serves both
//--- directions and the spread is applied analytically when a level is converted back to an SL/TP
//--- multiple: a long fills at close+spread, so reaching its target needs travel = reward + spread and
//--- its stop trips at travel = risk - spread. Storing barrier prices instead would need four ladders
//--- and would bake today's spread into the cache.
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed
The derivation read the stop from q75 of ADVERSE travel and the target from q50
of FAVOURABLE travel. Over one horizon those distributions are broadly the same
shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no
matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1
payoff needing 64.3%. That was never a measurement, it was two mismatched
constants.
The reachability line printed beside it - "target on 50.0% of bars, stop on
25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm
anything, and it read as validation.
WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width;
ratio is EV-neutral (a driftless walk reaches +m before -k with probability
k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread
is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per
unit of travel. So:
RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%.
SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST,
taking the first rung whose implied 2x target is still reached often
enough to be a trainable class.
That last clause is the difference from the min-reward:risk raise removed in
2026-08-09, which forced target = 2 x stop with NO reachability test, landed on
6.66*ATR reachable on 3.3% of bars, and trained the model to predict something
that essentially never happened. Same ratio; the scale now retreats until the
data says the target is attainable. Every rung is logged.
LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's
"best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it
pinned to the top rung. A recommendation landing exactly on the edge of its own
search space is a boundary, not a finding: it cannot tell "5 ATR is optimal"
from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else
needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and
the horizon constraints (decided >= 60%, reachability floor) now bind instead of
a constant.
THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked
the configured pair up in its integer grid, and DeriveBarrierGeometry produces
CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so
cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2
scores -1.00000", which reads as a catastrophic score and actually means "never
evaluated". Worse, the grid skipped target<stop entirely because it "inverts the
trade's whole premise" - while the derivation was shipping exactly that. The
incumbent is now always scored as a peer (never crowned; it is already in force
and is not an enum pairing the scan could adopt).
BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless
SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was
62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp
better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and
a loss at (SL + spread), matching the expectancy scan's convention exactly so
the two reports cannot disagree.
It also feeds FitDirConfThreshold, which is the correctness half: the operating
point subtracts break-even from precision, so the frictionless figure made every
candidate threshold look better by the width of the spread - 2.2pp against a
measured edge of 2.3pp, i.e. very nearly all of it.
Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD".
Forces a full relabel and retrain. Requested.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
//--- CEILING REMOVED 2026-08-17. The ladder used to stop at 5.00 and the expectancy scan's "best
//--- resolvable pair on width alone" came back as stop 5.05 / target 4.95 - i.e. it pinned to the top
//--- rung. A recommendation that lands exactly on the edge of its own search space is not a finding, it
//--- is a boundary, and it cannot distinguish "5 ATR is optimal" from "5 ATR is all we let it consider".
//--- Since EV = edge x width with no upper bound in the arithmetic, the only thing that should stop the
//--- search is the HORIZON failing to resolve the trade - which the scan already tests (decided >= 60%)
//--- and the derivation now tests too (reachability floor). Let those bind instead of a constant.
//--- Extended to 20*ATR with widening spacing: 4x the old reach, and the constraints self-limit.
//--- Cost: m_ladderUpAt/m_ladderDownAt are bars x COUNT ints each (~1.3MB at 11k bars), and the
//--- excursion head widens to 2 x COUNT outputs. Both scale linearly and both are parameterised, so
//--- nothing here needs a matching edit elsewhere.
# define BARRIER_LADDER_COUNT 14
const double BARRIER_LADDER [ BARRIER_LADDER_COUNT ] =
{ 0.50 , 0.75 , 1.00 , 1.50 , 2.00 , 3.00 , 4.00 , 5.00 , 6.50 , 8.00 , 10.00 , 13.00 , 16.00 , 20.00 } ;
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in
3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever
chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of
eleven guesses is not deriving anything.
WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It
ranks pairings by how predictable their OUTCOME is - a question about direction.
The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing
absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE
scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x.
Hence the scan failing its own gate on every run, and its "winner" wandering
2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is
strongly measurable, so derive the geometry from that instead.
stop = q25 of measured ADVERSE travel (ordinary noise does not reach it)
target = q50 of measured FAVOURABLE travel (reached ~half the time, by
construction, inside the horizon)
Continuous, in ATR units, superseding the enum multiples. Reachability ("target
on X% of bars, stop on Y%") and the implied break-even are printed so the choice
is auditable rather than trusted.
FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the
horizon with the target (first-passage time grows with the band) and the
excursions are measured OVER the horizon, so target -> horizon -> excursions ->
target is a real loop - deriving once sizes the target from travel measured
under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at
3 passes, and says so if it does not settle.
Does NOT create expectancy, and the log says as much: chance precision equals
break-even at every geometry (m/(m+k) on both sides). It buys a target the
market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio
forces a target the market rarely reaches, it WARNS rather than overriding -
the ratio is the user's risk policy, so the honest move is to state its cost.
That is the collision that once rejected 100% of setups.
Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg
files written earlier today still load (their length guard finds no doubles) and
a model that carries them was trained on them and never re-derives.
Also fixes a message from e5ceed6 that claimed "this model resumed from disk"
unconditionally - it printed above a "seeding era 0" line on a brand-new model,
because the branch fires whenever the cache is not built, which is equally true
before a fresh model's first prebuild. A diagnostic that misreports its own
trigger is worse than one that says nothing: it gets quoted back as evidence.
FORCES A FULL RETRAIN (labels change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- The derivation is a FIXED-POINT ITERATION, not a one-shot. ComputeBarrierHorizonBars scales the
//--- horizon with the target (first-passage time grows with the band), and the excursions are measured
//--- OVER that horizon - so target -> horizon -> excursions -> target is a loop. Deriving once would set
//--- the target from travel measured under the OLD horizon and quietly mis-state it. Re-measure until the
//--- multiples stop moving, capped so a pathological oscillation cannot spin forever.
fix: normalise the asymmetry target - the raw one is confounded by volatility
Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three;
raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500
(p=0.1045). That looked like the first directional signal this project has
found. It probably is not, and the test as built could not tell.
(up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its
null on every instrument - and the directional part is symmetric noise eps, then
up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A
pure volatility predictor scores positive MI against a 3-bin (up-dn) while
carrying no directional information at all. Crucially that confound REPLICATES,
so reproducing on two instruments is not evidence against it - and the effect
sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries
~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a
leaked fraction of the volatility signal, not an independent one.
So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only
target a directional claim may rest on. The verdict now separates the cases and
NAMES the confound when raw clears while normalised does not, instead of
reporting the raw line as a finding.
Two bugs of mine in the same block, both caught by output rather than review:
- The derived-geometry line had a MISORDERED argument list: it printed
"stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the
multiple and the multiple as the quantile. Real values were 2.61 stop /
8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen.
- THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25
"so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the
stop - hit three times in four. The printed reachability said exactly that
("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate.
This is the entire reason reachability is measured and printed rather than
assumed.
Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop
still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And
bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the
end of the sorted array.
The geometry from the previous run is NOT usable and the asymmetry result is
unresolved, not established. Both are decided by the next run.
FORCES A FULL RETRAIN (the stop quantile changes every label).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
# define BARRIER_DERIVE_MAX_PASSES 5
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in
3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever
chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of
eleven guesses is not deriving anything.
WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It
ranks pairings by how predictable their OUTCOME is - a question about direction.
The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing
absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE
scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x.
Hence the scan failing its own gate on every run, and its "winner" wandering
2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is
strongly measurable, so derive the geometry from that instead.
stop = q25 of measured ADVERSE travel (ordinary noise does not reach it)
target = q50 of measured FAVOURABLE travel (reached ~half the time, by
construction, inside the horizon)
Continuous, in ATR units, superseding the enum multiples. Reachability ("target
on X% of bars, stop on Y%") and the implied break-even are printed so the choice
is auditable rather than trusted.
FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the
horizon with the target (first-passage time grows with the band) and the
excursions are measured OVER the horizon, so target -> horizon -> excursions ->
target is a real loop - deriving once sizes the target from travel measured
under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at
3 passes, and says so if it does not settle.
Does NOT create expectancy, and the log says as much: chance precision equals
break-even at every geometry (m/(m+k) on both sides). It buys a target the
market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio
forces a target the market rarely reaches, it WARNS rather than overriding -
the ratio is the user's risk policy, so the honest move is to state its cost.
That is the collision that once rejected 100% of setups.
Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg
files written earlier today still load (their length guard finds no doubles) and
a model that carries them was trained on them and never re-derives.
Also fixes a message from e5ceed6 that claimed "this model resumed from disk"
unconditionally - it printed above a "seeding era 0" line on a brand-new model,
because the branch fires whenever the cache is not built, which is equally true
before a fresh model's first prebuild. A diagnostic that misreports its own
trigger is worse than one that says nothing: it gets quoted back as evidence.
FORCES A FULL RETRAIN (labels change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
# define BARRIER_DERIVE_TOLERANCE 0.05
fix(geometry): the reachability floor measured the WRONG WINDOW - my bug from bc57aca, and it cost real width
RECONCILED: the derivation reported "target reached on 17.7% of bars" while the
label cache reported Buy on 35.9%. Nothing was broken. They measure different
windows, and both are correct:
EXCURSION window ~12 bars (the SWING MEDIAN) - what m_excUpCache accumulates
over. Deliberately short: sizing a barrier off travel
measured over a horizon that itself scales with the barrier
is circular, and it ran away to 14-31*ATR on EURUSD/USDCAD
in 2026-08-07. That guard is correct and stays.
BARRIER horizon 64 bars - what the LABEL walk and the first-passage ladder
run over, and how long the EA actually holds the trade.
So `up >= target` is a 12-bar question and `label == Buy` is a 64-bar one, and
the second can freely exceed the first. TripleBarrierLabel gates the excursion
accumulation on `idx - t <= excWindow` while the barrier walk and the ladder run
the full horizon - the split is explicit and intentional.
THE BUG IS MINE. bc57aca's scale ladder tested reachability with `up[i] >= tp`,
i.e. it asked the 12-bar question about a 64-bar trade. That understates
reachability by ~2x, which is why EVERY wide rung was rejected and the geometry
fell back to the tightest rung at 1.61/3.21. The data supported considerably
wider; the test was just asking the wrong question.
FIX: LadderWinShare() reads the answer off the first-passage ladder - target
touched strictly before the stop, over the full horizon, tie to the stop. That
is the identical question the label walk asks, so the ladder share and the Buy
rate should now agree to within rung discretisation. Both legs snap to the
SMALLEST rung at or above the requested multiple (harder target, harder stop) so
the floor stays conservative.
Expect the scale ladder to select a WIDER rung on the next relabel. On this
data the excursion test read 17.7% at q50 where the true full-horizon share is
35.9%, so rungs that scored 8.1% and 2.8% were likely well above the floor.
ALSO:
- Window reconciliation now PRINTED every derivation: excursion travel share,
ladder win share, and the label cache's Buy share side by side, with the
ladder-vs-label gap flagged if it exceeds rung discretisation. Those two must
agree; if they ever stop agreeing, one of them is wrong and the line says so.
- Renamed tpReach/slReach -> tpTravel/slTravel and relabelled the log line. They
describe the EXCURSION window and are near-tautological there (a q50 stop is
exceeded by ~50% of bars); calling them "reached within the horizon" is what
made the two quantities look like one.
- BARRIER_MIN_TP_REACH_PCT is now BARRIER_MIN_REACH_FRACTION_OF_BE (0.60) x
break-even instead of a hardcoded 20.0. Break-even for 1:RR is 100/(1+RR), so
the absolute floor silently tightened as RR rose - 0.60x at RR=2 but 0.80x at
RR=3, penalising the user for asking for a bigger target. Evaluates to exactly
20.0% at the shipped RR=2, so this is a no-op today and correct if the knob
moves.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:51:08 -04:00
//--- Reachability floor for the scale ladder, expressed as a FRACTION OF BREAK-EVEN rather than as an
//--- absolute percentage. It has to scale with BARRIER_TARGET_RR or the knob silently changes how strict
//--- the floor is: break-even for a 1:RR trade is 100/(1+RR), so the old absolute 20% was 0.60x
//--- break-even at RR=2 but would be 0.80x at RR=3 - tightening the test simply because the user asked
//--- for a bigger target. 0.60 reproduces exactly 20.0% at the shipped RR=2, so this is a no-op today
//--- and correct tomorrow.
//--- What it means: a long at this geometry must WIN on at least 60% as often as break-even requires.
//--- Below that the horizon is truncating the trade rather than the market refusing to pay - and the
//--- positive class gets too rare to train on, which is the failure the 2026-08-09 min-RR raise caused.
# define BARRIER_MIN_REACH_FRACTION_OF_BE 0.60
# define BARRIER_MIN_TP_REACH_PCT \
( BARRIER_MIN_REACH_FRACTION_OF_BE * 100.0 / ( 1.0 + BARRIER_TARGET_RR ) )
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective
The last run could not have demonstrated an edge either way, and nothing in the
log said so. Four changes so it does.
1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets
every era; m_oosSamples only resets on a full model reset. So 'always-long %'
decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and
0.0% at era 2219. This is the SAME bug already found and fixed for
logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left
in the one line whose whole job is to be the reference every other number is
read against. Correct at era 1, wrong everywhere after - including the '62%
zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is
finally readable.
2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate
'short by a hair' from 'short by an amount no strategy could cover'. The era
line now prints the required win rate, the SE, the effective n and the
lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars
and L=75.6 there are ~63 independent observations, putting the bar near 66% at
typical coverage.
3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages
at every ladder level, so each candidate geometry's resolution time is
readable without training on it - L-vs-width becomes a measurement across the
whole ladder in ONE run rather than a second chart. Each rung reports L,
n_eff, min provable edge and min provable EV.
4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability
are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio,
so min provable EV ~ width^2 while the cost saving from width is only linear.
Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that
clears reachability) is right once an edge is known; MEASURE (narrowest that
keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it
still has to be shown. The direction does not depend on the exponent, and
item 3 makes the exponent checkable.
Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a
rejected rung reports the previous rung's lifespan as its own; per-rung
detectability is labelled IS-sample based (the deriver may not see the holdout),
so absolute figures are optimistic while the ranking is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
//--- WHICH END OF THE SCALE LADDER WINS, and it depends on what phase the project is in. This is not a
//--- taste knob; the two objectives are genuinely opposed and each is correct in its own phase.
//---
//--- WIDE is right once an edge is KNOWN. EV per trade = edge x width and the spread is a fixed cost, so
//--- width is pure cost efficiency: at 10.48*ATR the round trip costs 0.9% of the move, at 4.82 it costs
//--- 2.0%.
//---
//--- NARROW is right while the edge still has to be DEMONSTRATED, and the scaling is brutal. Labels
//--- overlap by their lifespan L, so an N-bar window holds N/L independent observations (see
//--- EffectiveSampleSize). First-passage time for the band [-m,+k] grows like m*k, and at a fixed ratio
//--- that is width^2 - so:
//--- SE ~ sqrt(L/N) ~ width / sqrt(N)
//--- min detectable edge = 2*SE ~ width
//--- min detectable EV = edge x w ~ width^2
//--- DOUBLING THE WIDTH QUADRUPLES THE SMALLEST EV YOU CAN PROVE, while buying only a linear improvement
//--- in cost. Measured 2026-08-17 on SP500 H4: a 4,738-bar OOS window at L=75.6 holds ~63 independent
//--- observations, which puts the deploy gate's required win rate above anything reachable. The run was
//--- not failing to find an edge; it could not have measured one either way.
//---
//--- The exponent above is theory - the measured L came in 4.2x shorter than the m*k formula predicts -
//--- but the DIRECTION does not depend on the exponent: L rises with width under any sane model, so
//--- narrower always buys independent samples. The ladder now prints measured L and the implied
//--- detectable EV per rung (see LadderWinShare), so the exponent is checkable rather than assumed.
# define BARRIER_SCALE_MEASURE 0 / / narrowest rung that stays cost - efficient - maximises detectability
# define BARRIER_SCALE_DEPLOY 1 / / widest rung that clears reachability - maximises EV per trade
# define BARRIER_SCALE_OBJECTIVE BARRIER_SCALE_MEASURE
//--- The floor that stops MEASURE mode running to the tightest rung, and the mirror of the reachability
//--- floor that stops DEPLOY mode running to the widest. Round-trip cost is 2*spread, so this caps how
//--- much of the move the spread is allowed to eat. At the measured 0.047*ATR spread, 3% admits any
//--- width down to ~3.1*ATR - which is where the ladder should stop, not where a constant happens to.
# define BARRIER_MAX_COST_FRACTION_PCT 3.0
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
# define MI_TARGET_BARRIER 0 / / shipped 3 - class triple - barrier label
# define MI_TARGET_EXC_UP 1 / / ( maxHigh - entry ) / ATR over the horizon , 3 equal - frequency bins
# define MI_TARGET_EXC_DOWN 2 / / ( entry - minLow ) / ATR
# define MI_TARGET_EXC_RANGE 3 / / up + down : pure realised volatility , the control that SHOULD clear
fix: normalise the asymmetry target - the raw one is confounded by volatility
Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three;
raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500
(p=0.1045). That looked like the first directional signal this project has
found. It probably is not, and the test as built could not tell.
(up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its
null on every instrument - and the directional part is symmetric noise eps, then
up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A
pure volatility predictor scores positive MI against a 3-bin (up-dn) while
carrying no directional information at all. Crucially that confound REPLICATES,
so reproducing on two instruments is not evidence against it - and the effect
sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries
~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a
leaked fraction of the volatility signal, not an independent one.
So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only
target a directional claim may rest on. The verdict now separates the cases and
NAMES the confound when raw clears while normalised does not, instead of
reporting the raw line as a finding.
Two bugs of mine in the same block, both caught by output rather than review:
- The derived-geometry line had a MISORDERED argument list: it printed
"stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the
multiple and the multiple as the quantile. Real values were 2.61 stop /
8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen.
- THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25
"so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the
stop - hit three times in four. The printed reachability said exactly that
("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate.
This is the entire reason reachability is measured and printed rather than
assumed.
Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop
still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And
bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the
end of the sorted array.
The geometry from the previous run is NOT usable and the asymmetry result is
unresolved, not established. Both are decided by the next run.
FORCES A FULL RETRAIN (the stop quantile changes every label).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
# define MI_TARGET_EXC_ASYM 4 / / up - down : RAW asymmetry - CONFOUNDED BY VOLATILITY , see below
//--- SCALE-FREE asymmetry, and the only one of the two that can support a directional claim.
//--- (up-dn) is NOT scale-free: if sigma is predictable - and RANGE clears at ~4x its null on every
//--- instrument tested - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a
//--- large sigma pushes the value into BOTH outer terciles. A pure volatility predictor therefore scores
//--- positive MI against a 3-bin (up-dn) while carrying no directional information whatsoever, and it
//--- does so consistently across instruments - so replication does not rule it out. Measured 2026-08-07:
//--- raw ASYM cleared on EURUSD and USDCAD at p=0.0050 exactly where RANGE was strongest.
//--- Dividing by (up+dn) removes the scale factor and leaves the question actually being asked: given
//--- that price moved, WHICH WAY did it move further. Bounded in [-1,+1] by construction.
# define MI_TARGET_EXC_ASYM_NORM 5
fix: make the indicator tuner actually measure, and gate what it installs
ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates
returned exactly 0.00359 nats): the tune loop re-inits the indicators and then
scores, with no RefreshData() between.
ReInitADIndicators() does its part - Create() builds a NEW handle carrying the
new parameters, and the feature cache is flagged stale so features really are
recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and
only Refresh() copies data out of a handle into those. So every candidate was
scored on values still held from the PREVIOUS handle. My earlier guess in the
diagnostic ("suspect the feature cache") was wrong: the cache invalidation works.
Two things land together, because neither is safe alone:
1. RefreshData() after the re-init, so a candidate is scored on its own features.
2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and
the maximum of N draws from a null beats its incumbent almost every time - so
"it beat the incumbent" installs noise. This selector is the highest-stakes of
the three found in this audit because it ACTS: it overwrites the user's
configured indicator settings and forces BuildFreshTopology(), so the network
then trains on whatever the noise picked. Fixing (1) without (2) would have
made a dormant bug actively harmful.
The gate draws the winner's own permutation null once, then corrects the p-value
for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather
than the max-of-N resample used by the geometry scan because each candidate here
has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only
the one null. Exact under independence, mildly anti-conservative under positive
dependence - stated in the comment rather than hidden. A rejected winner restores
the configured settings, which best[] cannot do since the descent mutates it.
Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate()
calculates asynchronously, so if the spread is STILL zero the handles simply are
not done and the tuner needs to yield between candidates rather than score them
back to back - a state machine like the label prebuild. That distinction is now
readable from the log instead of requiring another guess.
No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 09:31:06 -04:00
//--- Significance the indicator tuner's winner must reach, AFTER correcting for having been chosen out of
//--- N candidates. Same 0.05 as elsewhere; what matters is that a gate exists at all, since this selector
//--- overwrites the user's configured indicator settings and forces a fresh topology.
# define MI_TUNE_ALPHA 0.05
2026-08-16 14:44:11 -04:00
//--- WHAT THE TUNER OPTIMISES FOR (re-pointed 2026-08-16). The sweep used to score candidates against
//--- the BARRIER label - direction - which every measurement here puts at the noise floor (best-of-999
//--- at p=1.0000, lag profile empty, normalised asymmetry collapsing on three instruments). Optimising
//--- indicator parameters for MI against a noise target means the winner gate correctly rejects nearly
//--- everything ("no improvement" run after run) - the sweep was climbing a flat landscape. RANGE is
//--- the one target with measured signal (4x its null, p=0.005, WITH a working positive control), and
//--- it is what the excursion head - the component these features actually serve under the
//--- meta-labeling architecture - is trained to predict. So the tuner now selects indicator settings
//--- for how much they know about REALISED VOLATILITY, the channel that prices barriers and sizes
//--- risk. The MI report still scores the barrier label alongside (ReportFeatureLabelInformation is
//--- unchanged) so a directional miracle would still be seen - it just no longer steers the tuner.
# define MI_TUNE_TARGET MI_TARGET_EXC_RANGE
fix: correct the lag profile across lags too - it contradicted itself
3271f1e tested each of ~21 lags against its OWN null at alpha 0.05 and starred
whatever cleared. That is about one false positive per run before any signal
exists, and because neighbouring lags share nearly their entire feature window
the false positives arrive in CLUSTERS that read like a hump.
It did exactly that on SP500 H1, twice in one afternoon on identical data:
13:55 nothing clears at any lag headline MI p=0.4478
16:22 k6/k10/k12/k16 starred, headline MI p=0.8756, observed
"information survives to lag 16" BELOW its own null mean
Same 31 features, same 2009 samples, same 287 blocks, cross-asset absent in
both - so this was not two different measurements. Non-replication on identical
data is the signature of an uncorrected multiple comparison, and acting on the
second run would have pinned the lookback to 17 off noise.
Galling detail: 04ee2e1 had just added exactly this correction to the
barrier-geometry scan one function below. The rigorous bar went on the report
with 6 candidates and the naive one stayed on the report with 21.
So the lag profile now uses the same construction as the geometry winner test:
one draw from every lag, keep the largest, repeat; a lag clears only by beating
that distribution. Draws centred leave-one-out to match how the observed excess
is centred. Independence across lags overstates the spread of the maximum
(neighbours share their window), so it errs toward rejecting.
Also: the positive branch now says to re-run before acting, because one run of
this report has demonstrably not been a result; and MI_LAG_MAX_PROFILE caps the
retained-draw matrix rather than trusting a derived m_historyBars.
Read-only diagnostic. No input, topology or label change: no retrain, and a
training run already in flight stays valid.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:28:57 -04:00
//--- Ceiling on profiled lags, sizing the retained-draw matrix. m_historyBars is derived and could in
//--- principle exceed this; the profile then covers the first MI_LAG_MAX_PROFILE-1 lags and says so via
//--- the lag count it prints, rather than overrunning the buffer.
# define MI_LAG_MAX_PROFILE 32
fix: gate the barrier-geometry winner on a family-wise null, not its own
The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain".
That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two
numbers, with no test that either is distinguishable from zero.
bestExcess is a MAXIMUM over the eligible candidates. The maximum of several
draws from a null sits well above any single draw from it, so a max-shaped
statistic tested against a single-candidate null crowns a winner on noise
almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the
lag profile committed in 3271f1e measures the pure-noise swing on this exact
data at +/-0.0004, peaking at +0.00042 with nothing clearing its own null at
any lag. The advisory was one ratio away from talking us into relabelling and
retraining all four topologies to chase that.
So build the null OF THE MAXIMUM: retain every candidate's permutation draws,
take one draw from each candidate, keep the largest, repeat. The winner must
beat that distribution.
- draws centred LEAVE-ONE-OUT, so a draw is centred by a mean excluding it -
exactly how the observed score is centred. Centring a draw by a mean that
contains it shrinks it toward zero and would deflate the null.
- only ELIGIBLE candidates enrol: the family the max was taken over is the
family to correct for, and a clamped or sub-minRR pairing can never win.
rrOK hoisted above the draws for this.
- draws per candidate 20 -> MI_GEOMETRY_PERMUTATIONS (40): they now have to
resolve an upper tail, which is where 20 draws are thinnest.
- MI_GEOMETRY_ALPHA 0.05, stricter than the lag profile's: a wrong lookback
costs input width, a wrong geometry costs a full retrain from era 0.
Independence across candidates overstates the spread of the max (the real
candidates share features and overlapping label windows), so the gate errs
toward rejecting - the safe direction when passing costs a retrain.
Read-only diagnostic. No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 14:04:16 -04:00
//--- Draws per candidate in the barrier-geometry scan. Raised from the ranking-only 20 because these draws
//--- now do a second job: they build the FAMILY-WISE null that decides whether the winner is real (see
//--- ReportBarrierGeometryScan). A max-statistic lives in the upper tail of the null, and a tail is exactly
//--- where 20 draws are thinnest. Cost is draws x candidates on an already-extracted sample - the whole
//--- scan ran in 3.2s at 20 draws, so this is single-digit seconds, once, before era 0.
# define MI_GEOMETRY_PERMUTATIONS 40
//--- Family-wise significance for the geometry winner. Stricter than MI_LAG_ALPHA because the two decisions
//--- are not comparable: a lag profile that guesses wrong costs some input width, whereas acting on this
//--- one means RELABELLING and retraining every topology from era 0. The gate must be hard to pass.
# define MI_GEOMETRY_ALPHA 0.05
//--- Ceiling on scanned candidates, sized to the shipped grid (2 stop multiples x 6 target multiples). Only
//--- used to size the fixed draw matrix below; the loop still skips ineligible pairings.
# define MI_GEOMETRY_MAX_CANDIDATES 12
fix(autotune): MI scorer read an array nobody filled; add the permutation floor
THE TUNER WAS A SILENT NO-OP. Every chart logged
auto-tune complete - 17 candidate settings scored in ~139s,
feature/label mutual information 0.0000 -> 0.0000 nats (no improvement)
0.0000 is not a weak result, it is a broken measurement: finite-sample MI
is biased UPWARD, so even pure noise scores above zero. Cause:
ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the
bar's features to TempData and never touches m_featureCache - only the
caching wrapper BufferTempData() writes that array. It then read
m_featureCache, which ReInitADIndicators had just invalidated. Every
column came back constant, FeatureColumnMI returned 0 for all of them,
and all 17 candidates tied at exactly zero. 139 s per chart to return the
settings it started with.
Now reads the values back out of TempData, where they actually land. And
an exactly-zero best score is called out as a fault rather than reported
as "no improvement", because that is what it is.
ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has
been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats -
at these sample sizes the same order as any real edge in this domain - so
a raw MI figure is uninterpretable on its own. Shuffling the labels
destroys every genuine association while leaving sample size, binning and
class proportions intact, so the score it produces IS this dataset's
noise floor, measured rather than approximated. The log now reads
feature/label information - X nats against a shuffled-label floor of Y
and says outright whether the features carry usable information about the
target. It needs no training, no topology and no convergence, so unlike
every accuracy number in this codebase it cannot be confounded by an
optimizer or an objective. If the score sits on the floor, no change of
architecture can help - which is the question the last three days of
zero-edge results have been circling.
DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by
any amount. At ~11,000 directional calls the standard error of the
precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma
noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires
EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the
actual call count, so the bar scales with the evidence instead of needing
a hand-picked constant.
Recorded with it, because it is why chance is the right reference at all:
under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k),
and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k).
The label's own base rate IS the break-even rate, at every SL/TP setting.
So "beats chance" and "is profitable" are the same test, and no choice of
SL/TP can manufacture an edge - only prediction can.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
//--- How many standard errors a checkpoint's directional precision must clear chance by before it is
//--- considered deployable - see the edgeFloorPct block in Train(). Two sigma is the conventional
//--- "not a fluke" bar and lands near 1pp at the ~11,000 directional calls a full OOS era produces.
//--- CRITICAL CONTEXT, and the reason chance is the right reference at all: under a driftless random
//--- walk the probability of touching +k*ATR before -m*ATR is m/(m+k), and the break-even win rate for a
//--- k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every
//--- SL/TP setting. So "beats chance" and "is profitable" are the same test, no SL/TP choice can
//--- manufacture an edge, and this margin is measuring expectancy directly.
# define EDGE_MIN_SIGMAS 2.0
fix(geometry): do not adopt a pairing that cannot be certified - information and detectability are different objectives
Closes a gap 62a719f opened. Making ReportBarrierGeometryScan authoritative put two
objectives in charge of one decision without reconciling them:
- the scan maximises entry-time INFORMATION, in nats;
- the deploy gate needs enough INDEPENDENT observations to certify an edge.
They pull opposite ways. A wider pairing carries more information per call AND takes
longer to resolve, and overlapping labels are worth ~1/L each - so tripling the
horizon divides the independent sample by ~3 and multiplies the standard error the
gate has to beat by ~sqrt(3). USDJPY's current winner is exactly that trade: 2:8 at
h192 against an incumbent 1.61:3.21 at h64. Until today the adoption was inert so it
never mattered; from 62a719f it decides the geometry, and it would have made that
swap silently on the next fresh run.
The guard inverts the deploy identity for the WINNER's own pairing - certifying an
edge d needs z^2 p(1-p)/d^2 independent calls, at that pairing's break-even - and
compares it against what the OOS window can physically supply at that horizon. If
even a generous 10pp edge is out of reach, the pairing is not adopted and the log
says it lost on detectability rather than on information.
Conservative by construction: the horizon is an UPPER bound on the mean label
lifespan, so oosBars/horizon is a LOWER bound on available independent observations.
The guard therefore only fires when the pairing is hopeless, never merely hard.
This is the same quantity ReportDetectability publishes per run, applied at the one
moment it can still change a decision instead of after the geometry is already
pinned. More information per trade is worth nothing if it buys too few independent
trades to prove it - WIDTH is not free, and on this window it is the binding
constraint, not the nats.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:47:37 -04:00
//--- The edge, as a fraction, that the barrier-geometry adoption must at least be ABLE to certify before
//--- it will switch geometries (see the detectability guard in ReportBarrierGeometryScan). Deliberately
//--- GENEROUS - 10pp over break-even is far more than anything this project has ever measured - because
//--- the guard is meant to catch pairings that are hopeless rather than merely hard. A geometry that
//--- cannot support even a 10pp edge on this window is one no amount of training could ever cash.
# define ADOPT_MIN_DETECTABLE_EDGE 0.10
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- BOTH-DIRECTIONS FLOOR. A model that has stopped calling one side entirely is degenerate, and the
//--- coverage+precision test above cannot see it: coverage counts directional calls without caring
//--- that they are all the same direction, and one-sided precision can sit above chance while the
//--- model is simply riding the sample's drift. Observed 2026-08-09: the perceptron reported
//--- "Sell:0%" in every one of its 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance,
//--- deployed, and sprayed buy arrows across the chart.
//---
//--- Deliberately far below m_minDirectionalRecallPct (the 40%-per-class diagnostic): that floor is
//--- effectively unreachable on this data and was demoted for exactly that reason, so reusing it here
//--- would block every deployment rather than the pathological ones. 10% only catches a side that has
//--- genuinely gone silent, and it applies to Buy and Sell only - Neutral is the majority class and a
//--- low Neutral recall is a model taking positions, not a broken one.
# define DEPLOY_MIN_SIDE_RECALL_PCT 10.0
feat: gate deployment on the null of the MAXIMUM, not the per-era null
EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM
over every era a run ranks. A 2-sigma one-sided test passes on noise with
probability 0.0228 per era, so over N eras the chance at least one clears
it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The
gate was near-certain to open on a long run whatever the data held.
It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance -
+1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the
call counts these runs produce that is p_family 0.92..0.9999.
Every OTHER best-of-N decision here already carries this correction, and
every one REJECTS on this data: the barrier-geometry winner (null of the
maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI
lag profile (null of the maximum over 21 lags). The one decision that
ships a model to a live account had none.
BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to
deploy:
z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n)
p_single = P(Z >= z)
p_family = 1 - (1-p_single)^N
against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN
snapshotted precision/chance/call-count, not the latest era's, because
the model that ships is the one that has to clear the bar.
N counts CANDIDATE eras (coverage measurable, at least one directional
call) - an era that called nothing directional could never have become
the best, so counting it would make the gate stricter than the search
that actually happened.
Conservative on purpose: consecutive eras share OOS bars and differ by
one gradient step, so they are nowhere near N independent draws and the
true family-wise error is below this bound. This gate decides what trades
real money and the house posture is reject-unless-demonstrated.
Effect at 2900 directional calls / N=112: required edge goes 1.76pp ->
2.92pp. A real edge clears it; +1.5pp does not.
Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and
the m_trainingComplete assignment - which must stay identical or the flag
persisted into the .nnw disagrees with the decision to stop, and a reload
runs inference on a model the ladder refused.
NOT applied to the two operator paths (era-cap deploy, panel Deploy
button). Those stay the operator's call; ReportSelectionGateVerdict()
logs the verdict beside them so an authorised deploy can never later be
misread as a validated one.
NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather
than pulling in Math\Stat. Verified against reference values to 6dp:
Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are
ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1".
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- FAMILY-WISE DEPLOYMENT GATE. EDGE_MIN_SIGMAS above is a PER-ERA test, and the deployed model is the
//--- MAXIMUM over every era a run produced - which is the one construction this project has repeatedly
//--- proven crowns noise. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so
//--- over N eras the chance that at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70,
//--- 93% by era 112. The gate was therefore near-certain to open on a long run no matter what the data
//--- contained, and it did - HYBRID deployed on 2026-08-08 at dir-precision 35.5% vs 34% chance, +1.5pp,
//--- selected as the best of 112 eras whose per-era values wandered between 30% and 35.5%.
//--- Every OTHER best-of-N decision in this codebase already carries this correction and every one of
//--- them REJECTS on this data: the barrier-geometry winner ("tested against the null of the maximum over
//--- 6", p=0.3902), the indicator tuner (Sidak "p_family = 1 - (1-p)^N", p=1.0000), the MI lag profile
//--- ("against the null of the MAXIMUM over 21 lags"). The one decision that actually ships a model to a
//--- live account had none. See BestCheckpointSurvivesSelection().
//--- Sidak on the era count, matching the tuner's idiom. NOTE it is CONSERVATIVE here: consecutive eras
//--- share the same OOS bars and differ by one gradient step, so they are nowhere near N independent
//--- draws and the true family-wise error is below this bound. Erring strict is deliberate - this gate
//--- decides what trades real money, and the codebase's whole posture is reject-unless-demonstrated.
# define DEPLOY_FAMILY_WISE_ALPHA 0.05
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- OVERSAMPLING CONSTANTS REMOVED 2026-07-31 (MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION).
//--- Data-level class-balance oversampling is gone; every bar is queued exactly once and the imbalance
//--- is corrected analytically in the gradient by the logit-adjusted loss. See the queueing block in
//--- Expert\AIBase\Training.mqh for the four successive oversampling designs that collapsed before it,
//--- and the class-imbalance audit in Variables\Inputs.mqh for the nine inputs this consolidated.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- TRIPLE-BARRIER LABELS (Lopez de Prado, "Advances in Financial Machine Learning", ch. 3).
//--- 2026-08-01: replaced exact-pivot ZigZag labels. ZigZag REMAINS the input-feature source
//--- (m_useSwingContext) and the horizon source below - only the TARGET changed.
//---
//--- Why. The old label marked the single exact bar where a ZigZag pivot confirmed: ~1,164 Buy, ~1,164
2026-08-01 11:27:28 -04:00
//--- Sell, ~35,841 Neutral - a 31:1 imbalance, and every correction mechanism in this file (logit
//--- adjustment, prior EMA, bias seed, recall floor, alternation gate, NMS, four oversampling designs)
//--- was downstream of that one choice. The imbalance was self-inflicted by the target, not a property
//--- of the market: the reference book (references\neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag
//--- but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50, nothing to correct.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//---
//--- What replaces it. For each bar, place the EA's OWN stop and target around a hypothetical entry at
//--- that bar's close and ask which barrier a real trade would touch first, within a horizon:
//--- long reaches TP before SL, short does not -> Buy
//--- short reaches TP before SL, long does not -> Sell
//--- neither resolves in the model's favour -> Neutral
//--- The barriers come from SL_Mode/TP_Mode (m_sl_mode/m_tp_mode, inherited from CExpertSignalCustom),
//--- so the label and the trade CANNOT drift apart, and `dir-precision` in the era line stops being a
//--- proxy and literally becomes the win rate of the strategy under its own exit rules. That is the
//--- number this project never had. No new user input: the two that shape the label already exist.
//---
//--- Expected balance. At the shipped SL_ATR_x1 / TP_ATR_x3 the gambler's-ruin probability of touching
//--- +3 ATR before -1 ATR is 1/(1+3) = 25% per side, so roughly 25/25/50 - about 2:1 rather than 31:1.
//--- Measured for real at the end of the prebuild; do not assume it.
//---
//--- INTRABAR AMBIGUITY IS RESOLVED PESSIMISTICALLY. When one bar's range spans both barriers, OHLC
//--- cannot say which came first, so the label counts it as the STOP. A win rate built on the
//--- optimistic reading is exactly the kind of number that evaporates live.
# define BARRIER_TIE_GOES_TO_STOP 1
//--- Vertical (time) barrier, in bars. DERIVED, never configured: the median distance between confirmed
//--- ZigZag pivots over the training window - i.e. this symbol/timeframe's own natural swing horizon,
//--- measured from the same indicator the features already read. Snapped to the ladder below so the
//--- estimate has to move ~30% to change the answer; without that quantization a horizon that drifted as
//--- history downloaded would silently relabel a partially-trained model's targets mid-run. Same
//--- measure-once-then-quantize contract as ComputeFirstLayerWidth().
//--- It is deliberately NOT in the weights-filename fingerprint - a filename keyed on a measured
//--- quantity orphans a trained model the moment the measurement moves. See BuildConfigFingerprint.
fix(labels): the 128-bar horizon ceiling was truncating the shipped label
The corrected geometry scan exposed something bigger than the geometry
question it was asked. Every pairing from 2:6 upward came back CLAMPED -
including 2:6, the SHIPPED configuration.
First-passage time for a driftless walk leaving [-m,+k] goes as m*k, and
the measured swing median here is ~12 bars at m*k=1, so 2:6 wants ~144
bars and 3:10 wants ~360. The ladder stopped at 128. A clamped label
stops meaning "does the target come before the stop" and quietly becomes
"...within 128 bars", while the deployed EA holds until SL or TP with no
bar limit. So the target the models have been trained on all along was
not the strategy the EA executes, and the trades it silently reclassified
as Neutral were the SLOW WINNERS - precisely the ones a 1:3 barrier
exists to capture. Timeout share stayed ~0% throughout, which is why this
never showed up: the truncation lands in Neutral, not in the timeout
counter that was watching for it.
Ladder extended to 384 (12..128, 192, 256, 384) so every selectable
geometry gets an honest horizon. Cost is one embargo of at most 384 bars
out of ~38k.
Second fix, same class of error as the H(Y) one: the scan's "best
eligible" was 2:2, a 1:1 barrier, against a shipped Min_Risk_Reward_Ratio
of 1:2. Training four topologies on that target would have produced a
model whose every setup is rejected at the door - the exact failure
behind four consecutive Market rejections for "no trading operations".
Sub-minRR geometries are now ineligible and marked [<minRR], printed
rather than hidden.
Also drops the dense-depth tag from the display name ("Perceptron 3L" ->
"Perceptron"). Depth is derived, so it names nothing a user chose; the
config tag [PAI-0be2] already disambiguates concurrent charts and does it
for every input rather than one. Full topology still logged by "config -".
Compiles 0 errors / 0 warnings, standard and Market. Build tag
horizon-384-v1. Changes the LABEL for every geometry, so the next scan
supersedes the previous numbers - and a retrain is required before any
model trained under the truncated target means anything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:54:45 -04:00
//--- 2026-08-01: the ladder used to stop at 128 and that ceiling, not the barrier geometry, was the
//--- binding constraint. First-passage time for a driftless walk leaving [-m, +k] is proportional to m*k,
//--- and the measured swing median here is ~12 bars at m*k=1 - so EVERYTHING from 2:6 upward wanted more
//--- than 128 bars and got truncated, including the SHIPPED 2:6 configuration. A truncated label stops
//--- meaning "does the target come before the stop" and quietly becomes "...within 128 bars", while the
//--- deployed EA holds until SL or TP with no bar limit. That is a train/deploy mismatch in the target
//--- itself, and it silently converted the slow winners - exactly the trades a 1:3 barrier exists to
//--- catch - into Neutral. Extended to cover the whole selectable grid: 3:10 needs ~360 bars.
//--- Cost is one embargo of BARRIER_HORIZON_MAX bars out of ~38k, i.e. nothing.
# define BARRIER_HORIZON_LADDER_COUNT 11
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
# define BARRIER_HORIZON_MIN 12
fix(labels): the 128-bar horizon ceiling was truncating the shipped label
The corrected geometry scan exposed something bigger than the geometry
question it was asked. Every pairing from 2:6 upward came back CLAMPED -
including 2:6, the SHIPPED configuration.
First-passage time for a driftless walk leaving [-m,+k] goes as m*k, and
the measured swing median here is ~12 bars at m*k=1, so 2:6 wants ~144
bars and 3:10 wants ~360. The ladder stopped at 128. A clamped label
stops meaning "does the target come before the stop" and quietly becomes
"...within 128 bars", while the deployed EA holds until SL or TP with no
bar limit. So the target the models have been trained on all along was
not the strategy the EA executes, and the trades it silently reclassified
as Neutral were the SLOW WINNERS - precisely the ones a 1:3 barrier
exists to capture. Timeout share stayed ~0% throughout, which is why this
never showed up: the truncation lands in Neutral, not in the timeout
counter that was watching for it.
Ladder extended to 384 (12..128, 192, 256, 384) so every selectable
geometry gets an honest horizon. Cost is one embargo of at most 384 bars
out of ~38k.
Second fix, same class of error as the H(Y) one: the scan's "best
eligible" was 2:2, a 1:1 barrier, against a shipped Min_Risk_Reward_Ratio
of 1:2. Training four topologies on that target would have produced a
model whose every setup is rejected at the door - the exact failure
behind four consecutive Market rejections for "no trading operations".
Sub-minRR geometries are now ineligible and marked [<minRR], printed
rather than hidden.
Also drops the dense-depth tag from the display name ("Perceptron 3L" ->
"Perceptron"). Depth is derived, so it names nothing a user chose; the
config tag [PAI-0be2] already disambiguates concurrent charts and does it
for every input rather than one. Full topology still logged by "config -".
Compiles 0 errors / 0 warnings, standard and Market. Build tag
horizon-384-v1. Changes the LABEL for every geometry, so the next scan
supersedes the previous numbers - and a retrain is required before any
model trained under the truncated target means anything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:54:45 -04:00
# define BARRIER_HORIZON_MAX 384
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Fallback when the ZigZag scan finds too few pivots to take a median from (a cold history buffer, or
//--- a symbol so quiet that Depth-12 emits almost nothing). Mid-ladder, and it logs when it fires.
# define BARRIER_HORIZON_FALLBACK 32
//--- Minimum confirmed pivots required before the median is trusted rather than the fallback.
# define BARRIER_HORIZON_MIN_SAMPLES 20
//--- Share of bars one class must hold before the era-0 output-bias cold-start seed fires - see the
//--- trigger in AdvanceLabelCachePrebuild(). A +-3.0 bias seed is a correction at a 94%-Neutral prior
//--- and a distortion at a 50% one, so the threshold marks where "dominant" actually starts.
# define COLD_START_SEED_MIN_DOMINANCE 0.70
2026-07-15 21:47:09 -04:00
//--- Reduce-on-regression learning-rate decay. `eta` (AI\Network.mqh) is a plain global read fresh
//--- by every weight-update call on every backend (native/OpenCL/CPU-DLL/DirectML-DLL alike), so
//--- shrinking it here takes effect on the very next backProp() everywhere at once. A fixed step
//--- size that is large enough to escape a bad random init quickly (see the sharp era 5->14 OOS
//--- accuracy climb this was tuned against) is, by the same token, large enough to overshoot once
//--- training gets close to a good solution - the era 14->16 regression right after that climb is
//--- the classic signature of that overshoot, not a structural bug. The existing best-checkpoint
2026-07-29 00:38:05 -04:00
//--- mechanism just below (m_bestOosForecast + CNet::CaptureWeights) already guarantees the FINAL deployed
2026-07-15 21:47:09 -04:00
//--- model can't regress; this constant lets the live training process itself settle down instead
//--- of continuing to oscillate around a good solution once it finds one.
# define ETA_DECAY_REGRESSION_PCT 5.0 / / only decay after a real regression , not per - era noise
# define ETA_DECAY_FACTOR 0.7
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric
Four of the six findings from research/training_pipeline_audit_2026-08-09.md
(F4 mini-batching and F6 feature re-encode deliberately deferred - see the
report's implementation-status section for why):
- F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%,
which is 15-bit - provably non-uniform on every full-history era over 32,768
queued samples. New 30-bit ShuffleRandomIndex().
- F2: plateau warm restarts were a no-op whenever eta already sat at its
ceiling (the normal state of a non-regressing plateau) - the ladder was just
a 24-era countdown. Restarts now overshoot to 5x the ceiling
(PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience
window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has
real range.
- F3: checkpoint restores put weights back but kept the rejected trajectory's
Adam moments, so the optimizer immediately pushed back toward the rolled-back
state (the restore->regress->restore oscillation). CNet::ResetOptimizerState()
zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta
untouched) on every mid-run restore, every boosted restart, and the
deploy-time restore that online learning continues from.
- F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk,
so the selection metric the checkpoint ranking and deploy gate read is a pure
function of the checkpoint instead of partly measuring BN drift. Defensive
unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and
the OOS continual-learning simulation stay adaptive by design.
Compiled clean (0 errors, 0 warnings) via the staged-tree recipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- 1e-5, not the old 1e-4: with the ceiling at 3e-4 the old floor gave the whole decay schedule a
//--- 3x dynamic range - three 0.7x decays and it was pinned, so "reduce LR on regression" could never
//--- actually settle a run that kept oscillating (2026-08-09 audit, F2). 30x leaves the schedule room
//--- to genuinely calm down; the recovery bump still climbs back at 1/0.7 per new best, so a run that
//--- resumes improving is not stuck crawling.
# define ETA_MIN 0.00001
fix(training): escape the recall-gate catch-22 that let runs decay unchecked
Evidence (MQL5\Logs, SP500 H1, 2026-07-29):
Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51%
LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44)
Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%)
CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122)
Every model peaks early then decays monotonically toward Neutral, and nothing
stops it: the restore-best-weights + decay-eta handler is gated on
m_bestPassedRecall, which stays false forever when no checkpoint ever clears the
per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The
plateau ladder cannot end such a run either (stage 3 refuses to deploy without a
recall pass, so it resets ~27 times), making it a 1000-era one-way trip.
The gate's own justification had expired. It was written when the pre-pass
tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral
most confidently". The balanced-selection change replaced that with
`balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a
Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot
anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so
far", which is worth defending; and isWorseEra is itself a balanced-accuracy
regression, so it cannot fire merely for trading Neutral calls for Buy/Sell.
The original concern still holds while the best-so-far IS near-collapse, so the
escape is margin-guarded: defend the checkpoint only once balanced accuracy sits
more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of
100/3. Against the run above that engages for all three stuck topologies
(42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still
explores freely.
Two inputs restored to the regime that actually produced a deploy:
- MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th
00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown
reachable here - a floor above what the config can reach is the same "target
set too high" failure the surrounding comment already warns about.
- OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant
(Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6%
true base rate - under-calling, with no headroom to converge down from. The
deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into
the floor from above. Raw over-calling is the intended starting condition; live
calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's
own note says to judge over-calling by live-fired precision, not raw counts.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
//--- Balanced accuracy (macro-recall) of a model that puts every bar in ONE class: (0+0+100)/3. It is
//--- the FLOOR of the balanced metric, not a midpoint - any genuinely multi-class model scores above it.
refactor(ai): derive the first dense layer's width instead of asking for it
InitialNeurons was an input whose only defensible value depends on two
things the user cannot see when picking from a dropdown: how wide the input
vector ended up after feature selection, and how much in-sample data the
study period actually yields. Left to a hand-picked constant it was badly
wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a
292,583-weight model, against ~36,500 training bars of which only ~2,236
are directional. That is 6.6 weights per training bar, and it EXPANDS a set
of highly correlated inputs rather than compressing them.
The symptom was already in the logs and had been read as a depth problem:
the shallowest topology consistently beat the deepest (perceptron 52.7%
balanced, hybrid 41.3%). Over-parameterization predicts that ordering just
as well as covariate shift does, and only one of the two had been addressed.
ComputeFirstLayerWidth() budgets roughly one first-layer weight per
in-sample bar. Measured across the configurations in use:
M15 10y -> 256 units, 129,071 weights, 0.73 per bar
H1 10y -> 64 units, 28,727 weights, 0.65 per bar
H4 10y -> 16 units, 7,559 weights, 0.68 per bar
Two design points that matter:
- It estimates in-sample bars from the STUDY PERIOD and timeframe, not
from Bars(). What is downloaded grows over a terminal's lifetime, and a
topology that widened as history filled in would re-key its own weights
file and discard a trained model.
- The result is snapped down to a coarse power-of-two ladder, so the
estimate would have to be wrong by ~2x to change the answer.
Every field it reads is already part of the weights-filename fingerprint,
so the derived value needs no fingerprint entry of its own. The public
setter is removed - it could only have been called after construction, and
would either be ignored or silently re-key the model mid-run.
Where the data cannot support even the floor (D1 over 10 years is under
2,000 bars) it now says so and names the fixes, rather than quietly
training a model with more weights than examples.
The DB config fingerprint drops the term too, which re-keys existing
pattern databases once - correct, since a model an order of magnitude
smaller should not inherit the old one's win-rate history.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
//--- ComputeFirstLayerWidth() constants. SECONDS_PER_YEAR is the mean Julian year (365.25 days), the
//--- same convention MQL5's own date arithmetic uses. MARKET_OPEN_FRACTION allows for closed hours and
//--- weekends - ~0.72 is right for both a 24/5 FX week and an index with extended sessions, and the
//--- ladder in that function makes anything in the 0.6-0.85 range land on the same rung anyway.
//--- FIRST_LAYER_MIN_WIDTH is a floor for degenerate configs (a very short study period, or a symbol
//--- whose history cannot support the requested window) - below this the taper has nothing to work with.
# define SECONDS_PER_YEAR 31557600.0
# define MARKET_OPEN_FRACTION 0.72
fix(ai): cap logit-adjustment strength to the head's usable logit range
tau=1.0 inverted the collapse instead of curing it. The head is SIGMOID, so
each output is bounded to [0,1] and the widest logit gap the net can express
between two classes is CLASS_LOGIT_SCALE * (1-0) = 6. The offsets are
tau*log(prior_c), whose spread on this 30:1 imbalance is 3.42 - so tau=1.0
spent 57% of the ENTIRE expressible range on the prior correction.
The network did the only thing available to it: saturate Buy/Sell outputs to
1.0 to overcome a -3.42 training handicap. The offsets are absent at
inference, so that surplus made every bar directional. Measured across all
five still-training charts: Neutral recall 0%, directional calls on ~100% of
bars, win rate 5-7% against a ~6% base rate - no information whatsoever -
while balanced accuracy read a flattering 58-64% because two of its three
terms sat near 95%. OOS accuracy 6%.
Menon et al. assume an unbounded logit head where a 3.42 shift is negligible
against the reachable range. It is not negligible here, so the strength is
now expressed RELATIVE to the range actually available:
tau_eff = min(tau_cfg, LOGIT_ADJUST_MAX_RANGE_FRACTION * SCALE / spread)
At 20% that gives tau 0.35 on this data. Deliberately a fraction rather than
a tau ceiling: it stays correct if CLASS_LOGIT_SCALE changes, if the head
becomes unbounded, or on any symbol whose imbalance differs. The input
remains effective below the cap, so dialling it down needs no rebuild.
Simulated at a signal strength where the task is genuinely learnable, the
precision/recall frontier is monotone: tau 1.0 -> 49.6% call rate at 6.4%
precision (base rate 6.1%, i.e. worthless); tau 0.35 -> 2.0% at 15.5%;
tau 0.15 -> 0.2% at 33.3%. The capped value lands in the same regime the
pre-logit-adjustment run occupied (1-6% of bars at 20-35% win rate).
Also logs the measured priors, the spread, and whether the cap bound.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 23:20:07 -04:00
//--- Ceiling on how much of the classification head's LOGIT RANGE the logit-adjustment offsets may
//--- consume, as a fraction. The head is SIGMOID, so each output is bounded to [0,1] and the widest
//--- logit difference the net can express between two classes is CLASS_LOGIT_SCALE * (1 - 0) - six,
//--- at the shipped scale. The offsets are tau*log(prior_c), whose spread on a 30:1 imbalance is
//--- log(0.939) - log(0.031) = 3.42, so an untamed tau=1.0 spends 57% of the ENTIRE expressible range
//--- on the prior correction alone. Measured 2026-07-29: every chart did the only thing it could -
//--- saturate its Buy/Sell outputs to 1.0 to overcome a -3.42 handicap during training - and since the
//--- offsets are absent at inference, that surplus made EVERY bar directional. Neutral recall 0%, calls
//--- on ~100% of bars, win rate 5-7% against a ~6% base rate: no information at all, while balanced
//--- accuracy read a flattering 64% because two of its three terms were ~95%.
//--- Menon et al. assume an unbounded logit head, where a 3.42 shift is negligible against the range
//--- the network can reach. It is not negligible here, so the strength has to be expressed relative to
//--- the range actually available. Deliberately a FRACTION rather than a tau ceiling: it stays correct
//--- if CLASS_LOGIT_SCALE changes, if the head becomes unbounded, or on any other symbol/timeframe
//--- whose class imbalance differs - none of which a hardcoded tau would survive.
# define LOGIT_ADJUST_MAX_RANGE_FRACTION 0.20
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Minimum directional call rate a checkpoint must reach to be considered deployable, expressed as a
//--- FRACTION OF THE TRUE DIRECTIONAL BASE RATE rather than an absolute percentage - a model that calls
//--- a direction a quarter as often as one actually occurs is sparse but usable; one that calls ten
//--- times a decade is not, however precise those ten calls were. Derived rather than configured so it
//--- adapts to any symbol/timeframe/label rule without a second input to keep in sync.
# define MIN_COVERAGE_FRACTION_OF_BASE_RATE 0.25
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- DIRECTIONAL CONFIDENCE THRESHOLD (2026-08-09). The training loss and the checkpoint-selection
//--- metric want different things, and until now only the second one knew it. Logit-adjusted
//--- cross-entropy pushes the head to call every class at roughly its adjusted prior - it has no term
//--- for "how often should I trade" - so the model calls a direction on 87-91% of bars. The selection
//--- metric meanwhile is precision x coverage credit, saturating at the coverage floor: above the
//--- floor, extra calls earn NOTHING and only precision counts. So selection wants few, good calls and
//--- the loss produces many mediocre ones, and all selection could do was pick the least-bad era out of
//--- whatever the loss happened to hand it. Nothing pushed the model toward selectivity.
//---
//--- This closes that gap without touching the loss (which is doing its own job correctly - it is
//--- estimating class probabilities, and a probability estimate should not be distorted to encode a
//--- trading policy). The decision RULE gets the policy instead: emit a directional call only when the
//--- softmax margin between the winning direction and its best rival clears a threshold, and pick that
//--- threshold to maximise precision subject to still clearing the same coverage floor the selection
//--- metric uses. Standard cost-sensitive-decision practice (Elkan 2001): estimate probabilities, then
//--- choose the operating point separately.
//---
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- FITTED ON A HELD-OUT CALIBRATION SLICE, APPLIED TO OOS AND LIVE. It is not enough for the fit to
//--- avoid the graded data (it always did - pass 3 comes after): the curve it reads must also be one the
//--- weights have not MEMORIZED, and harvesting it from pass 2's own backprop samples failed that second
//--- requirement badly. Measured 2026-08-10 by pairing every fit against the same era's OOS result:
//---
//--- PAI era 1 IS 25% coverage @ 66.1% win (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
//--- PAI era 24 IS 66% coverage @ 76.4% win (+9.5pp) -> OOS 65% (-2pp) gap +11.4pp
//--- PAI era 76 IS 90% coverage @ 79.6% win (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
//--- LSTM era 9 IS 77% coverage @ 81.6% win (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
//---
//--- The gap grows monotonically with era while OOS stays flat, i.e. the IS curve stops describing the
//--- model's behaviour on unseen bars within a handful of eras. That is fatal HERE specifically, because
//--- FitDirConfThreshold's objective branches on the SIGN of (p - break-even): the memorized curve says
//--- +12pp at 95% coverage, so `coverage x (p - p0)` correctly maximises coverage and returns ~0.02 -
//--- fire on every bar. The `p < p0 -> get more selective` branch, which is the actual regime and the
//--- entire point of 983a6a3, could never fire because IS never showed p < p0.
//---
//--- So the slice below is carved out of the IS span, purged from backprop on BOTH sides by one label
//--- horizon, and scored after pass 2 has finished training. It costs DIR_CONF_CALIB_PCT_OF_IS of the
//--- training data. That is worth paying for a reason beyond honesty: the deploy gate needs
//--- dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero DILUTES any edge that is
//--- concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward
//--- chance by construction. A threshold that can actually be selective is the only mechanism by which a
//--- small, concentrated edge could ever clear that gate.
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//---
//--- Bin count is a resolution/noise trade-off: the margin lives in [0,1], so 50 bins put each candidate
//--- threshold 0.02 apart, fine enough to sit near the precision peak and coarse enough that each bin
//--- still holds thousands of the ~38k IS bars.
# define DIR_CONF_THRESHOLD_BINS 50
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Below this many directional calibration calls the histogram is too sparse to choose an operating
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- point from, and a threshold fitted on a handful of bars is just the best-of-N error at a smaller
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- scale. The model then KEEPS THE PREVIOUS ERA'S THRESHOLD rather than falling back to 0.0: "trade
//--- every bar" is the most dangerous setting in the range, so it must never be what a failed
//--- measurement decays to. At era 0 the previous value is 0.0 anyway, so the first-era behaviour is
//--- unchanged.
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
# define DIR_CONF_MIN_FIT_CALLS 200
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Share of the IS span held out to fit the operating point on. 15% of a ~38k-bar IS span is ~5.7k
//--- bars, ~28x DIR_CONF_MIN_FIT_CALLS even before allowing that ~99% of bars resolve directionally
//--- under the measured geometry - so the fit is never sparse, and each of the 50 bins still holds
//--- enough calls to be a rate rather than a coin flip. Larger buys precision in the fit at a direct
//--- cost in training data; smaller makes the operating point itself noisy, which is the failure this
//--- whole mechanism exists to avoid.
# define DIR_CONF_CALIB_PCT_OF_IS 15
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
//--- (A RANK_SLICE_PCT_OF_OOS constant lived here for a few hours on 2026-08-16, reserving a share of
//--- the OOS window for the pattern-database backfill. It is gone: the CALIBRATION BAND already has
//--- every property it was carving for - see StartPatternDatabaseBackfill. Carving a second reserved
//--- region cost the deploy gate 20% of its sample for nothing, and in its first placement it also
//--- blanked ~10 months of chart arrows, because arrows are only drawn on bars pass 3 grades.)
feat: excursion-size head (Stage 1, measurement only)
Direction is closed - normalised asymmetry fails on three instruments
with a working positive control, and the classifier's own best-of-999
era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is
a different question and RANGE clears at ~4x its null.
Checked the denomination before building on that, since the source memo
warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is
predictable" is a claim about travel RELATIVE to current ATR, not a
restatement of "ATR is autocorrelated". It is exactly the part a fixed
multiple (stop 3.31*ATR, target 1.64*ATR) discards.
A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches
ladder rung k) upward and downward. Survival parameterisation rather than
regressing the multiple, because it needs nothing new from CNet: sigmoid
outputs and the per-neuron delta the `total != 3` branch already applies
(a quantile head would need a linear activation and a pinball gradient in
Network.mqh, Network.cl and the DirectML path, on a class four topologies
share). Targets are free - m_ladderUpAt already records first-touch age
per rung with 0 meaning never reached.
Separate net, not extra outputs on the classifier: more outputs would
change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push
the count off 3 - the exact condition backProp uses to select the joint
softmax gradient the 3-class head depends on. The classifier is
bit-for-bit unaffected and this is removable without trace.
STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the
constant per-rung base rate - the baseline a fixed ATR multiple already
assumes - with both predictors fitted IS and evaluated OOS, so neither
gets a look at the test set. Positive skill justifies Stage 2 (drive
SL/TP and sizing off ExcursionQuantile, which is defined and deliberately
uncalled). Zero or negative means ATR already carries everything and
Stage 2 must not be built.
Trains only on primary occurrences: the replay queue oversamples for
CLASS balance, and a direction-balanced sample is a biased SIZE sample.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION-SIZE HEAD (see Expert\AIBase\Excursion.mqh for the whole rationale). Hidden width is
//--- deliberately small: the question it asks has a known low-dimensional answer (volatility
//--- clustering), and era time here is taken from a classifier already at ~300 s/era.
# define EXCURSION_HIDDEN_UNITS 24
//--- Below this many held-out bars the Brier skill score is noise and no verdict is printed.
# define EXCURSION_MIN_SCORED 500
//--- Skill (percent) the head must beat before Stage 2 is justified. Not zero: replacing a constant
//--- that cannot fail with a learned quantity that can needs to buy more than a rounding error, and a
//--- Brier skill under a couple of percent is inside the era-to-era wobble of the estimate itself.
# define EXCURSION_SKILL_USEFUL_PCT 2.0
fix: the excursion gate would have passed Stage 2 on an artifact I made
Second-opinion review killed the +4.2% far-rung result, correctly, and
the mechanism is my own bug. A head trained toward {0.05,0.9} converges
to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1,
POSITIVE where p < 1/3, growing monotonically as the rung gets farther.
Against a baseline frozen at the IS rate, an upward-biased head scores
positive Brier skill whenever the OOS rate merely sits above the IS rate.
Predicted signature: huge negatives near, ~zero at p=1/3, growing
positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs
were not the clean end of a distorted measurement, they were the other
face of the same artifact. Everything before 25aca83 is void.
The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4
topologies x N eras, reported per era - a best-of-~300 with no interval
and no multiplicity control, which is the shape of the four traps already
documented here. It now needs FOUR things at once:
DECISION RUNGS only the rungs ExcursionQuantile actually reads at the
live geometry (target 1.62, stop 3.31 ATR), fixed
before looking. Skill at 5 ATR is skill about a
distance no order is placed at - and the TARGET side
currently interpolates 1.5/2.0, which measured -2.2%
and -1.3%.
DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64
horizon bars, so ~16k scored bars is ~250 independent
ones and every SE over the full set is ~8x understated.
VS ORACLE the best constant achievable ON THE SCORED BLOCK,
closed form from H and n (Brier = H*(1-H/n)). A head
that learned only a LEVEL nearer the OOS rate than the
frozen IS constant scores positive against the old
baseline and <= 0 here. This is the control that
separates per-bar skill from base-rate drift.
MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing
constrained 8 independent sigmoids to obey that, and
ExcursionQuantile returns the FIRST crossing - so a
tangled curve is misread exactly where the head is
least sure. Counted and reported, not silently used.
The pass message now also states what a pass would and would not buy:
expectancy is -costs at zero directional edge whatever the stop distance,
and under prop DD limits LOWER variance also lowers P(reach target before
limit), so "better drawdown" is a choice of failure mode, not a win.
Still owed before any Stage 2: a race against a trailing-quantile
incumbent and a vol-feature logistic. Beating a frozen global constant is
the weakest admissible bar for replacing a global constant.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
//--- Minimum DISJOINT (non-overlapping-horizon) observations before that tally is allowed to decide
//--- anything. ~16k scored bars over a 64-bar horizon leaves ~250 independent ones, which is the real
//--- sample size; below this the disjoint skill is noise quoted to one decimal place.
# define EXCURSION_MIN_DISJOINT 200
//--- Share of bars whose predicted survival curve is non-monotone. Above this the 8 sigmoids are not
//--- describing one distribution and ExcursionQuantile's first-crossing read is not well defined.
# define EXCURSION_MAX_MONO_VIOL_PCT 5.0
feat: race the excursion head against a trailing-quantile incumbent
Beating a frozen global constant is the weakest admissible bar for
replacing a global constant. The honest incumbent is a rolling rung
frequency: it adapts to the volatility regime - exactly what the head
claims to predict - and needs no model, no 760 inputs and no training.
Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one
ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon`
entries are held back UNRESOLVED: a bar's rung outcomes are only known
one horizon later, so using them would be lookahead and would flatter the
incumbent into an opponent the head could never fairly beat. Pass 3 walks
oldest-to-newest, so "pushed more than horizon bars ago" is exactly
"resolved by now". Each push is O(rungs), not O(window).
The head's decision-rung Brier is pro-rated to the trailing estimate's
coverage before the ratio, since the incumbent only scores bars where its
window is warm.
This line is worth reading on its own, independently of the head: if the
trailing quantile beats the global constant, that is a cheap risk-control
win available with no machine learning at all - and it is the same number
either way, so the run answers both questions in one pass.
The ring is deliberately NOT reset per era - it estimates the market, not
the era, and re-warming 500 bars every era would leave the incumbent
unusable over the first chunk of every scoring pass, handing the head a
free win on exactly those bars.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
//--- TRAILING-CLIMATOLOGY window, in bars, over RESOLVED outcomes only. This is the real incumbent for
//--- "replace a global ATR multiple": a rolling rung frequency adapts to the volatility regime, which is
//--- exactly the thing the head claims to predict, and it needs no model at all. ~2000 H1 bars is about
//--- three months - long enough that the rate is not noise, short enough to track a regime. The head's
//--- margin over THIS, not over a frozen constant, is what would justify 760 inputs.
# define EXCURSION_TRAIL_WINDOW 2000
//--- Resolved bars the trailing window must hold before its estimate is allowed to score anything.
# define EXCURSION_TRAIL_MIN_N 500
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x
Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other"
30->337s). My estimate had been "single-digit percent". The cost is
per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is
19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k
forward/backward calls x several layer submits each, and its 760-wide
layer exceeds the CPU DLL's inline threshold so each one pays a real
handoff. The classifier's own net time tripled too, from contention with
a second pool on an already-full box.
Three changes, all backend-neutral because they remove submits rather
than tune threads:
SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar
of their horizon, so 16k consecutive bars were always ~250 independent
observations - the full-sample tally was never worth more than the
disjoint one, it just quoted an n that was ~64x too large. Dropping it
costs nothing statistically and removes 63 of every 64 forward passes.
The two parallel tallies collapse into one, which is also less code.
The trailing ring still advances on every bar: it needs the outcome
SEQUENCE, and that is array lookups, not a forward pass.
TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and
strongly autocorrelated - neighbouring bars carry near-identical
excursion information - so per-bar training buys resolution the target
does not have. Strided on ATTEMPTS, not acceptances, so a stretch of
unlabelled bars cannot silently change the spacing.
OWN TIMING COLUMN. The head's passes were landing in the era line's
"other" bucket, which is how a 3.6x regression read as an unexplained
jump in the one column nobody attributes. A cost that cannot be seen in
the timing line cannot be traded off against anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
//--- Train the head on one primary bar in this many. Excursion targets are strongly autocorrelated -
//--- neighbouring bars share almost all of their horizon - so consecutive samples are near-duplicates,
//--- and the cost of this net is per-dispatch rather than per-FLOP on every backend.
# define EXCURSION_TRAIN_STRIDE 4
refactor(ai): derive the first dense layer's width instead of asking for it
InitialNeurons was an input whose only defensible value depends on two
things the user cannot see when picking from a dropdown: how wide the input
vector ended up after feature selection, and how much in-sample data the
study period actually yields. Left to a hand-picked constant it was badly
wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a
292,583-weight model, against ~36,500 training bars of which only ~2,236
are directional. That is 6.6 weights per training bar, and it EXPANDS a set
of highly correlated inputs rather than compressing them.
The symptom was already in the logs and had been read as a depth problem:
the shallowest topology consistently beat the deepest (perceptron 52.7%
balanced, hybrid 41.3%). Over-parameterization predicts that ordering just
as well as covariate shift does, and only one of the two had been addressed.
ComputeFirstLayerWidth() budgets roughly one first-layer weight per
in-sample bar. Measured across the configurations in use:
M15 10y -> 256 units, 129,071 weights, 0.73 per bar
H1 10y -> 64 units, 28,727 weights, 0.65 per bar
H4 10y -> 16 units, 7,559 weights, 0.68 per bar
Two design points that matter:
- It estimates in-sample bars from the STUDY PERIOD and timeframe, not
from Bars(). What is downloaded grows over a terminal's lifetime, and a
topology that widened as history filled in would re-key its own weights
file and discard a trained model.
- The result is snapped down to a coarse power-of-two ladder, so the
estimate would have to be wrong by ~2x to change the answer.
Every field it reads is already part of the weights-filename fingerprint,
so the derived value needs no fingerprint entry of its own. The public
setter is removed - it could only have been called after construction, and
would either be ignored or silently re-key the model mid-run.
Where the data cannot support even the floor (D1 over 10 years is under
2,000 bars) it now says so and names the fixes, rather than quietly
training a model with more weights than examples.
The DB config fingerprint drops the term too, which re-keys existing
pattern databases once - correct, since a model an order of magnitude
smaller should not inherit the old one's win-rate history.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
# define FIRST_LAYER_MIN_WIDTH 16
refactor(ai): derive the dense taper's shape, not just its first layer
Deriving the first layer's width left NeuronsReduction and MinNeuronsCount
behind as inputs calibrated for something that no longer exists. Against a
hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine
funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to
64 -> 20 -> 20: the reduction factor stops mattering after one step, and
"minimum neurons per layer" silently becomes the width of every layer but
the first. Two knobs whose labels no longer describe what they do.
The taper now runs geometrically from the derived first-layer width down to
a final hidden layer sized off the output count, spread evenly over however
many layers the chosen AIType implies:
MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights
MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450
CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763
and it stays a funnel at the floor, where the old rule could not:
D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3
Both inputs are removed. With the width derived there is no freedom left in
the taper, so keeping either would only let the user contradict the
derivation. The layer COUNT stays selectable, because it is bundled into
AIType alongside the conv/LSTM front-end - depth is an architecture choice,
not a data-derived quantity, and pairing them means the two cannot
contradict each other.
m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing
reads them to build a topology any more, but they hold positional slots in
the .cfg sidecar and the weights fingerprint, and changing either value
would re-key every model on disk for no behavioural reason.
The DB config fingerprint drops both terms.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 14:03:42 -04:00
//--- Where the dense taper ENDS. The last hidden layer wants to be small enough to force the network to
//--- commit to a compressed representation, but comfortably wider than the decision itself so it is not
//--- the bottleneck - a few units per class is the usual heuristic. The absolute floor covers the
//--- single-output regression head, where 4x1 would be absurdly narrow.
feat(ai): real conv receptive field + the reference's channel pool
CONV's convolution used window = step = one bar, which is a per-bar
projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never
mixed information across time, so "convolutional" described the layer type
and nothing about what it computed. Same finding that sank HYBRID's LSTM.
Pooling was removed on 2026-07-29 for being misconfigured against the conv
output's memory layout. That removal was right; leaving the conv at a
one-bar window was not. The two belong together: the NeuroNet_DNG reference
(references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels
byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with
pool(window=4, step=4), and the pool only earns its place because a conv
with a real receptive field sits above it.
The input is bar-major (BufferTempData appends m_neuronsCount contiguous
features per bar), so a flat window of k*m_neuronsCount spans exactly k
bars - the receptive field needed NO kernel change. The conv output is
position-major, so window == step == window_out is a clean
max-over-channels, which is what the reference does and what the existing
pool kernels already implement correctly.
New chain at H1 defaults (420 = 20 bars x 21):
conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152
pool w=8 s=8 -> 19
conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars)
We deliberately stop before the reference's SECOND pool: a channel pool
emits one scalar per position, so a trailing pool would hand the dense stack
18 values and force it to fan out 18 -> 64. That is a bottleneck below every
learnable layer - the same class of mistake the 2026-07-29 removal was about.
Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor
tracked sliding POSITIONS, but a conv's real width is units_count *
window_out. Any pool stacked on a conv would therefore have sized against a
width window_out times too small and silently built the wrong shape. Both
branches now read the built layer's actual Neurons(), which is what the
batch-norm branch already did for the same reason.
Also closes the architecture-pinning trap: a .nnw persists the window each
conv was built with, so an existing CONV/HYBRID model would have loaded
cleanly and gone on training under the OLD architecture. The conv weight
tensor is (window+1)*window_out, so this cannot be repaired in place -
EnforceTopologyContract now detects it, reports both shapes, and retrains.
Conv chain shape is derived in one place (ConvReceptiveFieldBars /
ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions /
ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup
config line, so what is built and what is logged cannot drift.
Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
//--- Conv receptive field, in BARS. The reference (references\MQL5\Experts\EDL\Trajectory.mqh) uses a
revert(ai): restore the 4eae763 front-ends - both my rewrites stopped signaling
CONV and LSTM were signaling at 4eae763. Two changes I made after it each
broke one of them, and neither was caught by the metric I was reading.
CONV, same data one hour apart on SP500 H1:
19:44 window = 1 bar era 5: dir-precision 18%, 45 live fires, best bal 3.1 -> 4.5
20:46 window = 2 bars era 5: no directional calls, 0 live fires, best bal frozen at era 1
LSTM: the sequence rewrite has been live since 18:48 (the `lstm 420->64`
config line only derives that way under the new per-timestep budget) and has
been OOS recall Neutral:100% with a flat IS error in every era since, past
era 100.
Both are reverted behind a switch rather than deleted, because both
DIAGNOSES stand: a conv with a one-bar window is a 1x1 conv that cannot mix
across time, and the old LSTM really did apply one gate step to the whole
flattened input. What does not stand is shipping either on the strength of
an offline correctness proof.
CONV_RECEPTIVE_FIELD_BARS 2 -> 1 (also drops pool + conv2 via
HasSecondConvStage, restoring the
exact 4eae763 front-end)
LSTM_SEQUENCE_MODE 0 (single-timestep layer; sizing follows,
since a recurrence budgets on the
per-step width and this one does not)
Kept, because they are correct independent of the above:
- per-layer dW/W era line (18f63c7) - the instrument that should have
caught both of these in one era instead of a retrain cycle each
- CNet conv/pool sizing-cursor fix (a pool stacked on a conv would have
sized against a width window_out times too small)
- .nnw architecture guard - forces the retrain this revert needs, since
models trained since 20:46 carry a 2-bar conv tensor
- LSTM_FORGET_BIAS_INIT and both offline checks (gradcheck 2.3e-10,
flowcheck) - dormant at LSTM_SEQUENCE_MODE 0, correct when re-enabled
The lesson is in the two #define comments: a gradient check proves the math,
not that the layer trains in situ. Re-land each behind the dW/W line.
Both builds compile 0 errors, 0 warnings. Forces a CONV/LSTM/HYBRID retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:04:23 -04:00
//--- window of 2 positions with step 1, stacked twice for an effective field of 3; a value of 2 here
//--- matches it at our bar granularity. Not an input: it is a structural property of the front-end, and
//--- a user who picks it is choosing against the pool and second-conv shapes derived around it.
//---
feat(ai): true multi-bar conv and true sequence LSTM
CONV and LSTM were each configured as a strictly lossier perceptron, which
is exactly what the panel showed: PAI 24% > CONV 18% > HYBRID 12% ~ LSTM
12%, monotone in how much reaches the dense stack (420 / 160 / 32 / 16).
CONV - receptive field 1 -> 3 bars, and the pool is gone.
Reading the reference kernels settled why 34d6aa4 killed CONV.
FeedForwardConv emits POSITION-MAJOR output (matrix_o[out + window_out*i]),
and FeedForwardProof is a flat contiguous max over `window` at stride
`step`. On that layout any window <= window_out maxes ACROSS FILTERS
within one position - it cannot pool over time at all. Our stage used
window = step = filterCount: one max over all 8 filters per position,
discarding 87.5% of the conv output and leaving only the argmax filter
with gradient. That is a property of the reference's layout, not a
porting bug, so there is no correct pool to swap in. Springenberg et al.
ICLR 2015 is the answer already cited in this file: no pooling, get the
hierarchy from strided convolution. The second conv went with it - its
window was counted in raw elements while its comment claimed positions,
so a "2-position" window actually spanned 2 filters of position 0.
Filter count now derives from the WINDOW (RF * features / 2) instead of
one bar, which at RF 3 was under-sizing the stage 3x.
Shape: 20 bars x 21 -> 18 positions x 16 filters = 288.
LSTM - sequence mode back on, forget bias 2.0 -> 1.0.
The forward path rules out the "no gradient" reading of the 2026-07-30
failure: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 and
unrolls that sample's own window, so nothing leaks between shuffled
samples. Flat IS error + Neutral:100% is equally the signature of an
output that does not vary with the input, and that is what bias 2.0
produces: c* = i*g/(1-sigmoid(b)) ~ 8.3*i*g, |c*| ~ 4.2, tanh pinned at
0.9995 with derivative 1e-3, so h_T is near-binary and set by the gate
biases rather than the bars. Choosing 2.0 off the reach sweep was a
method error - reach trades against saturation and the sweep never
measured saturation. 1.0 is the Gers/Jozefowicz/Keras default and leaves
tanh derivative ~0.1.
Both builds 0/0. DLL unchanged (CPU_LSTMSeqForward/Backward already
exported). Forces a retrain of CONV, LSTM and HYBRID - the .nnw pins
architecture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:42:33 -04:00
//--- RE-LANDED 2026-07-31 at 3, WITHOUT the channel pool that came with it the first time.
//--- Why the 34d6aa4 attempt failed, established by reading the reference kernels rather than guessing:
//--- CNeuronConvOCL emits POSITION-MAJOR output - `matrix_o[out + window_out * i]`, i.e.
//--- [pos0 f0..fN][pos1 f0..fN]... (References\MQL5\Experts\NeuroNet_DNG\NeuroNet.cl, FeedForwardConv).
//--- The reference pool (FeedForwardProof) is a flat contiguous max over `window` elements at stride
//--- `step`. On position-major data ANY window <= window_out therefore maxes ACROSS FILTERS INSIDE ONE
//--- POSITION - it cannot pool over time at all. The stage we built used window = step = filterCount,
//--- which is exactly one max over all 8 filters per position: 87.5% of the conv's output discarded, and
//--- only the argmax filter receiving gradient at each position. That is the measured regression (era 5:
//--- 18% dir-precision and 45 live fires at RF=1, versus "no directional calls" and 0 fires at RF=2).
//--- It is a property of the reference's own layout, not a porting mistake, so there is no "correct pool"
//--- to swap in here: pooling over time is not expressible on this layout without a transpose.
//--- The literature answer, already cited elsewhere in this file, is Springenberg et al. ICLR 2015
//--- ("Striving for Simplicity: The All Convolutional Net"): drop pooling, get the hierarchy from strided
//--- convolution instead. So this stage is now a single TRUE convolution - a CONV_RECEPTIVE_FIELD_BARS-bar
//--- window sliding one bar at a time - and nothing else. 3 bars is the smallest window that can express
//--- a turning point (before/at/after), which is what the ZigZag labels mark.
//--- Verify a change here against CNet::LayerLearningReport's per-era norm(d|W|%/|dW|%) line, not against
//--- accuracy: |dW| >> d|W| means the layer is rotating (learning), the two roughly equal with the norm
//--- falling means it is only being decayed away. Reading accuracy is what made the first regression take
//--- a full retrain cycle to spot.
# define CONV_RECEPTIVE_FIELD_BARS 3
revert(ai): restore the 4eae763 front-ends - both my rewrites stopped signaling
CONV and LSTM were signaling at 4eae763. Two changes I made after it each
broke one of them, and neither was caught by the metric I was reading.
CONV, same data one hour apart on SP500 H1:
19:44 window = 1 bar era 5: dir-precision 18%, 45 live fires, best bal 3.1 -> 4.5
20:46 window = 2 bars era 5: no directional calls, 0 live fires, best bal frozen at era 1
LSTM: the sequence rewrite has been live since 18:48 (the `lstm 420->64`
config line only derives that way under the new per-timestep budget) and has
been OOS recall Neutral:100% with a flat IS error in every era since, past
era 100.
Both are reverted behind a switch rather than deleted, because both
DIAGNOSES stand: a conv with a one-bar window is a 1x1 conv that cannot mix
across time, and the old LSTM really did apply one gate step to the whole
flattened input. What does not stand is shipping either on the strength of
an offline correctness proof.
CONV_RECEPTIVE_FIELD_BARS 2 -> 1 (also drops pool + conv2 via
HasSecondConvStage, restoring the
exact 4eae763 front-end)
LSTM_SEQUENCE_MODE 0 (single-timestep layer; sizing follows,
since a recurrence budgets on the
per-step width and this one does not)
Kept, because they are correct independent of the above:
- per-layer dW/W era line (18f63c7) - the instrument that should have
caught both of these in one era instead of a retrain cycle each
- CNet conv/pool sizing-cursor fix (a pool stacked on a conv would have
sized against a width window_out times too small)
- .nnw architecture guard - forces the retrain this revert needs, since
models trained since 20:46 carry a 2-bar conv tensor
- LSTM_FORGET_BIAS_INIT and both offline checks (gradcheck 2.3e-10,
flowcheck) - dormant at LSTM_SEQUENCE_MODE 0, correct when re-enabled
The lesson is in the two #define comments: a gradient check proves the math,
not that the layer trains in situ. Re-land each behind the dW/W line.
Both builds compile 0 errors, 0 warnings. Forces a CONV/LSTM/HYBRID retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:04:23 -04:00
//--- Master switch for the sequence-LSTM front-end (AI\Network.mqh CNeuronLSTMOCL).
//--- 1 = the layer is a real recurrence over bars: one shared gate block applied at every timestep,
//--- with backpropagation-through-time. Gradient-checked to 2.3e-10 (DirectML\lstm_seq_gradcheck.cpp)
//--- and signal/gradient-reach-checked (DirectML\lstm_seq_flowcheck.cpp).
//--- 0 = the pre-2026-07-30 layer: ONE gate step over the whole flattened input, no recurrence.
//---
feat(ai): true multi-bar conv and true sequence LSTM
CONV and LSTM were each configured as a strictly lossier perceptron, which
is exactly what the panel showed: PAI 24% > CONV 18% > HYBRID 12% ~ LSTM
12%, monotone in how much reaches the dense stack (420 / 160 / 32 / 16).
CONV - receptive field 1 -> 3 bars, and the pool is gone.
Reading the reference kernels settled why 34d6aa4 killed CONV.
FeedForwardConv emits POSITION-MAJOR output (matrix_o[out + window_out*i]),
and FeedForwardProof is a flat contiguous max over `window` at stride
`step`. On that layout any window <= window_out maxes ACROSS FILTERS
within one position - it cannot pool over time at all. Our stage used
window = step = filterCount: one max over all 8 filters per position,
discarding 87.5% of the conv output and leaving only the argmax filter
with gradient. That is a property of the reference's layout, not a
porting bug, so there is no correct pool to swap in. Springenberg et al.
ICLR 2015 is the answer already cited in this file: no pooling, get the
hierarchy from strided convolution. The second conv went with it - its
window was counted in raw elements while its comment claimed positions,
so a "2-position" window actually spanned 2 filters of position 0.
Filter count now derives from the WINDOW (RF * features / 2) instead of
one bar, which at RF 3 was under-sizing the stage 3x.
Shape: 20 bars x 21 -> 18 positions x 16 filters = 288.
LSTM - sequence mode back on, forget bias 2.0 -> 1.0.
The forward path rules out the "no gradient" reading of the 2026-07-30
failure: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 and
unrolls that sample's own window, so nothing leaks between shuffled
samples. Flat IS error + Neutral:100% is equally the signature of an
output that does not vary with the input, and that is what bias 2.0
produces: c* = i*g/(1-sigmoid(b)) ~ 8.3*i*g, |c*| ~ 4.2, tanh pinned at
0.9995 with derivative 1e-3, so h_T is near-binary and set by the gate
biases rather than the bars. Choosing 2.0 off the reach sweep was a
method error - reach trades against saturation and the sweep never
measured saturation. 1.0 is the Gers/Jozefowicz/Keras default and leaves
tanh derivative ~0.1.
Both builds 0/0. DLL unchanged (CPU_LSTMSeqForward/Backward already
exported). Forces a retrain of CONV, LSTM and HYBRID - the .nnw pins
architecture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:42:33 -04:00
//--- RE-LANDED 2026-07-31, together with a corrected LSTM_FORGET_BIAS_INIT - see that constant, which is
//--- the part that was actually wrong.
//--- The 2026-07-30 attempt reported "flat IS error + Neutral:100%", which reads as "no gradient" but is
//--- equally the signature of an output that does not VARY with the input. The forward path rules the
//--- first out: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 (t==0 takes the nullptr branch),
//--- unrolls T = m_historyBars steps over that sample's own window, and emits h_T. State does not leak
//--- between shuffled samples, so shuffling is not the problem either. What DOES depend on the constant
//--- is saturation: with forget bias b the cell tends to c ~ i*g/(1-sigmoid(b)) over T steps, so b = 2.0
//--- gives c ~ 8*i*g, tanh(c) pins to +/-1, and h_T = o*tanh(c) becomes near-binary and set by the gate
//--- biases rather than by the bars. That is exactly "input-independent output".
//--- The tell in the era line is "OOS raw out B:min..max": a collapsed span there is this failure, not a
//--- gradient failure. Check it and the norm(d|W|%/|dW|%) line together before concluding anything.
# define LSTM_SEQUENCE_MODE 1
refactor(ai): derive the dense taper's shape, not just its first layer
Deriving the first layer's width left NeuronsReduction and MinNeuronsCount
behind as inputs calibrated for something that no longer exists. Against a
hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine
funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to
64 -> 20 -> 20: the reduction factor stops mattering after one step, and
"minimum neurons per layer" silently becomes the width of every layer but
the first. Two knobs whose labels no longer describe what they do.
The taper now runs geometrically from the derived first-layer width down to
a final hidden layer sized off the output count, spread evenly over however
many layers the chosen AIType implies:
MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights
MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450
CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763
and it stays a funnel at the floor, where the old rule could not:
D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3
Both inputs are removed. With the width derived there is no freedom left in
the taper, so keeping either would only let the user contradict the
derivation. The layer COUNT stays selectable, because it is bundled into
AIType alongside the conv/LSTM front-end - depth is an architecture choice,
not a data-derived quantity, and pairing them means the two cannot
contradict each other.
m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing
reads them to build a topology any more, but they hold positional slots in
the .cfg sidecar and the weights fingerprint, and changing either value
would re-key every model on disk for no behavioural reason.
The DB config fingerprint drops both terms.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 14:03:42 -04:00
# define HIDDEN_TAPER_OUTPUT_MULTIPLE 4
feat: derived taper restored; DB ranking reads a reserved slice, shrunk
TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor.
The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS
first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This
codebase MEASURES that width, and on the live SP500 H4 config it is 16 units -
already floored, with the budget printing "11360 estimated in-sample bars
cannot support a 800-wide input ... roughly 1.1 weights per training bar -
expect overfitting". At 16 units a floor of 20 makes lastHidden >=
m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch
and the width taper - the only part derived from this symbol's data - became
dead code on all four ensemble members, with depth (2 -> 4) set entirely by
counting feature domains. ComputeLayerWidths had already rejected this exact
pair of constants in its own comment.
The causal floor's premise does not hold either: layers are not inference
steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is
Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko
1989 and Hornik 1991 give universal approximation from a single hidden layer.
Depth buys parameter efficiency for compositional functions, not reasoning
hops. ForceHiddenLayers remains for measuring depth directly.
RANKING SLICE - the backfill no longer reads the window it is judged on.
The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window,
so win rates measured back over it are selection-inflated, and the backfill
was writing exactly those into the table filter weights rank on: the
selection set consumed twice, beside a deploy gate that applies a Sidak
correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS
window, plus a label-horizon purge, is now reserved and graded by nothing -
not pass 3, not checkpoint selection, not the gate. The backfill reads only
that. The gate keeps ~80% of its measurement (power goes as the square root,
so ~10% of a sigma), and the slice is the newest data, which is the regime
about to be traded. RankSliceBars returns 0 when no honest slice fits and the
backfill then REFUSES and says so, rather than falling back to the scoring
window and looking like a success.
SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate
by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw
ratio at the minimum sample count carries a ~15pp standard error, so a tier
that went 8-2 was handed weight 80 and outranked a tier measured over
hundreds of calls at 55 - the ranking was being driven by which small tier got
lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour).
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:49:52 -04:00
//--- REVERTED to 8 (2026-08-16, same day it was raised to 20). The 20 came from the MQL5 "4 hidden
//--- layers" article, whose floor is load-bearing on ITS first-layer width of 1000: 1000 -> 300 -> 90
//--- -> 27 needs a floor to stop, and 20 is where it stops. This codebase MEASURES the first layer
//--- instead, and on the live SP500 H4 config that measurement is 16 units (32 with a front end) -
//--- already floored, with the capacity budget printing "11360 estimated in-sample bars cannot support
//--- a 800-wide input ... roughly 1.1 weights per training bar - expect overfitting". At 16 units a
//--- floor of 20 makes lastHidden >= m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its
//--- first branch and THE WIDTH TAPER NEVER RUNS - the one part of the derivation that is actually
//--- measured from this symbol's data became dead code on all four ensemble members. See also the
//--- pre-existing note in ComputeLayerWidths, which had already rejected this exact pair of constants.
# define HIDDEN_TAPER_MIN_WIDTH 8
2026-07-30 09:22:11 -04:00
//--- ComputeConvFilterCount()/ComputeLstmHiddenSize() bounds. Both stages used to be inputs with a
//--- hand-picked constant default (16 filters, 32 hidden units) chosen with no reference to how wide the
//--- input actually ended up or how much data there is to fit them - the same defect the first-layer
//--- width had before it was derived. CONV_COMPRESSION_DIVISOR is the ratio the conv stage should
//--- compress a bar's feature vector by: the layer is a per-bar projection (window = step = one bar's
//--- features, see AddConvStage), so filters > features EXPANDS a correlated input at the very bottom of
//--- the stack, which is over-parameterization in its purest form. Halving is the conventional bottleneck
//--- choice and holds at any feature count.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- ComputeHiddenLayerCount() bounds. HIDDEN_TAPER_TARGET_RATIO is the per-layer compression the taper
2026-08-16 21:08:41 -04:00
//--- aims for, and depth is however many such steps it takes to get from the derived first-layer width
//--- to the output-tied final width. Two layers is the floor because one dense layer plus the head is a
//--- linear model with a single non-linearity; five is the ceiling because beyond it the per-step
//--- compression is so mild that the extra depth adds vanishing-gradient risk without adding abstraction.
feat: derived taper restored; DB ranking reads a reserved slice, shrunk
TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor.
The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS
first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This
codebase MEASURES that width, and on the live SP500 H4 config it is 16 units -
already floored, with the budget printing "11360 estimated in-sample bars
cannot support a 800-wide input ... roughly 1.1 weights per training bar -
expect overfitting". At 16 units a floor of 20 makes lastHidden >=
m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch
and the width taper - the only part derived from this symbol's data - became
dead code on all four ensemble members, with depth (2 -> 4) set entirely by
counting feature domains. ComputeLayerWidths had already rejected this exact
pair of constants in its own comment.
The causal floor's premise does not hold either: layers are not inference
steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is
Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko
1989 and Hornik 1991 give universal approximation from a single hidden layer.
Depth buys parameter efficiency for compositional functions, not reasoning
hops. ForceHiddenLayers remains for measuring depth directly.
RANKING SLICE - the backfill no longer reads the window it is judged on.
The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window,
so win rates measured back over it are selection-inflated, and the backfill
was writing exactly those into the table filter weights rank on: the
selection set consumed twice, beside a deploy gate that applies a Sidak
correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS
window, plus a label-horizon purge, is now reserved and graded by nothing -
not pass 3, not checkpoint selection, not the gate. The backfill reads only
that. The gate keeps ~80% of its measurement (power goes as the square root,
so ~10% of a sigma), and the slice is the newest data, which is the regime
about to be traded. RankSliceBars returns 0 when no honest slice fits and the
backfill then REFUSES and says so, rather than falling back to the scoring
window and looking like a success.
SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate
by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw
ratio at the minimum sample count carries a ~15pp standard error, so a tier
that went 8-2 was handed weight 80 and outranked a tier measured over
hundreds of calls at 55 - the ranking was being driven by which small tier got
lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour).
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:49:52 -04:00
//--- REVERTED to 2.0 (2026-08-16, same day it was changed to 10/3 - the MQL5 article's Pareto 70% cut).
//--- Halving is the conventional funnel and, unlike 10/3, it still produces a taper at the widths this
//--- codebase actually derives. See HIDDEN_TAPER_MIN_WIDTH above for the measurement that decided it.
# define HIDDEN_TAPER_TARGET_RATIO 2.0
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
# define MIN_HIDDEN_LAYERS 2
# define MAX_HIDDEN_LAYERS 5
//--- EstimatedInSampleBars() fallback for a chart whose history has not finished downloading. Below the
//--- trusted-bar floor the measurement says more about the sync state than about the symbol.
# define TOPOLOGY_BUDGET_MIN_TRUSTED_BARS 500
# define TOPOLOGY_BUDGET_FALLBACK_YEARS 10
2026-07-30 09:22:11 -04:00
# define CONV_COMPRESSION_DIVISOR 2
# define CONV_FILTERS_MIN 4
# define CONV_FILTERS_MAX 32
# define LSTM_HIDDEN_MIN 8
# define LSTM_HIDDEN_MAX 128
fix(training): escape the recall-gate catch-22 that let runs decay unchecked
Evidence (MQL5\Logs, SP500 H1, 2026-07-29):
Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51%
LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44)
Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%)
CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122)
Every model peaks early then decays monotonically toward Neutral, and nothing
stops it: the restore-best-weights + decay-eta handler is gated on
m_bestPassedRecall, which stays false forever when no checkpoint ever clears the
per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The
plateau ladder cannot end such a run either (stage 3 refuses to deploy without a
recall pass, so it resets ~27 times), making it a 1000-era one-way trip.
The gate's own justification had expired. It was written when the pre-pass
tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral
most confidently". The balanced-selection change replaced that with
`balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a
Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot
anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so
far", which is worth defending; and isWorseEra is itself a balanced-accuracy
regression, so it cannot fire merely for trading Neutral calls for Buy/Sell.
The original concern still holds while the best-so-far IS near-collapse, so the
escape is margin-guarded: defend the checkpoint only once balanced accuracy sits
more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of
100/3. Against the run above that engages for all three stuck topologies
(42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still
explores freely.
Two inputs restored to the regime that actually produced a deploy:
- MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th
00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown
reachable here - a floor above what the config can reach is the same "target
set too high" failure the surrounding comment already warns about.
- OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant
(Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6%
true base rate - under-calling, with no headroom to converge down from. The
deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into
the floor from above. Raw over-calling is the intended starting condition; live
calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's
own note says to judge over-calling by live-fired precision, not raw counts.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
# define BALANCED_COLLAPSE_PCT ( 100.0 / 3.0 )
//--- How far above that floor a best-so-far checkpoint must sit before the regression handler is
//--- allowed to defend it during the pre-recall-pass phase. See the guard's comment in Train(): the
//--- point is to distinguish "the best we have is still basically a collapse, keep exploring freely"
//--- from "we found a real multi-class state and are now sliding off it", which is the case that ran
//--- unchecked for 228 eras on SP500 H1 (2026-07-29).
# define BALANCED_WORTH_DEFENDING_MARGIN_PCT 5.0
2026-07-25 15:55:56 -04:00
//--- PLATEAU LADDER ------------------------------------------------------------------------------
//--- The two branches above only fire on a NEW BEST (isBetterEra) or on a real REGRESSION
//--- (isWorseEra, a drop of more than ETA_DECAY_REGRESSION_PCT below the best). Between them sits a
//--- dead zone - "not better, not meaningfully worse" - where NOTHING happened: no checkpoint, no
//--- restore, no eta change. A run that settles into that band is stuck, and before this ladder
//--- existed it stayed stuck until the era cap (observed 2026-07-25: balanced accuracy pinned in a
//--- 64-69% band for 15 consecutive eras while eta sat frozen and the softmax outputs slowly
//--- compressed toward uniform, with ~970 eras still to burn before the cap would end it).
//---
//--- So: count eras since the last NEW BEST, and escalate when that count says the run has stopped
//--- improving on its own. Two ideas from the literature, in this order:
//--- 1. WARM RESTART (Loshchilov & Hutter, SGDR, ICLR 2017). On a plateau the correct move is a
//--- BIGGER step, not a smaller one - decaying eta into a plateau just entrenches whatever basin
//--- the model is sitting in. Note this is the opposite of the isWorseEra branch's decay, and
//--- deliberately so: decay answers overshoot, restart answers stagnation.
//--- 2. GAMMA ANNEALING (Mukhoti et al., "Calibrating Deep Neural Networks using Focal Loss",
//--- NeurIPS 2020, which schedules gamma DOWN over training rather than fixing it). Focal loss's
//--- (1-pt)^gamma modulator goes to ~0 on everything the model already classifies well, so late
//--- in a run the surviving gradient comes almost entirely from genuinely ambiguous bars - and
//--- near a pivot, ZigZag labels ARE ambiguous. Meanwhile WEIGHT_DECAY keeps pulling every
//--- weight toward zero on every step regardless (see AI\Network.mqh's note that a weight's
//--- sustainable magnitude is ~ its gradient SNR / WEIGHT_DECAY). Annealing gamma hands the
//--- easy-but-correct bars their gradient back, restoring the signal side of that ratio.
//--- Annealing is MONOTONE (gamma only ever decreases within a run), matching the scheduled-gamma
//--- literature; eta may still be bumped back up by the existing recovery bump.
//---
//--- Escalation is per-stage, every PLATEAU_PATIENCE_ERAS eras without a new best. ANY new best
//--- resets the counter and the stage to 0 (the ladder is a response to stagnation, so evidence the
//--- run is moving again retires it) - except the annealed gamma, which stays where it got to.
# define PLATEAU_PATIENCE_ERAS 8 / / eras with no new best balanced accuracy before escalating a stage
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric
Four of the six findings from research/training_pipeline_audit_2026-08-09.md
(F4 mini-batching and F6 feature re-encode deliberately deferred - see the
report's implementation-status section for why):
- F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%,
which is 15-bit - provably non-uniform on every full-history era over 32,768
queued samples. New 30-bit ShuffleRandomIndex().
- F2: plateau warm restarts were a no-op whenever eta already sat at its
ceiling (the normal state of a non-regressing plateau) - the ladder was just
a 24-era countdown. Restarts now overshoot to 5x the ceiling
(PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience
window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has
real range.
- F3: checkpoint restores put weights back but kept the rejected trajectory's
Adam moments, so the optimizer immediately pushed back toward the rolled-back
state (the restore->regress->restore oscillation). CNet::ResetOptimizerState()
zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta
untouched) on every mid-run restore, every boosted restart, and the
deploy-time restore that online learning continues from.
- F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk,
so the selection metric the checkpoint ranking and deploy gate read is a pure
function of the checkpoint instead of partly measuring BN drift. Defensive
unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and
the OOS continual-learning simulation stay adaptive by design.
Compiled clean (0 errors, 0 warnings) via the staged-tree recipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
# define PLATEAU_STAGE_RESTART 1 / / first boosted warm restart ( see PLATEAU_RESTART_BOOST )
# define PLATEAU_STAGE_ANNEAL 2 / / second boosted warm restart ( the gamma anneal it named is gone )
2026-07-25 15:55:56 -04:00
# define PLATEAU_STAGE_DEPLOY 3 / / exhausted : deploy the best checkpoint and finish the run
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric
Four of the six findings from research/training_pipeline_audit_2026-08-09.md
(F4 mini-batching and F6 feature re-encode deliberately deferred - see the
report's implementation-status section for why):
- F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%,
which is 15-bit - provably non-uniform on every full-history era over 32,768
queued samples. New 30-bit ShuffleRandomIndex().
- F2: plateau warm restarts were a no-op whenever eta already sat at its
ceiling (the normal state of a non-regressing plateau) - the ladder was just
a 24-era countdown. Restarts now overshoot to 5x the ceiling
(PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience
window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has
real range.
- F3: checkpoint restores put weights back but kept the rejected trajectory's
Adam moments, so the optimizer immediately pushed back toward the rolled-back
state (the restore->regress->restore oscillation). CNet::ResetOptimizerState()
zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta
untouched) on every mid-run restore, every boosted restart, and the
deploy-time restore that online learning continues from.
- F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk,
so the selection metric the checkpoint ranking and deploy gate read is a pure
function of the checkpoint instead of partly measuring BN drift. Defensive
unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and
the OOS continual-learning simulation stay adaptive by design.
Compiled clean (0 errors, 0 warnings) via the staged-tree recipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Restart amplitude. Restoring eta merely TO its ceiling was a NO-OP whenever the run plateaued
//--- without ever tripping the regression decay - eta was still AT the ceiling, so stages 1 and 2
//--- assigned the value eta already had and the "ladder" was just a 24-era countdown to deploy
//--- (2026-08-09 audit, F2). Escaping a basin needs a rate LARGER than the one that settled into it:
//--- SGDR restarts span 10-100x; this is deliberately tamer because MAX_WEIGHT_DELTA and the
//--- best-checkpoint restore already bound the blast radius, and 5x the 3e-4 ceiling lands on 1.5e-3 -
//--- inside the ordinary Adam range (Kingma & Ba's own default is 1e-3). The boost is BOUNDED: the
//--- era-end anneal in Train() walks eta geometrically back to the ceiling over PLATEAU_PATIENCE_ERAS
//--- eras (a one-cycle kick, not a new permanent rate), and each restart also resets the optimizer's
//--- moment state (CNet::ResetOptimizerState) so the kick explores rather than replaying the stale
//--- momentum of the plateau it is escaping.
# define PLATEAU_RESTART_BOOST 5.0
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it
Points 3 and 4 of the four-point plan.
1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute.
The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it
fires, every one of those eras has been evaluated out of sample, so all of them sit
in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training
longer therefore does not merely cost time - it RAISES the bar the eventual winner
has to clear.
The new stop reads the TRAINING error, which the gate never looks at. When the
optimiser has stopped improving on data it can see, more eras will not find a better
model; they will only enlarge the OOS family. Ending there shrinks the correction,
and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted
an out-of-sample number.
That distinction is the whole point and it is the one this project has got wrong four
times: stop on IS and the family really is smaller; stop on OOS and those eras were
searched and still count. Both stops now exist; only this one buys a lower bar.
Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training
error is noisy per era - mini-batch order alone moves it - and ending a run that is
still learning costs far more than a few wasted eras. Improvement is RELATIVE
(IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and
it only acts when a checkpoint exists, since otherwise it would end a run with
nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot
early-stop on its first era against a previous run's best.
2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER.
The family-wise permutation gate already establishes that the RANKING is not noise.
It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is
biased upward by construction, being the largest of K noisy draws. The adoption
message quotes that raw maximum and compares it against the incumbent, so the number
a reader plans on is the inflated one.
The penalty is now measured, not assumed: the same permutation draws that produce the
p-value also produce, per draw, the MAXIMUM excess across all candidates under pure
noise. The mean of those maxima is exactly what a best-of-K selection is expected to
report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE
penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88),
and it needs no normality assumption because the draws ARE the null distribution.
Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large
effect is nearly untouched and a marginal one collapses toward zero.
Reported, not gated. The adoption decision still turns on the permutation p-value,
which is the right test for "is the ranking real"; the shrunk number is there so the
magnitude quoted beside it is one worth planning on. Closes the first of the two
EdgeFinder ports identified on 2026-08-12.
NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement
steer the target" is already true where it matters most - ReportGeometryExpectancyScan
ADOPTS the winning barrier geometry under the family-wise gate rather than advising
it, and the MI excursion suite publishes a verdict per instrument per config. What is
still missing is steering the TRAINING TARGET itself (direction vs excursion) off
those verdicts, and that is a design change rather than a surgical one - direction is
a closed verdict while excursion SIZE keeps clearing, so the honest version of that
change is a target-selection policy, not a flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
//--- IN-SAMPLE EARLY STOP (see the block in Train()). Relative improvement required for the training
//--- error to count as progress, and how much more patient this stop is than the OOS ladder. Training
//--- error is noisy per era - mini-batch order alone moves it - and ending a run that is still learning
//--- costs far more than a few wasted eras, so it waits several times as long before acting.
# define IS_ERROR_IMPROVE_FRAC 0.01
# define IS_ERROR_PATIENCE_MULT 3
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- PLATEAU_GAMMA_STEP removed 2026-07-31 with focal loss - the ladder escape is the warm restart.
2026-07-25 15:55:56 -04:00
//--- FILE-COMPATIBILITY SHIM for the removed "Min OOS accuracy % (converge)" input (was MinWR, default
//--- PCT_80). Convergence is decided by the plateau ladder now, so the value has no behavioural effect
//--- anywhere - but it still occupies a slot in TWO persisted identities that must not shift:
//--- 1. the topology .cfg layout (SaveTopologyConfiguration/LoadAndCompareTopologyConfiguration), and
//--- 2. the weights-FILENAME fingerprint (BuildConfigFingerprint) - change the hash and every existing
//--- model silently becomes unreachable and retrains from era 0.
//--- So keep writing/hashing the old default, and stop COMPARING the .cfg field (see that function) so a
//--- model saved under ANY previous MinWR still loads. Models trained with the shipped default 80 keep
//--- their exact filename and resume normally; one trained under a non-default MinWR gets a new filename
//--- and retrains once, which is unavoidable when a fingerprint field stops being a variable.
# define LEGACY_CONVERGE_WR_SLOT 80
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- Same treatment for the retired StudyPeriods input (removed 2026-07-30 - training now covers all
//--- available history). Its .cfg slot is positional and cannot be deleted without invalidating every
//--- deployed file, so a constant goes in and the field is no longer compared on load.
# define LEGACY_STUDY_PERIOD_SLOT 0
2026-08-11 21:53:37 -04:00
//--- LEGACY SLOT (was m_historyBars, DERIVED since 2026-08-11 - see DeriveHistoryBars). The literal
//--- is the shipped ind_Periods default, so every model trained at it keeps its filename; the real
//--- window lives in the .cfg (adopt-don't-compare, like every other derived shape field). A user
//--- who ran a non-default ind_Periods re-keys once, correctly - their old window is gone.
# define LEGACY_HISTORY_BARS_SLOT 20
//--- Derived-window rule: median confirmed swing leg, snapped DOWN to the ladder, capped. The floor
//--- is the ADZigZag Depth the feature comment always required; the cap is a wall-clock judgment -
//--- era time scales ~linearly with the window on every topology, and with the lag profile at the
//--- noise floor there is no evidence to buy more than ~1.6x today's cost on. Fallback = the old
//--- default, used when history is too thin to trust the measurement (same contract and warning as
//--- TOPOLOGY_BUDGET_MIN_TRUSTED_BARS - a window pinned from a handful of bars lasts a model's life).
# define HISTORY_BARS_FALLBACK 20
# define HISTORY_BARS_FLOOR 12
# define WINDOW_DERIVE_SPAN_BARS 20000
# define WINDOW_DERIVE_MIN_LEGS 30
# define WINDOW_SWING_WING 12
2026-07-17 09:11:42 -04:00
//--- Minimum true OOS samples a class needs this era before its recall is trusted as a real pass -
//--- see directionalRecallOK's declaration comment for the era-44-46 false-convergence this prevents.
//--- 10 is a low bar (still lets a genuinely thin early-run OOS window fall back to "not blocking"
//--- via recallPct==-1), just enough to rule out the zero/near-zero-sample degenerate case.
# define MIN_OOS_CLASS_SAMPLES_FOR_GATE 10
fix: the recall gate was unsatisfiable and the LR decay was a spiral
Both made the run structurally unable to succeed, independently of any
signal in the data. Found by reading the 13:01 log.
RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall
each >= 40%. First-touch resolution (ce52654) collapsed Neutral from
the ~94% majority it was under exact-pivot labels to a same-bar-tie
residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model
to identify 40% of coin-flip ties before it could converge. Measured:
CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on
every era. No model could ever satisfy it; every run was destined for
the plateau ladder or the era cap.
Only the DIRECTIONAL floors are load-bearing for the anti-collapse job
the gate exists to do: an all-Neutral model shows Buy and Sell recall
at 0% and is blocked by them. Neutral's own floor guarded the mirror
bias (over-calling Buy/Sell at Neutral's expense), which was real at
94% prevalence and is not at 0.65% - there, almost never calling
Neutral is correct rather than biased.
Prevalence-guarded rather than hardcoded off, so it returns by itself
if a future label rule makes Neutral substantial again. Deliberately
NOT extended to Buy/Sell: exempting a thin directional class reopens
the era-44-46 hole, which directionalRecallMeasured only half-covers -
it checks those classes were MEASURED, not that they passed.
ETA DECAY. A regressing era restored the checkpoint, reset the
optimizer and cut eta - all on the FIRST regression. The next era then
started from an identical state with a smaller step, regressed again,
and got the same treatment. The loop is self-sustaining and cannot
discover anything, because rolling the weights back is exactly what
removes the exploration that would end it.
Measured on PAI: eras 2-11 every one a regression against era 1, eta
0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras,
~45s each, reproducing era 1 exactly and unable to do anything else.
Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the
standard ReduceLROnPlateau formulation. A single bad era is noise, and
an improving era clears the counter so alternating runs never
accumulate into a decay.
Build tag -> gate-patience-v3. It had not moved in six commits, which
is why the running binary could not be identified from its own log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- A class this rare cannot be asked to carry the per-class recall floor. Applied to NEUTRAL ONLY -
//--- see the gate itself for why the directional classes are deliberately NOT exempted by prevalence.
//--- First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot
//--- labels to a same-bar-tie residue: 250 of 38,261 bars, 0.65%. Demanding >=40% recall on that asks
//--- the model to spend capacity identifying coin-flip ties, and it is unreachable - measured
//--- 2026-08-10, CONV/LSTM/HYBRID all reported "Neutral:0% (need >=40% each)" every era, so no model
//--- could ever satisfy m_objectiveMet and every run was structurally unable to converge.
# define MIN_GATE_CLASS_SHARE_PCT 5.0
//--- Consecutive regressing eras before the checkpoint is restored and eta decayed. Was effectively 1:
//--- every regression rolled the weights back to the same checkpoint, reset the optimizer AND shrank
//--- eta, so the next era restarted from an identical state with a smaller step and regressed again -
//--- a geometric collapse with no exploration between rungs. Measured 2026-08-10 on PAI: eras 2-11 all
//--- regressed against era 1's best, eta fell 0.000594 -> 0.000024, and dW/W read 0.000%/0.000% from
//--- era 2 onward - ten eras that reproduced era 1 exactly and could not have done anything else.
//--- Patience is the standard ReduceLROnPlateau formulation and restores the exploration the rollback
//--- was removing: regress a few eras from the restored point BEFORE concluding the step is too big.
# define ETA_DECAY_PATIENCE_ERAS 3
2026-07-19 11:04:38 -04:00
//--- Bound on FindConfirmedZigZagPivot()'s backward scan (m_useSwingContext feature block) - ADZigZag's
//--- own stock defaults (Depth=12) produce pivots frequently enough in normal conditions that this cap
//--- should rarely bind, but a long, unusually strong single-direction run could genuinely go this long
//--- without a confirmed opposite-type pivot. Generous rather than tight: the scan is cheap (plain array
//--- reads, no indicator recompute) and its result is cached per-bar by BufferTempData()'s feature cache,
//--- so the one-time cost of a long scan is paid at most once per unique bar, not per training pass.
# define SWING_SCAN_CAP_BARS 750
2026-07-15 21:47:37 -04:00
//--- EMA shadow-weight deployment blend rate - see m_shadowNet's declaration comment for the full
//--- rationale. 0.01 matches the Tau range (0.001-0.01) used for target-network soft updates in
//--- Dmitriy Gizlyk's reference NeuroNet_DNG-based RL algorithms (references\MQL5\Experts\*\Study.mq5) -
//--- small enough that no single era's raw weights can move the deployed model far, large enough that
//--- the shadow still tracks real, sustained learning within a few dozen eras rather than lagging
//--- forever.
# define SHADOW_WEIGHT_TAU 0.01
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1)
Completes the 2026-08-09 training audit. FORCES A RETRAIN of every
Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be
redeployed alongside the .ex5 - they carry new exports.
F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online
SGD (one weight update per bar), which is the mechanical source of the
era-to-era whipsaw every downstream guard was built to cope with. The O(n^2)
outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv /
AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the
optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so
there is one Adam/SGD implementation instead of four that can drift.
- the LSTM needs no outer-product kernel (WeightsGradient already holds the
sample's full dW) but could NOT simply be left un-zeroed between samples:
CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a
separate accumulator plus an elementwise add.
- batch-norm gamma/beta accumulate in host arrays, not new BatchOptions
slots - BN_OPT_STRIDE is baked into every persisted .nnw.
- scoped to pass 2; online learning keeps immediate updates. Every save /
checkpoint / scoring boundary flushes, scaling by the real sample count.
- degrades to per-sample updates (one log line) on a tier that cannot
accumulate, so old devices and DLL-free builds are unaffected.
- verified offline: DirectML/batch_accum_check.cpp drives the real exports
against an independent reference; at B=1 the accumulator matches the
shipped unbatched kernel's own gradient to 1.1e-16. Math only - the
in-situ check remains the per-layer dW/W report on a real era.
F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a
conv/LSTM front end had already reduced it, so an LSTM's dense stack was
charged for 1,280 inputs when it receives 64. Confirmed from the deployed
.cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now
budgeted against the front-end output and capped at it (never fan out), with
the derivation reordered so both stages settle first.
N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing
direction and Wyckoff stage into one scalar across a sign discontinuity. Split
into direction + [0,1] magnitude, the same convention the base OHLC block uses.
Information-preserving; 13 readings now occupy 16 inputs.
Compiled clean (0 errors, 0 warnings); both DLLs rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
//--- MINI-BATCH SIZE (2026-08-09 audit, F4). Number of samples whose gradients are summed before ONE
//--- optimizer step is taken. 1 restores the exact per-sample behaviour this engine had until now.
//---
//--- Why it existed as a problem: training was pure online SGD - one weight update per bar - so the
//--- gradient driving each update was a single noisy sample and the end-of-era weight state was a
//--- high-variance draw. That is the mechanical source of the era-to-era whipsaw the plateau ladder,
//--- the checkpoint restore and the shadow EMA were all built to cope with downstream. Update noise
//--- falls as ~1/sqrt(B), so 32 cuts it by ~5.7x while taking 32x fewer (much better-estimated) steps.
//---
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- REVISED 2026-08-09 (32 -> 8) after the first run at 32 regressed every topology. Two lessons,
//--- both measured, both now handled here rather than left implicit:
//---
//--- 1. An era is one PASS OVER THE DATA, and everything downstream that counts progress - the
//--- plateau ladder's patience, the selection metric, the eta anneal - is denominated in eras.
//--- Raising B does not change how much data an era sees; it divides how many optimizer STEPS
//--- that era takes. At 32 the perceptron declared convergence at era 41 on ~49k updates, where
//--- the same config had still been finding new bests at era 1028. The ladder had not become
//--- wrong, it had become 32x more impatient in the only unit it can measure.
//--- 2. Fewer steps must be paid for with a larger step. For SGD the compensation is linear in B
//--- (Goyal et al. 2017); for ADAPTIVE methods it is sqrt(B) - Krizhevsky 2014, derived for Adam
//--- specifically in Granziol et al. 2022. TrainBatchLrScale() below applies it.
//---
//--- 8 rather than 32 because the two costs compound: after the sqrt(B) LR bump an era still makes
//--- sqrt(B) less progress than the per-sample path, so PLATEAU_PATIENCE_ERAS has to stretch by the
//--- same sqrt(B) to stay equivalent (see TrainPlateauPatienceEras). At 32 that is 8 -> 45 eras per
//--- stage on a topology already taking 70 s/era; at 8 it is 8 -> 23, which is affordable. Noise
//--- still falls as 1/sqrt(B), so 8 keeps ~2.8x of the variance reduction that was the point.
//---
//--- 1 restores the exact per-sample behaviour this engine had until now: no accumulation, no LR
//--- scaling, no patience stretch (every helper below is an identity at B=1).
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1)
Completes the 2026-08-09 training audit. FORCES A RETRAIN of every
Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be
redeployed alongside the .ex5 - they carry new exports.
F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online
SGD (one weight update per bar), which is the mechanical source of the
era-to-era whipsaw every downstream guard was built to cope with. The O(n^2)
outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv /
AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the
optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so
there is one Adam/SGD implementation instead of four that can drift.
- the LSTM needs no outer-product kernel (WeightsGradient already holds the
sample's full dW) but could NOT simply be left un-zeroed between samples:
CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a
separate accumulator plus an elementwise add.
- batch-norm gamma/beta accumulate in host arrays, not new BatchOptions
slots - BN_OPT_STRIDE is baked into every persisted .nnw.
- scoped to pass 2; online learning keeps immediate updates. Every save /
checkpoint / scoring boundary flushes, scaling by the real sample count.
- degrades to per-sample updates (one log line) on a tier that cannot
accumulate, so old devices and DLL-free builds are unaffected.
- verified offline: DirectML/batch_accum_check.cpp drives the real exports
against an independent reference; at B=1 the accumulator matches the
shipped unbatched kernel's own gradient to 1.1e-16. Math only - the
in-situ check remains the per-layer dW/W report on a real era.
F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a
conv/LSTM front end had already reduced it, so an LSTM's dense stack was
charged for 1,280 inputs when it receives 64. Confirmed from the deployed
.cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now
budgeted against the front-end output and capped at it (never fan out), with
the derivation reordered so both stages settle first.
N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing
direction and Wyckoff stage into one scalar across a sign discontinuity. Split
into direction + [0,1] magnitude, the same convention the base OHLC block uses.
Information-preserving; 13 readings now occupy 16 inputs.
Compiled clean (0 errors, 0 warnings); both DLLs rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
//---
//--- Compile-time, not an input: it changes training dynamics but not the trained model's SHAPE, so it
//--- has no business in the weights fingerprint, and a user who picks a batch size is tuning something
//--- they cannot measure from the panel. Not every tier can honour it - see CNet::BatchSize.
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
# define TRAIN_BATCH_SIZE 8
fix(train): clamp the sweep to indicator-servable depth - the scan wall was CopyBuffer, not a cold indicator
Symptom: on a 3-chart run with contention ruled out (SP500 sitting at era 2552),
USDJPY and XAUUSD produced 0 usable windows out of 50,163 and 33,966 - forever,
re-sweeping on every discard, which is the panel oscillating 0->100%.
Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it
in its own context bounded by "Max bars in chart" (TERMINAL_MAXBARS), and
CopyBuffer past that limit does not short-read, it FAILS - so CDoubleBuffer keeps
nothing and EVERY index answers EMPTY_VALUE. ADMovingAverage is the only custom
indicator whose feature block REJECTS on EMPTY_VALUE (ADZigZag, also CiCustom,
neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins served at any depth), so the
sweep died on feature 25 of every bar while the 24 price features under it were
fine. That is exactly the "window had 24 of 832 values" the stall report named.
Perfectly depth-correlated, measured 2026-08-17:
SP500 16,234 bars -> era 2552 XAUUSD 33,982 -> 0 windows
XTIUSD 16,611 bars -> era 71 USDJPY 50,179 -> 0 windows
This RETIRES the 2026-08-17 cold-indicator reading of the same stall. f0cf659 was
right that the rejection must be transient and that the dead backoff had to arm -
the branch did change to 'cold-indicator backoff' - but waiting cannot fix a depth
the terminal will never grant. So Train() now clamps to TunableBarsCalculated()
(which existed and was only ever used for a tuner printout) and trains on the
history that IS available, naming TERMINAL_MAXBARS in the log so the cause is
readable next time. m_coldSweepTick still owns the genuinely transient case: that
reads back as -1, not a positive short count. Recomputed per era, so the clamp
lifts by itself if the setting is raised.
Also fixes a real off-by-one it was hiding: the MA block reads GetData(idx) AND
GetData(idx + 1) for its bar-over-bar change, but ResizeBuffers sized m_MA to
barIndex exactly - so the deepest bar of every sweep read one past the end and was
rejected as cold. Same shape as the +ichiKijun the Ichimoku/close pair already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:27:14 -04:00
//--- Floor for Train()'s indicator-depth clamp. Above it, a short-but-real history beats livelocking on a
//--- depth the terminal will never serve; below it, a positive-but-tiny BarsCalculated() is far more
//--- likely an indicator part-way through its first calculation than a hard cap, so the clamp stands
//--- down and leaves the case to m_coldSweepTick's backoff. Sized so a clamped era still holds an OOS
//--- window worth measuring rather than a few hundred bars of noise.
# define TRAIN_MIN_CLAMPED_BARS 2000
feat(depth): prime -> settle -> sweep, and name which handle is short
"Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in
1dda479 was wrong. Two other candidate causes are falsified too: the price series
is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new
H4 bar), and the handles have been stable since 13:07 with zero windows for the 17
minutes after, so it is not download-in-progress and not handle churn. The MA period
tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not
indicator cost either.
What IS verified stays verified: CopyBuffer past the calculated depth fails outright
rather than short-reading, so the buffer holds nothing and every index reads
EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag
neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25
of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated:
16k-bar charts train, 34k/50k get zero windows forever.
So the WHY is still open, and this fix does not depend on it. Per the user's protocol:
the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated()
every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever
it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned
depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two
training paths, which snapshotted a value that may still have been climbing; the clamp
remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b).
The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature
scan starves the indicator threads the request just woke, which is how the failure
sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the
0->100% oscillation on the panel.
Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and
stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the
cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
//--- SettledBars()'s wait. Probe spacing is a compromise: long enough that a busy terminal makes visible
//--- progress between reads (so growth is not mistaken for steadiness), short enough that a chart with
//--- nothing to wait for loses only seconds. Three steady probes ~= 9s of no movement before the depth is
//--- believed, which is cheap next to the 40-minute stalls it replaces.
# define DEPTH_SETTLE_PROBE_MS 3000
# define DEPTH_SETTLE_STABLE_PROBES 3
//--- Hard stop on the wait. A depth that has not settled in 10 minutes is not going to, and training on
//--- the history that IS there beats waiting forever - the give-up is logged as such so a settled depth
//--- is never confused with an abandoned one.
# define DEPTH_SETTLE_TIMEOUT_MS 600000
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- ERA-BARRIER LIVENESS. How long a member may sit on the same era before the barrier stops treating
//--- it as a member the others must wait for. EnsembleMinTrainingEra()'s exemption list covered only
//--- VOLUNTARY non-participation (deployed / stopped / paused) and its comment claimed "nothing
//--- deadlocks" on that basis - but a member that simply CANNOT finish an era is none of those three,
//--- so it pinned the minimum at its own era forever and every healthy member on the chart waited on it
//--- indefinitely. Observed 2026-08-17: USDJPY and XAUUSD, LSTM and ConvLSTM stuck at era 0 producing
//--- zero usable windows, with Perceptron and Convolutional frozen behind them for 38 minutes and
//--- SILENT while they waited (the hold reset the stall watchdog's clock, so the two members that could
//--- have reported the outage were the two the outage muted).
//--- 12 min: comfortably past the slowest healthy era on the deepest chart, so a slow member is never
//--- mistaken for a dead one.
# define ENSEMBLE_BARRIER_STUCK_MS 720000
//--- How often a member held at the barrier says so in the journal. The panel line is written every
//--- call; this is the durable record, and without it a frozen chart leaves no trace at all.
# define ENSEMBLE_BARRIER_REPORT_MS 120000
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first
THE ANSWER, off the instrumentation added in be39674, first run:
ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable
indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ...
Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982
MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not
short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem;
the previous session's five theories were all answering the wrong question.
And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on
that same chart at 15:24:18 and went on to train normally (feature health, 51
features, excursion head) reading the same indicator. Only ConvLSTM's handle -
the last member constructed - was dead. WHY is still not established. All four
members request ADMovingAverage with identical params, so MT5 hands them the SAME
refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over
exactly that shared handle; that is the obvious suspect and it is NOT yet proven,
so this commit does not act on it.
1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths.
"MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982
(h12)" says it is the SAME handle and someone released it; "(h-1)" says it was
never created. That is the difference between a refcount bug and a creation
failure and it is one field. This is the measurement the shared-handle suspicion
needs before anyone acts on it.
2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT.
A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles()
re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD
indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does
NOT release first: -1 means the terminal no longer knows the handle, so there is
nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could
decrement whatever now owns that number. 30s cooldown, because every ServableBars()
consumer reaches it including live inference on every tick. The feature cache is
dropped with it, and the log names before/after depths.
Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars
against a buffer that answers EMPTY_VALUE at every index - then discarding the era
and doing it again - is not a recovery.
3. THE SWEEP NOW HOLDS ON A DEAD HANDLE.
ServableBars() keeps answering `want` (its contract; live inference and online
learning have their own refusal paths and a 0 there reads as "no history at all").
SettledBars() - the training sweep's entry, the one caller that can afford to wait -
returns 0 instead, so Train() holds and reports rather than burning a full-history
pass it is guaranteed to throw away. A recreated handle is cold, so it primes
through the existing settle path on the next call. If the repair fails the member
holds indefinitely and says so every minute, and be39674's barrier liveness escape
releases the rest of the ensemble after 12 minutes - which is the correct
degradation and is exactly what the log shows happening.
4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST.
Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of
Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the
next free row on each member's FIRST PublishStatus() call, so the order was a race -
the members busy sweeping published before the ones sitting idle at the era barrier,
and be39674 sharpened it by (correctly) making a held member stop writing the terse
line. Rows are now keyed to m_ensembleIndex, the registration/construction order,
which is fixed for the life of the chart. Claimed on every publish rather than once,
so it is idempotent and refreshes the tag for a member whose ID was not final when it
first published (the config-tag suffix is appended during InitIndicators, after
EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded
from the model count, so a member that has not published yet leaves no gap and shifts
nobody.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
//--- Minimum gap between attempts to rebuild a dead indicator handle (see RepairDeadIndicatorHandles).
//--- Long enough that a terminal which is genuinely refusing to create the indicator is not hammered,
//--- short enough that a member recovers within one stall-report interval rather than one era.
# define HANDLE_REPAIR_COOLDOWN_MS 30000
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- sqrt(B) learning-rate compensation and the matching patience stretch. Both are exactly 1.0 at
//--- B=1, so the whole mini-batch apparatus vanishes when TRAIN_BATCH_SIZE is 1.
//---
//--- Note this is only sound because the Adam second moment was fixed on 2026-08-09 (see
//--- AI\Network.cl's UpdateWeightsAdam): the previous recursion pinned its denominator at ~b2 for any
//--- gradient below unit scale, which made the step size proportional to |g| instead of invariant to
//--- it. Under THAT optimizer, shrinking the gradient by averaging cost a further sqrt(B) on top of
//--- the B fewer steps, and no learning-rate rule stated in terms of B could have compensated.
double TrainBatchLrScale ( void ) { return MathSqrt ( ( double ) TRAIN_BATCH_SIZE ) ; }
int TrainPlateauPatienceEras ( void ) { return ( int ) MathRound ( PLATEAU_PATIENCE_ERAS * MathSqrt ( ( double ) TRAIN_BATCH_SIZE ) ) ; }
2026-07-24 11:52:19 -04:00
//--- Online continual-learning tunables (see OnlineLearnStep()). Live-chart-only: once a model is
2026-08-01 11:27:28 -04:00
//--- deployed (m_trainingComplete) it keeps adapting to newly-RESOLVED bars as their barrier outcome
//--- becomes known - the same supervised triple-barrier task it was trained on, never trade P&L
//--- (that would be RL).
2026-07-24 11:52:19 -04:00
//--- ONLINE_LEARN_MAX_CATCHUP caps how many newly-confirmed bars one step may backprop, so a weekend
//--- gap or a restart can't stall a tick with an unbounded catch-up loop (excess is picked up over the
//--- next few bars). ONLINE_ACC_SMOOTH is the guardrail EMA horizon (bars) for the rolling predict-
//--- before-learn accuracy - far shorter than CNet::recentAverageSmoothingFactor (10000) so the
//--- guardrail actually reacts within a realistic live sample count. The deployed shadow is only blended
//--- toward Net while that rolling accuracy holds at/above max(ONLINE_LEARN_MIN_ACC, deploy_baseline -
//--- ONLINE_LEARN_ACC_MARGIN); if it drops, Net keeps adapting (so it can recover) but the blend is
//--- FROZEN so drift can never reach live - the conservative "reject the deployment of a bad update"
//--- guardrail (no weight-snapshot/revert needed). Persist every ONLINE_LEARN_PERSIST_EVERY updates so
//--- a crash loses at most that many bars of adaptation.
2026-07-29 00:03:54 -04:00
//---
2026-08-01 11:27:28 -04:00
//--- CLASS IMBALANCE. This path streams the RAW live class distribution one bar at a time, so calling
//--- backProp() at the default sampleWeight of 1.0 makes it a majority-class drift vector by
//--- construction - it would pull a balanced, converged model back toward Neutral. The streaming-safe
//--- correction is COST-level: alpha-balanced focal loss (Lin et al. 2017, "Focal Loss for Dense Object
//--- Detection", eq. 5), weight = alpha_c * (1 - p_t)^gamma, a SINGLE loss designed for exactly this
//--- ratio. alpha_c is inverse class frequency from the persisted priors (m_priorBuy/Sell/Neutral),
//--- majority normalised to 1.0, scaled by ONLINE_LEARN_PARITY and capped; gamma is
//--- ONLINE_LEARN_FOCAL_GAMMA. ONLINE_LEARN_MAX_CLASS_WEIGHT is the uncapped ceiling,
//--- ONLINE_LEARN_ALPHA_CAP the tighter one actually applied.
//--- The two engines deliberately use DIFFERENT corrections. Train() corrects analytically in the
//--- gradient via the logit-adjusted loss, which is NOT available here: ApplyLogitAdjustment() only
//--- runs inside a training run, so a deployed-then-reloaded model carries no offsets and would
//--- otherwise stream skewed data in with no correction at all. These were shared inputs until
//--- 2026-07-31 and are now constants at those inputs' shipped defaults - behaviour is unchanged.
2026-07-29 00:03:54 -04:00
//--- ONLINE_LEARN_ETA_SCALE: this path previously inherited whatever value the GLOBAL `eta` happened to
//--- be left at by the last Train() chunk of ANY model instance (PAI/CONV/LSTM/HYBRID share it), which
//--- is arbitrary. eta is now pinned to this model's own converged rate scaled down for the duration of
//--- the loop, then restored, so a live adaptation step is deliberately gentler than a training step.
2026-07-24 11:52:19 -04:00
# define ONLINE_LEARN_MAX_CATCHUP 64
# define ONLINE_ACC_SMOOTH 50.0
# define ONLINE_LEARN_WARMUP 20
# define ONLINE_LEARN_MIN_ACC 40.0
# define ONLINE_LEARN_ACC_MARGIN 10.0
# define ONLINE_LEARN_PERSIST_EVERY 32
2026-07-29 00:03:54 -04:00
# define ONLINE_LEARN_MAX_CLASS_WEIGHT 5.0
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- Pinned to the shipped defaults of the removed OversampleParity (90%), ConstrainReplay (true ->
//--- cap 3.0) and FocalLossGamma (1.0) inputs - see the CLASS IMBALANCE note above.
# define ONLINE_LEARN_PARITY 0.9
# define ONLINE_LEARN_ALPHA_CAP 3.0
# define ONLINE_LEARN_FOCAL_GAMMA 1.0
2026-07-29 00:03:54 -04:00
# define ONLINE_LEARN_ETA_SCALE 0.25
2026-07-18 14:56:41 -04:00
//--- Free function, not a class method: needed by CExpertSignalAIBase's constructor init list for
//--- both m_modelEta and m_etaCeiling (see their declaration comments), which runs before member-init
//--- order could safely make one depend on another - this only touches the TrainingOptimizer input
//--- (Variables\Inputs.mqh) and the SgdLearningRate/AdamLearningRate inputs (AI\Network.mqh's `lr`
//--- macro resolves to AdamLearningRate), never class members.
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- The sqrt(B) mini-batch compensation is applied HERE, at the one point that decides the base rate,
//--- so it reaches m_etaCeiling, m_modelEta, the plateau ladder's boost and its anneal from a single
//--- edit rather than being re-derived at each of them. Identity at TRAIN_BATCH_SIZE 1.
2026-07-18 14:56:41 -04:00
double InitialEtaForOptimizer ( void )
{
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
return ( ( TrainingOptimizer = = SGD ) ? SgdLearningRate : lr ) * TrainBatchLrScale ( ) ;
2026-07-18 14:56:41 -04:00
}
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric
Four of the six findings from research/training_pipeline_audit_2026-08-09.md
(F4 mini-batching and F6 feature re-encode deliberately deferred - see the
report's implementation-status section for why):
- F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%,
which is 15-bit - provably non-uniform on every full-history era over 32,768
queued samples. New 30-bit ShuffleRandomIndex().
- F2: plateau warm restarts were a no-op whenever eta already sat at its
ceiling (the normal state of a non-regressing plateau) - the ladder was just
a 24-era countdown. Restarts now overshoot to 5x the ceiling
(PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience
window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has
real range.
- F3: checkpoint restores put weights back but kept the rejected trajectory's
Adam moments, so the optimizer immediately pushed back toward the rolled-back
state (the restore->regress->restore oscillation). CNet::ResetOptimizerState()
zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta
untouched) on every mid-run restore, every boosted restart, and the
deploy-time restore that online learning continues from.
- F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk,
so the selection metric the checkpoint ranking and deploy gate read is a pure
function of the checkpoint instead of partly measuring BN drift. Defensive
unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and
the OOS continual-learning simulation stay adaptive by design.
Compiled clean (0 errors, 0 warnings) via the staged-tree recipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//+------------------------------------------------------------------+
//| Uniform random index in [0, n) for Fisher-Yates shuffles. |
//| MQL5's MathRand() is 15-bit (0..32767). `MathRand() % n` with |
//| n > 32768 makes the modulo the IDENTITY on the generator's range: |
//| a swap target above 32767 can never be drawn, so any shuffle over |
//| more elements than that - every full-history H1 training queue - |
//| was provably non-uniform, partially re-admitting the correlated |
//| same-class gradient runs the shuffle exists to break (2026-08-09 |
//| audit, F1; see AI\Network.mqh's MAX_WEIGHT_DELTA comment for why |
//| those runs matter). Two draws give 30 uniform bits; the residual |
//| modulo bias at n ~ 1e5 against 2^30 is ~1e-4 - negligible here. |
//+------------------------------------------------------------------+
int ShuffleRandomIndex ( const int n )
{
if ( n < = 1 )
return 0 ;
int r30 = ( ( MathRand ( ) & 0x7FFF ) < < 15 ) | ( MathRand ( ) & 0x7FFF ) ;
return r30 % n ;
}
2026-07-14 22:36:27 -04:00
class CExpertSignalAIBase : public CExpertSignalCustom
{
protected :
string ID ;
2026-07-30 13:39:08 -04:00
//--- ID with the bracketed config tag stripped - "Hybrid 3L [HYB-9369]" -> "Hybrid 3L". The tag is a
//--- topology prefix plus a fingerprint hash: it exists to tell one CHART's model files from another's
//--- when reading the journal or the State\ folders, which is a developer's problem, not an owner's.
//--- Logs and the VerboseMode panels keep the full ID; the plain-language panels use this. Strips from
//--- the LAST " [" so a model name containing a bracket could not truncate the whole label.
string DisplayName ( void ) const
{
int cut = StringFind ( ID , " [ " ) ;
int next = cut ;
while ( next > = 0 )
{
cut = next ;
next = StringFind ( ID , " [ " , cut + 1 ) ;
}
return ( cut > = 0 ? StringSubstr ( ID , 0 , cut ) : ID ) ;
}
fix(ensemble): per-member arrow namespaces; ConvLSTM rename; dialog in purge list
The ensemble chart UI had a shared-namespace defect that answered the user
question "what do the arrows represent?" with "a bug": all four members drew
arrows under the same WarSig_<bartime> object names, so the chart showed
whichever member rendered LAST, one member Neutral deleted another member Buy
at the same bar, each member init sweep wiped the arrows the previous member
had just restored, and SaveChartSignals - which rebuilds the sidecar by
SCANNING the chart - persisted every other member arrows into its own history
(the exact cross-model laundering its own header warns about, now happening
BETWEEN ensemble members).
Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_,
WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor
purge and the sidecar scan are all member-scoped, and the tooltip names the
model. Global purges keep matching the bare WarSig_ prefix, which covers all
member namespaces plus old-format leftovers from earlier builds.
Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is
one member; the header is the ensemble) and the CONVLSTM member displays as
ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the
model folder and changing it would orphan every model trained under that path.
Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins
WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now
cover it - it was in neither list, so a dialog starved of its own Destroy()
left its controls on the chart permanently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 18:26:55 -04:00
//--- Per-MEMBER arrow namespace: "WarSig_PAI_", "WarSig_CONV_", ... All four ensemble
//--- members used to share the bare SIG_ARROW_PREFIX + bar-time name, which meant (a) the
//--- chart showed whichever member rendered LAST, not four opinions; (b) one member's
//--- Neutral deleted another member's arrow at the same bar; and (c) SaveChartSignals -
//--- which rebuilds the sidecar by SCANNING the chart - persisted every OTHER member's
//--- arrows into its own history, the exact cross-model laundering its header warns
//--- about. Global purges still work: they match on the bare "WarSig_" prefix, which
//--- prefixes all member namespaces (and any old-format leftovers from earlier builds).
string ArrowPrefix ( void ) const { return SIG_ARROW_PREFIX + m_id + " _ " ; }
feat(chart): filtered view - one arrow per trade the bot would actually take
Adds DrawUnfilteredSignals (default OFF) and, with it off, replaces the
per-model arrow layer with the decision the EA would really have made.
THE FILTERED ARROW IS DRAWN AT THE ORDER, NOT AT THE THRESHOLD. Clearing
Min_Vote_Open is not the same as trading: a setup can pass the vote and still
never reach the broker (invalid SL/TP, stops-level, ATR warm-up, unsynced
swing history), and every one of those lands in OpenParams' failure branch.
So DrawVoteArrow() fires only after the order parameters validate, and the
failure branch withdraws any arrow already standing on that bar. One arrow is
one entry the EA would have placed - carrying the vote, the threshold it
cleared, and the SL/TP the order would have had.
Classic signals now draw too, under their own name and weight, so a chart
running MA/RSI/MACD/Ichimoku alongside the nets reads the same way an
ensemble chart does. They can only be drawn from the aggregate's once-per-bar
pass, because unlike the AI members they have no cached per-bar scan.
Two subtleties that would each have produced a quietly wrong chart:
- The raw classic draw sits AFTER filter.Direction(), not beside the
journaling block. GetActivePattern*() are CONSUMING reads holding the
PREVIOUS evaluation - "one tick later", which at Expert_EveryTick=false is
one BAR later. Keyed off those and placed at StartIndex(), every classic
arrow would have been drawn one bar early, which on a chart is
indistinguishable from a model that genuinely leads. Peek*() accessors
(non-consuming) let pattern, weight and bar come from one evaluation.
- CExpertSignalAIBase::DrawObject() early-returns instead of gating its five
call sites, so the switch cannot be honoured in three passes and missed in
the fourth. Its delete counterparts stay ungated so flipping the input off
and rescanning clears the raw layer rather than stranding it.
SIG_ARROW_PREFIX and g_signalsVisible move from ExpertSignalAIBase.mqh down
to ExpertSignalCustom.mqh - the nearest common ancestor - because the classic
signals cannot see the AI header (it is included later in Warrior_EA.mq5).
The vote layer gets SIG_VOTE_PREFIX under the same bare prefix, so
WarriorChartPrefixes()' purge still reaches every arrow without knowing they
exist.
NOT YET BUILT: the reconstructed history behind attach. Filtered arrows
currently start where the EA starts. See the next commit.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:41:28 -04:00
//--- Yes: every subclass of this one is a neural net. Tells the aggregate's raw-arrow layer that
//--- this filter draws its OWN arrows (from its cached per-bar scans, which span the whole chart)
//--- and must not be drawn a second time from the once-per-bar live path. See
//--- CExpertSignalCustom::IsAIFilter for why this is a virtual and not an id comparison.
virtual bool IsAIFilter ( void ) const override { return true ; }
2026-07-14 22:36:27 -04:00
CiOpen m_Open ;
CiClose m_Close ;
CiHigh m_High ;
CiLow m_Low ;
CiVolumes m_Volumes ;
CiTime m_Time ;
2026-07-26 18:33:12 -04:00
//--- optional classic-indicator input features (m_useMA/m_useRSI/m_useMACD/m_useIchimoku below) -
//--- independent instances from the ones Signals\SignalMA.mqh/SignalRSI.mqh/SignalMACD.mqh/
//--- SignalIchimoku.mqh use for trade voting, since this class and those CSignal* classes are
//--- unrelated hierarchies (feature engineering vs. signal vote). The MA
feat: add unified MA type support to indicator tuner
Add `MA_TYPE_PRESETS` enum covering advanced (ALMA, DEMA, ZLEMA, T3, Kalman) and standard (SMA, EMA, SMMA, LWMA) moving averages. Integrate `maType` and `bestMaType` into `CADIndicatorTuner` struct, update flatten/unflatten routines, and bump `AD_TUNE_PARAM_COUNT` to 33. This allows the auto-tuner to search over MA type alongside period, improving feature discovery.
2026-07-23 15:02:09 -04:00
//--- feature now uses the SAME unified indicator as the classic vote (CustomIndicators\ADMovingAverage.mq5)
//--- via CiCustom, so its type/period are tunable (see ADIndicatorTuner maType/maPeriod).
CiCustom m_MA ;
2026-07-22 17:17:23 -04:00
CiRSI m_RSI ;
2026-07-26 18:33:12 -04:00
//--- built-in Ci* wrappers, not CiCustom: MACD is defined on plain EMAs and Ichimoku on plain
//--- highest/lowest midpoints, so there is no unified AD* custom indicator to route them through the
//--- way m_MA goes through ADMovingAverage. Periods come from the tuner (macdFast/macdSlow/
//--- macdSignal, ichiTenkan/ichiKijun/ichiSenkou).
CiMACD m_MACDFeature ;
CiIchimoku m_Ichimoku ;
2026-07-14 22:36:27 -04:00
//--- custom price-action/volume indicators (CustomIndicators\*.mq5), loaded via iCustom/CiCustom
CiCustom m_ADCumulativeDelta ;
CiCustom m_ADShorteningOfThrust ;
CiCustom m_ADWyckoffEventStream ;
CiCustom m_ADWyckoffFailedStructure ;
CiCustom m_ADWyckoffSignificantBarInversion ;
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth
Three findings from the 2026-08-11 audit:
1. The excursion head's trailing-quantile ring was deliberately never cleared
between eras ("a rolling estimate of the market, not of the era") - but
pass 3 re-walks the SAME OOS window every era, so at each walk's restart
the ring still held the outcome masks of the newest OOS bars from the
previous walk: the chronological FUTURE of the bars about to be scored.
For the first ~window+horizon pushes of every era the "trailing" incumbent
was partly a leading one - conservative for the gate (an informed incumbent
is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts.
The ring now clears at era-score reset; the warm-up bars simply don't score
the trail race, which the m_excTrailN gating already accounts for.
2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage)
against the incumbent's subset sum - valid only if head skill is uniform
across the OOS walk, while the trail-scored subset systematically excludes
each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/
m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593
made every scored bar disjoint). The dead trio is replaced by
m_excBrierHeadT: the head's Brier accumulated only on the bars the warm
incumbent also scored, so the race now compares both predictors on an
identical bar set.
3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a
cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the
sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so
BufferTempData cached an all-zero Wyckoff block as a success for the whole
bar frame: the one path the f6150ee only-cache-successes rule cannot see,
because it never fails (the ba13eef class, arriving through values that
never fail; a resumed model's era-0 prebuild starts milliseconds after
OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means
async warm-up (transient reject, retried), while deep bars beyond the
buffered depth keep the sanitize loop's neutral-fill so degraded history
still trains. Also fixed m_featureCacheValid's declaration comment, which
still described the pre-f6150ee cached-miss semantics.
Compile: 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
//--- "is this AD indicator still calculating?" probe - see the definition in Features.mqh for
//--- why cold must be a TRANSIENT rejection and not a zero-fill (2026-08-11)
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
bool ADIndicatorCold ( CiCustom & ind , string block ) ;
2026-08-13 10:23:11 -04:00
//--- GetTickCount() stamp of the last pass-1 sweep in which EVERY window failed on a transient
//--- cause (cold indicator). Non-zero arms a short era-start backoff so the retry loop stops
//--- starving the very indicator threads it is waiting on - see Train()'s fresh-era block.
uint m_coldSweepTick ;
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate
1dda479 clamped the training sweep. It left five other paths asking the indicators
for a depth they cannot serve, and on a live account the quiet ones are worse than
the stall was - a stalled chart is visible, a chart trading on a degraded feature
window is not.
ServableBars(want, context) is now the single gate, and all six go through it:
training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small
positive BarsCalculated is warm-up, which m_coldSweepTick owns)
label prebuild clamp - labels come from price/ADZigZag and would survive a
capped MA, but ResizeBuffers sizes EVERY buffer and a failed
CopyBuffer leaves m_MA EMPTY for the next reader, so this path
could silently re-break the block Train()'s clamp just fixed
live inference HOLD. Below `need` the swing block takes its degraded path and
inference runs on a different feature distribution than the model
was fitted on. This EA sizes real positions off that output, so
no signal beats a mismatched one
online learning HOLD, same reason and worse - this path WRITES to a live trading
model, so a mismatched (features, label) pair is not a wrong arrow,
it is a wrong weight update that compounds every bar
chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest
"Max bars in chart" is also 5000, so this one is genuinely
reachable; uncapped it repaints the window all-Neutral
research export clamp before the emptiness test, so a capped symbol exports the
depth it has rather than writing a CSV with a dead feature block -
an artefact that looks complete and is silently wrong
Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars
(16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so
the failure mode is unreachable rather than merely unlikely.
Not changed: a genuinely SHORT price history still takes the old degraded path at
every site. That is pre-existing behaviour and narrowing it would mute charts that
trade today, so it stays a separate decision rather than a side effect of this fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
//--- Last depth ServableBars() had to clamp to because the tunable indicators could not serve what was
//--- asked. Held only so the explanation prints when the cap CHANGES rather than once per era per call
//--- site; it depends on BarsCalculated() alone, never on the requested depth, so it is comparable
//--- across all of them. 0 = never clamped.
fix(train): clamp the sweep to indicator-servable depth - the scan wall was CopyBuffer, not a cold indicator
Symptom: on a 3-chart run with contention ruled out (SP500 sitting at era 2552),
USDJPY and XAUUSD produced 0 usable windows out of 50,163 and 33,966 - forever,
re-sweeping on every discard, which is the panel oscillating 0->100%.
Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it
in its own context bounded by "Max bars in chart" (TERMINAL_MAXBARS), and
CopyBuffer past that limit does not short-read, it FAILS - so CDoubleBuffer keeps
nothing and EVERY index answers EMPTY_VALUE. ADMovingAverage is the only custom
indicator whose feature block REJECTS on EMPTY_VALUE (ADZigZag, also CiCustom,
neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins served at any depth), so the
sweep died on feature 25 of every bar while the 24 price features under it were
fine. That is exactly the "window had 24 of 832 values" the stall report named.
Perfectly depth-correlated, measured 2026-08-17:
SP500 16,234 bars -> era 2552 XAUUSD 33,982 -> 0 windows
XTIUSD 16,611 bars -> era 71 USDJPY 50,179 -> 0 windows
This RETIRES the 2026-08-17 cold-indicator reading of the same stall. f0cf659 was
right that the rejection must be transient and that the dead backoff had to arm -
the branch did change to 'cold-indicator backoff' - but waiting cannot fix a depth
the terminal will never grant. So Train() now clamps to TunableBarsCalculated()
(which existed and was only ever used for a tuner printout) and trains on the
history that IS available, naming TERMINAL_MAXBARS in the log so the cause is
readable next time. m_coldSweepTick still owns the genuinely transient case: that
reads back as -1, not a positive short count. Recomputed per era, so the clamp
lifts by itself if the setting is raised.
Also fixes a real off-by-one it was hiding: the MA block reads GetData(idx) AND
GetData(idx + 1) for its bar-over-bar change, but ResizeBuffers sized m_MA to
barIndex exactly - so the deepest bar of every sweep read one past the end and was
rejected as cold. Same shape as the +ichiKijun the Ichimoku/close pair already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:27:14 -04:00
int m_indicatorDepthCapBars ;
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- One-shot latch for the "an enabled tunable indicator reports NO calculated bars" report in
//--- ServableBars(). Distinct from the cap latch above because it is a different condition: the cap
//--- means "serves less than asked" (recoverable, train on what there is), this means "serves
//--- nothing at any index" (a dead handle - there is no depth to clamp to). Cleared as soon as a
//--- real depth comes back so a second outage is reported rather than swallowed.
bool m_indicatorDepthDeadWarned ;
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first
THE ANSWER, off the instrumentation added in be39674, first run:
ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable
indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ...
Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982
MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not
short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem;
the previous session's five theories were all answering the wrong question.
And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on
that same chart at 15:24:18 and went on to train normally (feature health, 51
features, excursion head) reading the same indicator. Only ConvLSTM's handle -
the last member constructed - was dead. WHY is still not established. All four
members request ADMovingAverage with identical params, so MT5 hands them the SAME
refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over
exactly that shared handle; that is the obvious suspect and it is NOT yet proven,
so this commit does not act on it.
1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths.
"MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982
(h12)" says it is the SAME handle and someone released it; "(h-1)" says it was
never created. That is the difference between a refcount bug and a creation
failure and it is one field. This is the measurement the shared-handle suspicion
needs before anyone acts on it.
2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT.
A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles()
re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD
indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does
NOT release first: -1 means the terminal no longer knows the handle, so there is
nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could
decrement whatever now owns that number. 30s cooldown, because every ServableBars()
consumer reaches it including live inference on every tick. The feature cache is
dropped with it, and the log names before/after depths.
Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars
against a buffer that answers EMPTY_VALUE at every index - then discarding the era
and doing it again - is not a recovery.
3. THE SWEEP NOW HOLDS ON A DEAD HANDLE.
ServableBars() keeps answering `want` (its contract; live inference and online
learning have their own refusal paths and a 0 there reads as "no history at all").
SettledBars() - the training sweep's entry, the one caller that can afford to wait -
returns 0 instead, so Train() holds and reports rather than burning a full-history
pass it is guaranteed to throw away. A recreated handle is cold, so it primes
through the existing settle path on the next call. If the repair fails the member
holds indefinitely and says so every minute, and be39674's barrier liveness escape
releases the rest of the ensemble after 12 minutes - which is the correct
degradation and is exactly what the log shows happening.
4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST.
Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of
Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the
next free row on each member's FIRST PublishStatus() call, so the order was a race -
the members busy sweeping published before the ones sitting idle at the era barrier,
and be39674 sharpened it by (correctly) making a held member stop writing the terse
line. Rows are now keyed to m_ensembleIndex, the registration/construction order,
which is fixed for the life of the chart. Claimed on every publish rather than once,
so it is idempotent and refreshes the tag for a member whose ID was not final when it
first published (the config-tag suffix is appended during InitIndicators, after
EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded
from the model count, so a member that has not published yet leaves no gap and shifts
nobody.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
//--- Cooldown between attempts to rebuild a dead indicator handle (see RepairDeadIndicatorHandles).
//--- Every ServableBars() consumer can reach the repair - the training sweep, live inference on
//--- every tick, online learning - and a terminal genuinely refusing to create the indicator must
//--- not be hammered.
uint m_handleRepairTick ;
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- ERA-BARRIER LIVENESS STATE (see ENSEMBLE_BARRIER_STUCK_MS and BarrierEraHeartbeat()).
//--- Seen/Tick are the era-advance watchdog; Excluded is the latch that takes a non-advancing member
//--- out of the barrier minimum; HoldReportTick rate-limits the "I am waiting, and on whom" line that
//--- a held member now writes to the journal - previously a held member wrote nothing anywhere, which
//--- is why a two-chart outage on 2026-08-17 was visible only through the members that were NOT held.
long m_barrierEraSeen ;
uint m_barrierEraTick ;
bool m_barrierExcluded ;
uint m_barrierHoldReportTick ;
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate
1dda479 clamped the training sweep. It left five other paths asking the indicators
for a depth they cannot serve, and on a live account the quiet ones are worse than
the stall was - a stalled chart is visible, a chart trading on a degraded feature
window is not.
ServableBars(want, context) is now the single gate, and all six go through it:
training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small
positive BarsCalculated is warm-up, which m_coldSweepTick owns)
label prebuild clamp - labels come from price/ADZigZag and would survive a
capped MA, but ResizeBuffers sizes EVERY buffer and a failed
CopyBuffer leaves m_MA EMPTY for the next reader, so this path
could silently re-break the block Train()'s clamp just fixed
live inference HOLD. Below `need` the swing block takes its degraded path and
inference runs on a different feature distribution than the model
was fitted on. This EA sizes real positions off that output, so
no signal beats a mismatched one
online learning HOLD, same reason and worse - this path WRITES to a live trading
model, so a mismatched (features, label) pair is not a wrong arrow,
it is a wrong weight update that compounds every bar
chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest
"Max bars in chart" is also 5000, so this one is genuinely
reachable; uncapped it repaints the window all-Neutral
research export clamp before the emptiness test, so a capped symbol exports the
depth it has rather than writing a CSV with a dead feature block -
an artefact that looks complete and is silently wrong
Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars
(16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so
the failure mode is unreachable rather than merely unlikely.
Not changed: a genuinely SHORT price history still takes the old degraded path at
every site. That is pre-existing behaviour and narrowing it would mute charts that
trade today, so it stays a separate decision rather than a side effect of this fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
//--- One-shot latch for the live-inference hold in RefreshConvergedSignal(), so a held chart says so
//--- once instead of on every tick. Cleared the moment the depth is available again, so a second
//--- outage is reported rather than swallowed.
bool m_inferenceDepthRefusalWarned ;
fix(buffers): revert the MA +1 - it asked for a bar that does not exist and stopped every chart
REGRESSION I INTRODUCED IN 1dda479, live for ~15 minutes.
CSeries::BufferResize -> CheckLoadHistory -> CheckTerminalHistory succeeds only when
Bars() >= size. Train() calls ResizeBuffers with barIndex == Bars(), so sizing the MA
buffer to barIndex + 1 asks for one bar more than the symbol has and fails the WHOLE
ResizeBuffers call. The log named it exactly:
failed to get 50180 bars for USDJPY,PERIOD_H4 (Bars() = 50,179)
failed to get 33983 bars for XAUUSD,PERIOD_H4 (Bars() = 33,982)
StartLabelCachePrebuild() then bailed on the false return and stayed silent, so the
only symptom was Train() reporting "arming the first label-cache prebuild" forever
with labelCacheBars=0 - the panel's "getting ready".
The premise was wrong, not just the arithmetic. The MA block reads GetData(idx) AND
GetData(idx + 1), and at the OLDEST bar that second read is SUPPOSED to fail - there
is no older bar to difference against. Rejecting that one bar is correct behaviour;
buying it cost the entire history.
Two more things, since the same defect had a second instance and no alarm:
- The Ichimoku pair (closeBars and m_Ichimoku, both barIndex + ichiKijun) is the same
bug with a far larger constant, latent only because the feature is off. Both are now
clamped to Bars(). The oldest ichiKijun bars then have no cloud, which that block's
EMPTY_VALUE guard already handles per-bar - the right outcome.
- The prebuild's bare `return` on a false ResizeBuffers now says so once, naming the
depth and Bars(). MQL5's own "failed to get N bars" was in the log the entire time,
from a stack frame nothing connected to the prebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 14:12:47 -04:00
//--- One-shot latch for the label-prebuild block message. A prebuild that cannot prepare its buffers
//--- retries on every scheduled call forever, so this says it once rather than at chart-refresh rate.
bool m_prebuildBlockWarned ;
feat(depth): prime -> settle -> sweep, and name which handle is short
"Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in
1dda479 was wrong. Two other candidate causes are falsified too: the price series
is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new
H4 bar), and the handles have been stable since 13:07 with zero windows for the 17
minutes after, so it is not download-in-progress and not handle churn. The MA period
tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not
indicator cost either.
What IS verified stays verified: CopyBuffer past the calculated depth fails outright
rather than short-reading, so the buffer holds nothing and every index reads
EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag
neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25
of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated:
16k-bar charts train, 34k/50k get zero windows forever.
So the WHY is still open, and this fix does not depend on it. Per the user's protocol:
the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated()
every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever
it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned
depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two
training paths, which snapshotted a value that may still have been climbing; the clamp
remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b).
The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature
scan starves the indicator threads the request just woke, which is how the failure
sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the
0->100% oscillation on the panel.
Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and
stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the
cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
//--- SettledBars() probe state. m_depthSettleStart doubles as the "a wait is in progress" flag (0 =
//--- none), which is why it is cleared on every path that stops waiting.
uint m_depthSettleStart ;
uint m_depthProbeTick ;
int m_depthProbeLast ;
int m_depthProbeStable ;
2026-07-16 00:56:33 -04:00
//--- ground truth for the training labels (see m_swingConfirmationBars' declaration comment) -
//--- CustomIndicators\ADZigZag.mq5, a renamed/rebranded copy of the stock MQL5 ZigZag indicator,
//--- always created (not gated behind an Enable* input like the AD* feature indicators above,
//--- since it isn't an optional input feature - it IS the label) and always run at its own stock
//--- defaults (Depth=12, Deviation=5, Backstep=3) - never touched by AutoTuneIndicators, since
//--- tuning the ground truth itself alongside the model being scored against it would let a trial
//--- "improve" its OOS score by cherry-picking an easier target rather than a better model.
CiCustom m_ADZigZag ;
refactor(ExpertSignalAIBase): extract AutoTune param state into CADIndicatorTuner
CExpertSignalAIBase (4,326 lines, one class) carried 5 struct
definitions, 10 member fields, and 3 methods (Flatten/Unflatten/
PerturbRandom) purely for the AutoTuneIndicators search-space state -
entirely self-contained (never touches Net, Train()'s resumable state
machine, or anything else in the class). Extracted into a new
Expert/ADIndicatorTuner.mqh (CADIndicatorTuner), held as a single
m_indicatorTuner member.
TuneIndicatorsAndTrain() itself - the outer loop that actually
orchestrates Train()/Net/checkpointing around this tuner - turned out
to be exactly as tightly coupled to Train()'s resumable state machine
as Train() itself, so per the same caution already applied to Train()
in this refactor pass, it stays in CExpertSignalAIBase rather than
being pulled into the collaborator; it now calls the tuner's public
Flatten()/Unflatten()/PerturbRandom()/SaveAsBest()/RestoreBest()
instead of manipulating the structs inline.
All internal field-access renames (m_adCumDeltaParams.lookback ->
m_indicatorTuner.adCumDelta.lookback, etc., ~40 sites across the 5
InitAD*() indicator-setup methods) verified against a full grep sweep
- no leftover references to the old field/method names. Compiled
clean (MetaEditor, 0 errors/0 warnings).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 16:22:09 -04:00
//--- currently-active tunable input values for each AD indicator (AutoTuneIndicators search space)
//--- plus their own flatten/unflatten/perturb/best-tracking logic - see CADIndicatorTuner's
//--- declaration comment (Expert\ADIndicatorTuner.mqh) for why this is a separate collaborator
//--- rather than loose fields/methods here: it's entirely self-contained (never touches Net,
//--- Train()'s state machine, or anything else in this class), unlike TuneIndicatorsAndTrain()
//--- below, which orchestrates Train()/Net/checkpointing around it and stays here for exactly that
//--- reason. InpContextMode/InpSessionType/InpSessionCount are session choices, not accuracy
//--- knobs, and are hardcoded to the indicators' own defaults in InitAD*() rather than stored/
//--- tuned in the collaborator.
CADIndicatorTuner m_indicatorTuner ;
2026-07-14 22:36:27 -04:00
bool m_autoTuneIndicators ;
//--- rebuilds only the AD* indicator handles (in place) so ReInit picks up updated param structs
bool ReInitADIndicators ( CIndicators * indicators ) ;
2026-08-13 10:23:11 -04:00
//--- installs a loaded/saved param set into the tuner and rebuilds the handles ONLY when the set
//--- actually differs from what the live indicators already run - see the definition for the
//--- resume-time churn this exists to avoid.
bool AdoptIndicatorParams ( const double & loaded [ ] , CIndicators * indicators ) ;
2026-07-14 22:36:27 -04:00
//--- builds a fresh, untrained topology into Net (assumes Net is currently NULL); factored out of
//--- InitNeuralNetwork() so TuneIndicatorsAndTrain() can rebuild weights per trial without
//--- re-running indicator init (which would re-Add() the base indicators into indicators a 2nd time)
bool BuildFreshTopology ( ) ;
//--- CIndicators collection passed into InitNeuralNetwork(), retained so
//--- TuneIndicatorsAndTrain() can call ReInitADIndicators() between trials
CIndicators * m_indicatorsPtr ;
CNet * Net ;
2026-07-15 21:47:37 -04:00
//--- EMA "shadow" copy of Net, blended a small step (SHADOW_WEIGHT_TAU) toward Net's weights at
//--- the end of every era (see Train()'s era-end block) rather than replaced outright. Live
//--- trading/inference (RefreshLatestSignal()) reads from this, not from Net directly, so any
//--- single era's raw weights - including an Adam overshoot event - can only ever nudge what's
//--- actually deployed by SHADOW_WEIGHT_TAU, not overwrite it wholesale. Bootstrapped as a clone of
//--- Net (via the same Save()/Load() pattern m_simOosNet uses) the first time InitNeuralNetwork()
//--- runs with no prior shadow file, and persisted alongside Net.Save() thereafter. NULL only
//--- during the brief window before that first bootstrap completes - every read site falls back to
//--- Net in that case, never blocks on it.
CNet * m_shadowNet ;
2026-07-24 11:52:19 -04:00
//--- One-shot latch for EnsureShadowNet()'s clone bootstrap. Cloning a second full net can fail on the
//--- tester's CPU-DLL compute fallback (the chart's GPU/DirectML path succeeds); without this the
//--- bootstrap would retry on EVERY bar, re-initialising the compute backend each time - log spam and a
//--- multi-second-per-bar crawl through an inference-only backtest. A failed/skipped bootstrap is
//--- harmless: every read site falls back to Net, and a bootstrapped shadow is identical to Net until
//--- era-end blending diverges it (which never happens in a no-training run). Reset with m_shadowNet.
bool m_shadowBootstrapAttempted ;
//--- Online continual-learning state (see OnlineLearnStep(); tunables at ONLINE_LEARN_* above). Live-
//--- chart-only - a deployed (m_trainingComplete) model keeps learning from newly-confirmed ZigZag
//--- structure as bars mature, waiting the full m_swingConfirmationBars confirmation delay just like
//--- training so a still-provisional (repainting) recent bar is NEVER backpropped. m_enableOnlineLearning
//--- is the input gate. m_onlineLearnedUpToTime is the bar-TIME watermark of the newest bar already
//--- learned from (indexed by time, not now-relative index, so it survives the per-bar index-frame
//--- shift); persisted in the .stats sidecar (WST3). m_onlineRollingAcc is the guardrail EMA (0-100)
//--- of predict-before-learn hits (seeded from the deploy OOS baseline), m_onlineSamples the cumulative
//--- update count (both persisted). m_onlineBarsSincePersist drives periodic saves (in-memory only).
bool m_enableOnlineLearning ;
datetime m_onlineLearnedUpToTime ;
double m_onlineRollingAcc ;
long m_onlineSamples ;
int m_onlineBarsSincePersist ;
//--- Latched log state so the guardrail freeze/resume transition prints once per flip, not per bar.
bool m_onlineBlendFrozen ;
2026-07-14 22:36:27 -04:00
CArrayDouble * TempData ;
double dError ;
double dUndefine ;
double dForecast ;
double dPrevSignal ;
2026-08-01 11:27:28 -04:00
//--- ALTERNATION GATE REMOVED 2026-08-01 with the triple-barrier relabel. m_lastNonNeutralSignal
//--- suppressed any live Buy following another Buy with no Sell between. That was CORRECT under the
//--- exact-pivot target (a ZigZag can only emit a new pivot by FLIPPING type, so a repeat was
//--- provably a false fire) and the premise died with the target: a triple-barrier label is answered
//--- independently at every bar, so ten consecutive Buy setups inside one trend are simply correct.
//--- Do not reinstate it. It also meant a one-sided (`Sell:0%`) model got ONE trade per backtest,
//--- because the awaited opposite signal that reopens the gate never came.
diag: inference-path census, to explain zero-trade backtests
A backtest of the CONVERGED CONV model produced "Final directional result:
0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in
the log could separate the three candidate causes, and each needs a
different fix:
1. RefreshLatestSignal never called (new-bar gate never fires)
2. called, but bailing at one of its two early returns
3. running fine, and the model genuinely answers Neutral every bar
Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown
via StopTraining (which the tester reaches through OnDeinit). Three
increments per bar against a full feedForward - not worth gating.
Ruled out while writing this, so the next session does not re-derive it:
- the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts
at Neutral, so a first Buy would still fire and show up as one non-zero
direction. We saw zero. It IS still a live hazard for a one-sided model -
CONV currently calls Buy:17% Sell:0%, and after the first Buy every later
Buy is suppressed until a Sell that never comes - but it cannot explain
an all-zero run.
- shallow buffers do not hard-fail the feature builder: the swing-context
Donchian loop breaks gracefully when it runs off loaded history. It does
mean converged-path inference computes Donchian/return/SMA features over
a TRUNCATED window versus training, which is a real train/inference skew
worth its own fix, but it degrades features rather than zeroing them.
Both builds 0/0. Diagnostic only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
//--- Inference-path census counters - see PrintInferenceTally() for why these exist. Cheap enough to
//--- keep unconditionally: three increments per bar against a full feedForward.
long m_refreshOk ;
long m_refreshFailFeatures ;
long m_refreshFailShort ;
long m_refreshBuy ;
long m_refreshSell ;
long m_refreshNeutral ;
2026-08-01 11:27:28 -04:00
//--- VOTE-GATE census. RefreshLatestSignal() can answer Buy on hundreds of bars while
//--- LongCondition()/ShortCondition() still return 0 on every one, because those open with a
//--- readiness gate the refresh path never consults:
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- if(!m_trainingComplete && !(m_inferenceOnly && m_modelLoadedFromDisk)) return 0;
2026-08-01 11:27:28 -04:00
//--- In the tester that reduces to "the seeded _optcache.nnw must have LOADED"; when it has not, the
//--- run silently produces direction 0.00 on every bar. Without these two the census reads as "the
//--- model answers Neutral" - false, and it points at a completely different fix. They separate the
//--- model's ANSWER from whether that answer was allowed to become a vote.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
long m_voteGateBlocked ; // directional decisions the readiness gate discarded
long m_voteGatePassed ; // directional decisions that became a real vote
//--- Flag pair as of the first vote attempt, latched so the tally can name WHICH half of the gate
//--- failed rather than just reporting that it did. -1 = no vote was ever attempted.
int m_voteGateCompleteAtFirst ;
int m_voteGateLoadedAtFirst ;
//--- m_gateSnapshot removed with the alternation gate above - it existed only to roll that gate back
//--- when the parent discarded this filter's vote. The AI signal now holds no one-shot vote state, so
//--- BeginVote()/RevokeVote() fall back to the base class's empty implementations.
2026-07-21 00:03:45 -04:00
//--- Non-max suppression (NMS) of directional signals. Consecutive H4 bars near a real turn share
//--- almost their entire feature vector, so a single reversal fires the same class on a whole run of
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- adjacent bars - a visual/emission cluster around one event, not several distinct turns.
2026-08-01 11:27:28 -04:00
//--- DEFAULT OFF since 2026-08-01: suppressing repeats was justified by the exact-pivot target, and
//--- under triple-barrier labels a run of same-direction setups is real signal, so collapsing it
//--- DISCARDS trades rather than de-duplicating them. Kept opt-in for one marker per move.
//--- Keeps only the FIRST (earliest) bar of a same-direction run, dropping same-direction neighbours
//--- within m_signalClusterWindow bars; 0 disables. Per-direction, so a missed opposite signal never
//--- blocks a later reversal (unlike the alternation gate), and causal, so the live signal and the
//--- drawn history declutter identically. Post-processing ON TOP of the model - the raw per-bar
//--- recall/precision/accuracy stats stay un-NMS'd so they keep measuring the network itself.
//--- The cluster extends from the last SEEN same-direction bar, not the last KEPT one: measuring
//--- from the kept bar re-emitted an arrow every window+1 bars inside a long run.
2026-07-21 00:03:45 -04:00
int m_signalClusterWindow ;
//--- Per-bar predicted signed signal for THIS era (index = now-relative bar index; -2 = not scored
2026-07-21 13:18:35 -04:00
//--- this era). When NMS is on, the scan passes (1/2/3) ONLY record into this cache and draw nothing;
//--- PruneDirectionalClusters() then renders the whole declustered set once at era end. That is what
//--- keeps the chart from ever showing the raw mid-era clusters (the passes would otherwise draw every
//--- above-threshold bar as it scanned, and the sweep only cleaned up at era end). Recording (not
//--- drawing) also decouples NMS from pass 2's SHUFFLED order, which no inline cursor could dedup.
//--- Sized to bar count each fresh era.
2026-07-21 00:03:45 -04:00
double m_arrowSignalCache [ ] ;
2026-07-21 12:30:29 -04:00
//--- Live-side NMS state: bar TIME of the last SEEN signal per direction (advances on every same-
//--- direction bar, kept or suppressed, so a contiguous live run collapses to one) plus the cached
//--- accept/suppress decision for that exact bar (keeps repeated same-bar RefreshLatestSignal calls
//--- idempotent - re-evaluating the same bar returns its first decision, not a flipped one). 0 = none.
2026-07-21 00:03:45 -04:00
datetime m_nmsLiveBuyTime ;
datetime m_nmsLiveSellTime ;
2026-07-21 12:30:29 -04:00
bool m_nmsLiveBuyAccept ;
bool m_nmsLiveSellAccept ;
//--- Last KEPT live signal of either direction, for cross-direction resolution: a Buy and a Sell
//--- within m_signalClusterWindow bars are flicker at one turn zone (real opposite pivots are a whole
//--- leg apart), so only the higher-confidence side is kept. Confidence = |signed signal| = winning
//--- softmax probability. Same rule runs in PruneDirectionalClusters() for the historical chart.
datetime m_nmsLiveKeptTime ;
ENUM_SIGNAL m_nmsLiveKeptDir ;
double m_nmsLiveKeptConf ;
2026-07-14 22:36:27 -04:00
datetime dtStudied ;
long m_eraCount ; // cumulative era counter, persisted in the .nnw so restarts don't look like they reset progress
bool m_trainingComplete ; // persisted: true only once Train() converged (objective+stability), not just interrupted
bool bEventStudy ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- This instance's study-event id (STUDY_EVENT_ID_BASE + construction order) and the tick-count
//--- when bEventStudy was last armed - see the STUDY_EVENT_ID_BASE comment for why these exist.
//--- All arming goes through ArmStudyEvent() so the id and the watchdog stamp can never drift apart.
ushort m_studyEventId ;
uint m_studyArmedTick ;
2026-07-14 22:36:27 -04:00
//--- out-of-sample holdout: share (%) of the study period never trained on, used only to
//--- measure genuine forward accuracy so overfitting shows up in the stats, not just live/OOS trading
int m_oosSplitPct ;
double dOosError ; // smoothed OOS mismatch rate (0..100), lower is better
double dOosForecast ; // smoothed OOS accuracy (0..100)
int m_oosSamples ; // count of OOS predictions evaluated this Train() call
2026-07-19 14:50:52 -04:00
//--- per-era raw (pre-softmax) output-neuron stats over pass 3's OOS scan, reset at pass 3
//--- start; surfaced in the era-end log line. The per-bar logit spread (max-min across the 3
//--- outputs) is the collapse fingerprint: a healthy net differentiates bars (avg spread well
//--- above 0), a saturated all-one-class net pins all three outputs to the same value on every
//--- bar (avg spread ~0, mins/maxes flat) - see CLASS_LOGIT_SCALE's comment (AI\Network.mqh).
double m_oosOutMin [ 3 ] ;
double m_oosOutMax [ 3 ] ;
double m_oosOutSpreadSum ;
int m_oosOutCount ;
feat(diagnostics): split a reported "Neutral" into CHOSE vs TIED - they need opposite fixes
ApplyClassificationSoftmax() requires a STRICT majority over both rivals and
sends every tie, 2-way or 3-way, to Neutral. So "OOS recall Neutral:100%" is
two completely different events sharing one label:
CHOSE - the net genuinely ranks Neutral highest. A class-prior/label problem.
TIED - the top two are EXACTLY equal, so the net expressed no preference and
the tie-break reported Neutral. A SATURATION problem: the head is
SIGMOID, and a saturated sigmoid returns exactly 0.0f or 1.0f in the
DLL's float32, so two classes pinned to the same rail compare equal
and the bar is silently discarded.
Nothing in the logs could tell them apart, and the fixes point opposite ways.
Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99
- fully saturated - and broke out at era 27 as the spread fell to 0.75. That is
consistent with EITHER story. The user reports the Neutral phase on most runs,
so it is worth four longs to stop guessing.
Four per-era counters on the pass 3 OOS walk, reported as:
| Neutral CHOSE 12.4% / TIED 38.1% (of which B=S 1204) | rail 61.2%
m_oosNeutralStrict - Neutral strictly highest
m_oosNeutralTie - no strict winner; the tie-break produced Neutral
m_oosTieBuySell - the costly subset: Buy and Sell tied AT the top, i.e. a
DIRECTIONAL reading thrown away by float equality
m_oosRailBars - any raw output sitting on a sigmoid asymptote, the
saturation that makes exact ties possible at all
Read on the RAW logits, before ApplyClassificationSoftmax() overwrites TempData
in place. Legitimate because softmax is strictly monotone: it cannot change the
ordering and cannot break a tie either, so the raw reading and the decision
always agree. Placed alongside the existing min/max/spread capture so all the
output diagnostics describe the same values.
Measurement only - no decision path reads these.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:06:54 -04:00
//--- WHY "Neutral" WON, per OOS bar. ApplyClassificationSoftmax() requires a STRICT majority over
//--- both rivals and sends every tie - 2-way or 3-way - to Neutral. So a reported Neutral is two
//--- completely different events sharing one label, and they need OPPOSITE fixes:
//--- strict - the net really does rank Neutral highest. A class-prior / label problem.
//--- tie - the top two are EXACTLY equal, so the net expressed no preference and the
//--- tie-break reported it as Neutral. A saturation problem: the head is SIGMOID, and
//--- a saturated sigmoid returns exactly 0.0f or 1.0f in the DLL's float32, so two
//--- classes pinned to the same rail compare equal and the bar is silently discarded.
//--- Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99 (fully
//--- saturated) and broke out at era 27 as the spread fell to 0.75 - consistent with EITHER story,
//--- which is exactly why counting them apart is worth four longs. m_oosTieBuySell is the costly
//--- subset: a Buy-vs-Sell tie at the top is a DIRECTIONAL call destroyed by float equality.
long m_oosNeutralStrict ;
long m_oosNeutralTie ;
long m_oosTieBuySell ;
long m_oosRailBars ; // any raw output pinned to a sigmoid rail (<=0+eps or >=1-eps)
2026-07-14 22:36:27 -04:00
//--- per-era counts of the network's own classification of each bar it fed forward (IS+OOS),
2026-07-17 21:28:59 -04:00
//--- reset at the start of every era; surfaced in the status label text so class imbalance
2026-07-14 22:36:27 -04:00
//--- (e.g. the network collapsing to all-Neutral) is visible while training runs
int m_countBuySignals ;
int m_countSellSignals ;
int m_countNeutralSignals ;
//--- per-era counts of the *true* label of every bar fed forward (IS+OOS), reset alongside the
2026-07-17 21:28:59 -04:00
//--- predicted counts above. Used both to display the actual class distribution in the status
//--- label text and, more importantly, to drive IS-sample oversampling in Train() (see backProp calls
2026-07-14 22:36:27 -04:00
//--- below) - rarer classes get replayed more times so gradient descent doesn't just learn to
//--- always call the majority (usually Neutral) class.
int m_trueBuyCount ;
int m_trueSellCount ;
int m_trueNeutralCount ;
//--- snapshot of the class totals above, taken at the end of the PREVIOUS era (see Train()'s
//--- era-reset block) and held fixed for the whole of the current era. The oversampling ratio
//--- below is computed from these frozen counts rather than the live, still-accumulating
//--- m_true*Count values - using the live counts made the ratio order-dependent within a single
//--- era (chronological, oldest-to-newest bar processing means whichever class happens to be
//--- numerically behind at any given moment gets amplified up to 5x, even if that's just a
//--- transient artifact of which regime the era's early bars came from, not the class's true
//--- overall rarity) - a real source of run-to-run oscillation in the predicted class mix. All
//--- zero on era 0, when oversampling simply falls back to 1x (see the reps calc in Train()).
int m_prevEraTrueBuyCount ;
int m_prevEraTrueSellCount ;
int m_prevEraTrueNeutralCount ;
//--- per-era OOS confusion counts, reset each era; used to compute per-class OOS recall (hits/total)
2026-07-17 21:28:59 -04:00
//--- for the status label text and, more importantly, as an additional convergence gate alongside the
2026-07-14 22:36:27 -04:00
//--- blended dOosForecast accuracy - a model that "wins" only by calling everything Neutral will
//--- have high dOosForecast but near-zero Buy/Sell recall, and should NOT be allowed to converge.
int m_oosBuyHits , m_oosBuyTotal ;
int m_oosSellHits , m_oosSellTotal ;
int m_oosNeutralHits , m_oosNeutralTotal ;
2026-07-15 21:47:09 -04:00
//--- same per-era OOS confusion counts as above but keyed by PREDICTED class instead of true class,
//--- i.e. per-class precision (of the bars this era where the model called Sell, how many actually
//--- were Sell?) rather than recall (of the bars that actually were Sell, how many did it catch?).
//--- Recall alone can't distinguish "the model over-fires Sell and happens to also catch enough real
//--- Buys/Neutrals to clear their recall floors" from "the model is genuinely well-calibrated" - a
//--- skewed Predicted-this-era count (see m_countSellSignals) with recall still passing the gate is
//--- exactly that failure mode, and precision is what would expose it.
int m_oosBuyPredicted , m_oosBuyPredictedHits ;
int m_oosSellPredicted , m_oosSellPredictedHits ;
int m_oosNeutralPredicted , m_oosNeutralPredictedHits ;
fix: the deploy gate was benchmarking a win rate against a label frequency
The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a
driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade
is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test.
That invariant needs reward >= risk, and the measured geometry no longer
satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but
both-won bars were stripped out of Buy and Sell so the label base rate read
37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing
models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma
against 37.5% and loses money on every single trade. Live since 217b9bc.
Root cause is that label agreement stopped being the same question as trade
profitability. Buy implies winLong, but the converse fails on every both-won
bar, and the label can only name one of two directions that both pay.
So stop asking the model whether it matched a label and start asking whether
its trade paid:
- cache winLong/winShort per bar beside the label, under the same validity
flag; published from the barrier walk before the collapse to 3 classes
- dirPrecPct now counts wins on the side actually called
- chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook
m/(m+k) would credit SP500's drift to the model
- the NMS "what would I have made" pair, the live-fired precision, and the
IS/OOS cumulative win rates all move to the same test. IS and OOS are read
side by side as the overfitting signal, so measuring one in wins and the
other in agreement would put a fixed gap between them that has nothing to do
with generalization
- the confidence threshold is FITTED on wins too, so the operating point
maximises what the gate grades
- per-class label-agreement precision is still computed and logged; it is the
right diagnostic for class separation, just not for a deploy decision
- era line renamed dir-precision -> win-rate, chance -> chance=break-even
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- WOULD THE TRADE HAVE PAID. Same predicted-class keying as the *Hits pair above, but scored
//--- against m_winLongCache/m_winShortCache instead of against label agreement - "the Buy calls whose
//--- long actually reached target before stop", not "the Buy calls that matched a 3-class label".
//---
//--- These two questions coincided only while reward >= risk forced Buy and Sell to partition the
//--- resolved bars. Removing the minimum-reward:risk raise (2026-08-09) ended that: the measured
//--- geometry puts the target NEARER than the stop, so both directions can win on the same bar, the
//--- label can only name one of them, and the other call was being scored as an error despite its
//--- trade paying in full. The gap is not academic - it moved the zero-skill reference to 37.5% while
//--- break-even sat at 67.3%, i.e. the gate would happily ship a model that loses on every trade.
//---
//--- With this pair, dirPrecPct IS the win rate, and the invariant the gate rests on is restored
//--- exactly: an always-Buy model wins P(winLong) = m/(m+k) of the time, which is also the break-even
//--- rate for a k:m trade, so "beats chance" and "is profitable" are once again the same test.
int m_oosBuyPredictedWins ;
int m_oosSellPredictedWins ;
//--- Zero-skill denominators, MEASURED rather than assumed from m/(m+k): how many scored bars a long
//--- (resp. a short) would have won on, regardless of what the model called or what the label says.
//--- max() of the two is what an always-call-one-direction model scores, which is the reference the
//--- deploy gate needs. Measured, because the theoretical identity holds for a driftless walk and real
//--- instruments drift - SP500 makes always-long genuinely better than a coin, and a gate that used
//--- the textbook value would credit that drift to the model.
int m_oosWinLongTotal ;
int m_oosWinShortTotal ;
2026-07-18 14:56:41 -04:00
//--- Confidence calibration: the classification head's forward pass still uses SIGMOID per-neuron
//--- (see BuildFreshTopology()'s SIGMOID comment - bounded activation, avoids logit runaway), but
//--- the BACKWARD pass (CNet::backProp/backPropOCL in AI\Network.mqh) now trains all 3 neurons
//--- jointly against a true softmax+categorical-cross-entropy gradient (softmax_i - target_i), not
//--- 3 independent binary-cross-entropy targets. ApplyClassificationSoftmax() at READ time
//--- reproduces the same normalization the loss was actually trained against, so the value it
//--- returns is now a genuinely loss-consistent probability (still not literature-perfect
//--- calibration - no temperature scaling/Platt scaling has been applied - but no longer
//--- structurally decoupled from what was optimized). SignedAIConfidence()/g_AISignedConfidence
2026-07-18 15:53:04 -04:00
//--- exposes it, and Money\MoneyIntelligent.mqh:AdjustRiskAmount() scales position-sizing risk%
2026-07-18 14:56:41 -04:00
//--- directly off it.
2026-07-17 23:21:12 -04:00
//--- m_oosConfidenceSum accumulates MathAbs(dPrevSignal) (the claimed confidence) over every
//--- classified OOS bar this era; compared against this era's actual OOS accuracy
//--- ((m_oosBuyHits+m_oosSellHits+m_oosNeutralHits)/m_oosSamples) at era-end to derive
//--- m_confidenceCalScale - a single empirical multiplier ("the model claims 80% on average but is
//--- only right 60% of the time -> scale reported confidence by 0.75") applied to the MAGNITUDE
//--- only, never the sign/class decision, in SignedAIConfidence(). EMA-blended across eras (same
//--- smoothing factor as dOosForecast) so one noisy era can't swing it, and clamped to [0.3, 1.5]
//--- so a thin/degenerate OOS window can't drive it to something absurd. Starts at 1.0 (no
//--- correction) until the first era with OOS samples computes a real value.
double m_oosConfidenceSum ;
double m_confidenceCalScale ;
2026-07-14 22:36:27 -04:00
//--- minimum acceptable OOS recall (%) for the Buy and Sell classes individually before Train() is
//--- allowed to declare convergence; a class with zero OOS samples this era doesn't block (avoids a
//--- deadlock when a given era's OOS window happens to contain no examples of that class)
int m_minDirectionalRecallPct ;
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits
Three changes, all from the same principle: measure what is there before aiming
at it, and never certify a number you do not trade.
1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE.
MinRecall=40 was a constant doing a statistical job. Its reference point is the
33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the
constant was accidentally calibrated for exactly one sample size: on USDJPY CONV
(n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance
+ 0.9 SE. One chart was being held to a bar twice as strict as the other, for no
reason anyone chose.
CollapseRecallFloorPct() computes it per class from that class's own effective
sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use
applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at
n_eff 42.
BELOW chance, deliberately, and this is the substantive change rather than the
arithmetic. This gate's only job is refusing to call a COLLAPSED model converged.
It is not a quality bar; the deploy gate is the quality bar and it is already
rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the
cross-instrument pooled certificate). A convergence gate that ALSO demands
provably-above-chance recall on all three classes double-counts that job, and it
has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07,
and the 40 that replaced it made Neutral structurally unreachable once first-touch
resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make
a funded account safer, it stops the run converging at all.
Testing significantly BELOW chance instead catches what a fixed 40 was actually
catching - a model that has stopped emitting a class - and cannot become
unreachable by construction. It also fixes the direction the old constant scaled:
it now widens on a thin OOS window, where low recall genuinely cannot be told from
noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30)
is still correctly blocked on Sell.
This also resolves a standing contradiction the code half-admitted at the
isBetterEra comment: selection ranks on coverage-weighted PRECISION while
convergence gated on RECALL, so a sparse high-precision abstainer - precisely the
model that could clear the deploy bar - was blocked by the floor.
The era line now PRINTS the derived floor. Anyone comparing these recalls against a
remembered "40" is reading the wrong bar.
2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS.
The DEPLOY BAR line states the bar. It never said what reaching it would take, and
that is the actionable direction. ReportDetectability() inverts the same identity -
the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs
n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and
prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share
of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE.
Every term is a property of the CONFIGURATION - geometry via break-even, horizon via
mean label lifespan, window via oosCutoff - so no amount of training moves any of
them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the
same reason: that is the first moment the bar grid, the measured geometry and the
lifespan are real numbers rather than defaults. It gates nothing.
This is the EdgeFinder discipline applied to our own gate: establish what the market
and the measurement design have to offer, then point the net at it - rather than
spending a thousand eras chasing something this OOS window could never certify.
3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified).
Every ensemble member ran
g_LiveAISignedConfidence = SignedAIConfidence();
unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit
route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a
four-model chart an LSTM entry could be closed, and its stop moved, on the
Perceptron's opinion alone, decided by scheduling order. Not the vote, not a
weighted blend.
Now the mean across registered members, matching how the ensemble actually trades:
the open decision is the weighted-average vote, and an abstaining member contributes
0 and dilutes exactly as it does there. Members still training read 0, so a
half-trained ensemble reads WEAKER rather than louder - the safe direction for an
exit trigger. Deployed and paused members are included, which is the opposite of the
era barrier's exemption rule and correct for the opposite reason: that one asks who
must be waited for, this asks who has an opinion.
Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled
(101, unreachable on both scales it drives) and TrailingStrategy is off, so live
exits are SL/TP only and the certified hold-to-barrier win rate is what actually
gets traded. Fixed now precisely because the plan is to enable vote exits once the
models are accurate, at which point a scheduling-order exit would be both harmful
and very hard to see.
STILL OPEN, and needs a decision before vote exits go on: the member gate and the
ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the
certified number stop describing the traded one. Warrior_EA.mq5 currently argues
barrier models may keep vote exits because "their label IS the vote's own horizon" -
that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the
target-before-stop outcome the gate measured. Either grade the OOS call on the real
exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set
HoldToBarrier for ensemble members so the policy cannot drift from the certificate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
//--- THE OPERATIVE per-class floor, derived from the class's own effective sample instead of taken
//--- from the input above (which survives only as the fallback when the sample is too thin to give
//--- an SE). `classTrueCount` is that class's RAW true-label count this era; the overlap deflation
//--- is applied inside via EffectiveSampleSize().
//---
//--- WHY IT SITS BELOW CHANCE, which looks wrong at first and is the whole point. This floor's job is
//--- to refuse to call a COLLAPSED model converged - one that has stopped emitting a class at all -
//--- and nothing more. It is not a quality bar: the deploy gate is the quality bar, and it is already
//--- rigorous (chance + EDGE_MIN_SIGMAS x SE on the overlap-deflated sample, Sidak over the candidate
//--- eras, then the cross-instrument pooled certificate). A convergence gate that ALSO demands
//--- provably-above-chance recall on all three classes double-counts that job and has failed twice
//--- here by being unreachable: MinRecall=60 blocked every SP500 H1 run in 2026-07, and the 40 that
//--- replaced it made Neutral structurally impossible once first-touch resolution cut Neutral to a
//--- 0.65% residue. A floor nothing can reach does not make the product safer, it stops it converging.
//---
//--- So the test is one-sided in the other direction: recall significantly BELOW chance is evidence
//--- the model is actively AVOIDING that class, which is exactly what collapse looks like and is the
//--- only thing a fixed 40 was ever reliably catching. Being derived, it also scales the right way -
//--- it widens on a thin OOS window (where low recall genuinely cannot be distinguished from noise)
//--- and tightens on a rich one, where a fixed constant is either too strict or too soft depending
//--- only on which chart it happens to be running.
//---
//--- Measured on 2026-08-17: n_eff 195 (USDJPY CONV) -> floor 26.5%, and PAI's Sell recall of 18%
//--- is correctly blocked as a directional bias; n_eff 42 (SP500) -> floor 18.8%, correctly more
//--- tolerant. The old constant 40 was, by coincidence, chance + 2 SE at n_eff 195 and chance + 0.9
//--- SE at n_eff 42 - i.e. it was accidentally calibrated for one chart and wrong on the other.
double CollapseRecallFloorPct ( int classTrueCount ) ;
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator
Option (a) from the exit-policy question: the certified number must be the traded
number. Plus the modularity correction the user called for on 778b6c0.
1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER.
778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a
member average its siblings through g_warriorEnsemble. That trades a scheduling bug
for a coupling bug, and it is the wrong shape for this EA: every signal runs in its
own instance, minds its own state, and VOTES to the orchestrator, which is the only
thing allowed to combine opinions.
Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split
is enforced by shape rather than by convention:
- PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's;
- AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence.
CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also
republishes the aggregate into g_LiveAISignedConfidence, because the intelligent
trailing reads that global directly and must act on the same number the exit route
does rather than on a leftover from whichever member ticked last. A solo AI signal
owns slot 0, so the non-ensemble path is unchanged.
2. THE GATE NOW REPLAYS THE REAL EXIT RULE.
SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread
convention as ComputeLabelForBar - deliberately by copy, so a disagreement between
the two can only be a policy effect and never a discrepancy between two pieces of our
own arithmetic - and terminates at the FIRST of stop / target / vote reversal /
horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know
which came first, and the barrier is what the broker executes automatically, so
checking the vote first would credit the exit policy with escapes a real stop would
have taken out of its hands.
It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is
decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks
oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only
once m_oosDecisionSeries is complete over the whole OOS window can the replay run.
In ensemble mode that series carries the member's adjusted decision and the live exit
reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry
really can be closed by the ensemble turning against it.
3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING.
A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So
the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs
break-even" stops being a meaningful test - there is no fixed break-even for a
variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the
replay reports expectancy in R with its SE taken from the R distribution (overlap-
deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a
binomial.
This is the same class of error as win-based scoring in 2026-08-09: measuring a
variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits
are still off, is much cheaper than discovering it after they go on.
4. WHY THIS IS SAFE TO SHIP TODAY.
Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches
the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into
0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and
the simulation is arithmetically the same trade the deploy gate already certifies -
they cannot drift. The report says so explicitly, and prints ONCE per run in that
state; when vote exits are on it prints every era, because then the divergence is the
thing to watch. Nothing about today's numbers moves.
The gate switchover is wired but dormant by construction: it becomes exit-aware the
moment the input is enabled, which is exactly what "the certified number is the traded
number" has to mean.
KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The
rule-based averaged-vote close (m_threshold_close) depends on every other filter's
live vote, which pass 3 does not reproduce, so a position the classic filters would
have closed is held to its barrier here. The replay therefore holds LONGER than live
and overstates barrier-reached outcomes. Faithful only while the AI is the dominant
vote - which is the configuration this is being built for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
//--- THE EXIT POLICY ACTUALLY IN FORCE, pushed in from the same inputs the live path reads
//--- (Min_Vote_Close / HoldToBarrier). 0 = no vote-driven exit, which is what ships today.
//--- The gate has to know this because it certifies a WIN RATE measured on hold-to-resolution
//--- outcomes: entry, then the barrier decides. Every vote-driven exit closes EARLIER than that, so
//--- with them enabled the certified number stops describing the traded one. Warrior_EA.mq5 used to
//--- argue barrier models could keep vote exits because "their label IS the vote's own horizon" -
//--- it is not: a vote flip at bar 5 of a 64-bar horizon is not the target-before-stop outcome.
double m_exitVoteThreshold ;
bool m_exitHoldToBarrier ;
//--- The model's ADJUSTED signed decision per OOS bar, captured during pass 3 - the same value that
//--- votes live, 0 where it abstains. Needed because a vote-flip exit at bar t depends on what the
//--- model SAYS at bar t, which is a future bar relative to the entry being graded, so the outcome
//--- cannot be precomputed per bar the way the barrier win caches are. In ensemble mode this holds
//--- the COMBINED vote, because the live exit reads the ensemble average (see
//--- EnsembleLiveSignedConfidence) and an LSTM entry really can be closed by the ensemble turning
//--- against it - the coupling the user identified on 2026-08-17.
double m_oosDecisionSeries [ ] ;
//--- The trade the EA would ACTUALLY have taken from `entryIdx`, under the policy above: first of
//--- stop / target / vote reversal / horizon. Returns the outcome in R (risk multiples, so -1 is a
//--- full stop-out and tpMult/slMult a full target) rather than as a bool, because a vote exit lands
//--- somewhere in between and a boolean cannot carry that. false = unresolvable bar.
bool SimulateTradeOutcome ( int entryIdx , bool isLong , double & rMultiple ,
int & lifespanBars , bool & endedOnVote ) ;
//--- Per-era accumulators for the simulated-exit report (see ReportExitPolicyDivergence).
double m_simRSum ;
double m_simRSumSq ;
int m_simTrades ;
int m_simVoteExits ;
int m_simBarrierWins ;
bool m_exitReplayReported ;
void ReportExitPolicyDivergence ( void ) ;
//--- Replays every directional call of this era under the live exit policy. Runs AFTER pass 3, never
//--- inside it: a vote-flip exit for a trade at bar r depends on the decisions at bars r-1, r-2, ...
//--- which pass 3 has not produced yet when it grades r (it walks oldest-to-newest).
void SimulateExitPolicyOutcomes ( void ) ;
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks
Two things, one of which was a compile error.
1. ExitPolicy() was declared in the protected block but is pushed in from
Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters.
2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured
from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot
begin until whatever is in flight returns - so a scan still running after
_StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge
never gets its turn.
New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress.
Deliberately NOT m_trainingStopRequested: that latches, and a latched flag
would permanently disable scans that must run again on the next Start.
Guarded, longest first:
- TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings
on the way out (best[] is mutated in place; the tuner otherwise keeps the
last trial's parameters, which nothing chose).
- ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so
m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar
read the last candidate's multiples as the configured geometry).
- ReportFeatureLabelInformation / ReportExcursionInformation / lag profile -
nulls ABANDON rather than truncate: fewer draws is not a smaller null, it
is a wrong one, and p shifts toward significance. m_dirEvidence staying
false is the safe direction.
- SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line
is dropped instead of latching a partial expectancy as the run's only report.
- ReportGeometryExpectancyScan - per ladder rung.
- HttpGet - one choke point for up to a dozen blocking WebRequests per
first-pass Update(). An in-flight request cannot be cancelled; refusing to
start another is the whole remedy.
- PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain -
entry points, so a queued event cannot open an era during teardown.
TuneIndicatorsAndTrain's guard is the first statement, ahead of the
m_tuneFilterDone / g_ensembleChartTuneDone latches.
- OnTick / OnTimer / OnChartEvent.
Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3
yield on a 120 ms budget); the warm-up scans did not, and they are the longest
uninterruptible stretches the EA has.
StopTraining() is unchanged: the operator's Stop still finalises synchronously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
//--- ExitPolicy() itself is PUBLIC (with the other Warrior_EA.mq5 setters) - it is pushed in from the
//--- EA, not called from inside the class. These two are what it writes.
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits
Three changes, all from the same principle: measure what is there before aiming
at it, and never certify a number you do not trade.
1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE.
MinRecall=40 was a constant doing a statistical job. Its reference point is the
33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the
constant was accidentally calibrated for exactly one sample size: on USDJPY CONV
(n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance
+ 0.9 SE. One chart was being held to a bar twice as strict as the other, for no
reason anyone chose.
CollapseRecallFloorPct() computes it per class from that class's own effective
sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use
applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at
n_eff 42.
BELOW chance, deliberately, and this is the substantive change rather than the
arithmetic. This gate's only job is refusing to call a COLLAPSED model converged.
It is not a quality bar; the deploy gate is the quality bar and it is already
rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the
cross-instrument pooled certificate). A convergence gate that ALSO demands
provably-above-chance recall on all three classes double-counts that job, and it
has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07,
and the 40 that replaced it made Neutral structurally unreachable once first-touch
resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make
a funded account safer, it stops the run converging at all.
Testing significantly BELOW chance instead catches what a fixed 40 was actually
catching - a model that has stopped emitting a class - and cannot become
unreachable by construction. It also fixes the direction the old constant scaled:
it now widens on a thin OOS window, where low recall genuinely cannot be told from
noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30)
is still correctly blocked on Sell.
This also resolves a standing contradiction the code half-admitted at the
isBetterEra comment: selection ranks on coverage-weighted PRECISION while
convergence gated on RECALL, so a sparse high-precision abstainer - precisely the
model that could clear the deploy bar - was blocked by the floor.
The era line now PRINTS the derived floor. Anyone comparing these recalls against a
remembered "40" is reading the wrong bar.
2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS.
The DEPLOY BAR line states the bar. It never said what reaching it would take, and
that is the actionable direction. ReportDetectability() inverts the same identity -
the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs
n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and
prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share
of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE.
Every term is a property of the CONFIGURATION - geometry via break-even, horizon via
mean label lifespan, window via oosCutoff - so no amount of training moves any of
them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the
same reason: that is the first moment the bar grid, the measured geometry and the
lifespan are real numbers rather than defaults. It gates nothing.
This is the EdgeFinder discipline applied to our own gate: establish what the market
and the measurement design have to offer, then point the net at it - rather than
spending a thousand eras chasing something this OOS window could never certify.
3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified).
Every ensemble member ran
g_LiveAISignedConfidence = SignedAIConfidence();
unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit
route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a
four-model chart an LSTM entry could be closed, and its stop moved, on the
Perceptron's opinion alone, decided by scheduling order. Not the vote, not a
weighted blend.
Now the mean across registered members, matching how the ensemble actually trades:
the open decision is the weighted-average vote, and an abstaining member contributes
0 and dilutes exactly as it does there. Members still training read 0, so a
half-trained ensemble reads WEAKER rather than louder - the safe direction for an
exit trigger. Deployed and paused members are included, which is the opposite of the
era barrier's exemption rule and correct for the opposite reason: that one asks who
must be waited for, this asks who has an opinion.
Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled
(101, unreachable on both scales it drives) and TrailingStrategy is off, so live
exits are SL/TP only and the certified hold-to-barrier win rate is what actually
gets traded. Fixed now precisely because the plan is to enable vote exits once the
models are accurate, at which point a scheduling-order exit would be both harmful
and very hard to see.
STILL OPEN, and needs a decision before vote exits go on: the member gate and the
ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the
certified number stop describing the traded one. Warrior_EA.mq5 currently argues
barrier models may keep vote exits because "their label IS the vote's own horizon" -
that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the
target-before-stop outcome the gate measured. Either grade the OOS call on the real
exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set
HoldToBarrier for ensemble members so the policy cannot drift from the certificate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
//--- Mean of the three derived per-class floors, held only so the era log line can print the bar the
//--- recalls beside it were actually judged against.
double m_lastRecallFloorPct ;
fix(geometry): the ensemble was training on TWO DIFFERENT TARGETS - propagate the adopted barrier
MEASURED 2026-08-17 19:06 on USDJPY, in the fresh run:
19:06:38 LSTM adopting barrier geometry 2:10 ... geometry authority
19:06:40 LSTM triple-barrier labels - stop 2.00 target 10.00, horizon 256
19:06:44 PAI / CONV / HYB break-even 33.3%, mean label lifespan 19.2 bars
19:06:45 LSTM break-even 16.7%, mean label lifespan 81.4 bars
One chart, four members, two targets. A "Buy" from LSTM meant "10 ATR before a
2 ATR stop within 256 bars"; a "Buy" from PAI meant "3.21 before 1.61 within 64".
The orchestrator averages those votes and the joint gate certifies the average as
though they answered one question. And g_DerivedSlAtrMult - which places the LIVE
order - is a single global, so the stop actually sent was whichever member wrote
last: the same last-writer-wins class of bug as the live-exit confidence.
CAUSE, and it is mine. The geometry scan sits at the end of the MI chain, and
that chain runs ONCE PER CHART (g_ensembleChartMiReportDone) - whichever member
reaches it first measures and the rest skip. Harmless while the scan only
PRINTED; 62a719f made it authoritative and turned a skipped report into a
skipped DECISION. The indicator tuner already had this doctrine
(g_ensembleChartTuneSettings); the geometry had no equivalent.
- g_ensembleChartGeomAdopted/Sl/Tp/SlMode/TpMode: the donor publishes its
pairing, the siblings adopt it in the MI-skip branch. Ordering is safe by
construction - MQL5 is single-threaded per chart and the donor sets
g_ensembleChartMiReportDone only after the chain (and so the adoption)
returns, so any member taking the skip branch does so strictly afterwards.
- ApplyAdoptedGeometry(): the eleven side effects an adopted pairing must carry
- derived pair, legacy mode ints, g_Derived* live globals, .cfg rewrite, label
cache invalidation, horizon unlatch - in ONE function, because there are now
two callers and duplicating them is how the two paths drift.
- Guarded on era 0 for the donor's own reason: relabelling a partly trained net
moves the target out from under weights already fitted to the old one.
STILL OPEN: dead MA handles were not eliminated by cb30360. They now appear at a
different site (SP500 19:06:35, during "label prebuild", and on PAI - the member
that RAN the sweep), so there is a second handle-churn path I have not found.
Recovery works and the sharing diagnosis stands; the trigger is not only the
tuner's adopt branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 19:35:45 -04:00
//--- Installs an adopted barrier geometry and every side effect that must travel with it: the derived
//--- pair (the one authority), the legacy mode ints, the live-order globals, the .cfg rewrite, the
//--- label cache and the horizon latch. ONE function because there are two callers - the member whose
//--- scan chose it, and every other member on the chart, which learns it second-hand (the MI chain and
//--- therefore the scan run once per chart, so the siblings never measure it themselves). Duplicating
//--- eleven side effects across those two paths is exactly how a chart ends up with members training
//--- on different targets - see g_ensembleChartGeomAdopted.
void ApplyAdoptedGeometry ( double sl , double tp , int slMode , int tpMode ) ;
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits
Three changes, all from the same principle: measure what is there before aiming
at it, and never certify a number you do not trade.
1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE.
MinRecall=40 was a constant doing a statistical job. Its reference point is the
33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the
constant was accidentally calibrated for exactly one sample size: on USDJPY CONV
(n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance
+ 0.9 SE. One chart was being held to a bar twice as strict as the other, for no
reason anyone chose.
CollapseRecallFloorPct() computes it per class from that class's own effective
sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use
applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at
n_eff 42.
BELOW chance, deliberately, and this is the substantive change rather than the
arithmetic. This gate's only job is refusing to call a COLLAPSED model converged.
It is not a quality bar; the deploy gate is the quality bar and it is already
rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the
cross-instrument pooled certificate). A convergence gate that ALSO demands
provably-above-chance recall on all three classes double-counts that job, and it
has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07,
and the 40 that replaced it made Neutral structurally unreachable once first-touch
resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make
a funded account safer, it stops the run converging at all.
Testing significantly BELOW chance instead catches what a fixed 40 was actually
catching - a model that has stopped emitting a class - and cannot become
unreachable by construction. It also fixes the direction the old constant scaled:
it now widens on a thin OOS window, where low recall genuinely cannot be told from
noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30)
is still correctly blocked on Sell.
This also resolves a standing contradiction the code half-admitted at the
isBetterEra comment: selection ranks on coverage-weighted PRECISION while
convergence gated on RECALL, so a sparse high-precision abstainer - precisely the
model that could clear the deploy bar - was blocked by the floor.
The era line now PRINTS the derived floor. Anyone comparing these recalls against a
remembered "40" is reading the wrong bar.
2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS.
The DEPLOY BAR line states the bar. It never said what reaching it would take, and
that is the actionable direction. ReportDetectability() inverts the same identity -
the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs
n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and
prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share
of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE.
Every term is a property of the CONFIGURATION - geometry via break-even, horizon via
mean label lifespan, window via oosCutoff - so no amount of training moves any of
them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the
same reason: that is the first moment the bar grid, the measured geometry and the
lifespan are real numbers rather than defaults. It gates nothing.
This is the EdgeFinder discipline applied to our own gate: establish what the market
and the measurement design have to offer, then point the net at it - rather than
spending a thousand eras chasing something this OOS window could never certify.
3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified).
Every ensemble member ran
g_LiveAISignedConfidence = SignedAIConfidence();
unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit
route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a
four-model chart an LSTM entry could be closed, and its stop moved, on the
Perceptron's opinion alone, decided by scheduling order. Not the vote, not a
weighted blend.
Now the mean across registered members, matching how the ensemble actually trades:
the open decision is the weighted-average vote, and an abstaining member contributes
0 and dilutes exactly as it does there. Members still training read 0, so a
half-trained ensemble reads WEAKER rather than louder - the safe direction for an
exit trigger. Deployed and paused members are included, which is the opposite of the
era barrier's exemption rule and correct for the opposite reason: that one asks who
must be waited for, this asks who has an opinion.
Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled
(101, unreachable on both scales it drives) and TrailingStrategy is off, so live
exits are SL/TP only and the certified hold-to-barrier win rate is what actually
gets traded. Fixed now precisely because the plan is to enable vote exits once the
models are accurate, at which point a scheduling-order exit would be both harmful
and very hard to see.
STILL OPEN, and needs a decision before vote exits go on: the member gate and the
ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the
certified number stop describing the traded one. Warrior_EA.mq5 currently argues
barrier models may keep vote exits because "their label IS the vote's own horizon" -
that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the
target-before-stop outcome the gate measured. Either grade the OOS call on the real
exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set
HoldToBarrier for ensemble members so the policy cannot drift from the certificate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
//--- ONE-SHOT DETECTABILITY REPORT: how many calls this configuration must fire before the deploy
//--- gate could certify an edge of a given size AT ALL, and what share of the OOS window that is.
//--- The inverse of the DEPLOY BAR line, and the actionable direction of it - "chance + 2 SE = 42%"
//--- says the bar, this says what it would take to reach it. Purely a report; it gates nothing.
void ReportDetectability ( int oosBars ) ;
bool m_detectabilityReported ;
2026-07-26 18:33:12 -04:00
//--- There is deliberately NO minimum-confidence input here any more, and no member holding one.
//--- Confidence is expressed through the VOTE WEIGHT (ConfidenceTier() -> PatternWeightForTier()),
//--- not through a separate entry floor, so a weak call is not blocked at the AI's own boundary - it
//--- votes weakly and is then filtered by the same Min vote to open threshold that filters a weak
//--- classic vote. That is what makes ONE input genuinely govern both engines.
//--- The floor it replaced was a scale mismatch: Min_Vote_Open fed it directly as a 0..1 probability
//--- while ALSO being the 0..100 averaged-vote threshold, and a 3-class argmax winner is
//--- arithmetically >= 1/3, so every setting from 0 to 33 gated precisely nothing while every setting
//--- above that silently re-quartiled the confidence tiers as a side effect. See ConfidenceTier().
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- Stops the per-era EMA update of the measured class priors after the first real measurement, so
//--- the tau*log(prior) offsets stay pinned to the distribution the run started from.
2026-07-28 17:42:12 -04:00
bool m_freezePriorCalibration ;
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- THE class-imbalance correction: tau in Menon et al. 2021's logit adjustment. tau*log(prior_c) is
//--- added to each class logit in the TRAINING gradient only, so the network absorbs the offset and
//--- its RAW argmax is already balanced-error-optimal at inference - no second correction at read
//--- time. 0 disables it. This is the sole survivor of the nine-input imbalance section audited away
//--- on 2026-07-31 (see Variables\Inputs.mqh); it is the only one of them with a consistency
//--- guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection
//--- already ranks on. m_logitAdjustLogged keeps the once-per-run tau/cap report to one line.
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior
Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add
tau*log(prior_c) to each class logit inside the training gradient. Softmax
CE on adjusted logits is consistent for BALANCED error - the metric
checkpoint selection already ranks on - so the loss and the deploy decision
finally optimize the same thing.
The engine already computed a true softmax + categorical-CE gradient and
wrote it over the per-neuron sigmoid delta, so this is an offset added to
three logits in the two places that gradient is built (backProp scalar path
and backPropOCL). No backend, kernel or DLL change; the forward pass and
every inference path are untouched, which is the point - the network learns
to absorb the offset, so its raw argmax becomes the balanced-optimal
decision with nothing applied at inference.
Replaces rather than stacks. Minority replay is disabled while this is on,
and the post-hoc inference prior is forced off. Stacking is not a
theoretical worry: simulated on the measured 1118/1119/34298 distribution
in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced,
Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance -
and BOTH together score 45.4% with Neutral recall at 0%, worse than either
alone. Buda et al. 2018 predicts exactly that.
Motivation from the six-chart run: every topology took one direction to
~50% recall and abandoned the other, the direction chosen arbitrarily (the
batch-norm control went Buy 1% / Sell 42%, the inverse of the other five).
One era in 1,301 cleared the per-class recall floor.
Fingerprinted conditionally, so the converged 60.7% models on disk keep
their filenames and stay loadable as the fallback.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
double m_logitAdjustTau ;
fix(ai): cap logit-adjustment strength to the head's usable logit range
tau=1.0 inverted the collapse instead of curing it. The head is SIGMOID, so
each output is bounded to [0,1] and the widest logit gap the net can express
between two classes is CLASS_LOGIT_SCALE * (1-0) = 6. The offsets are
tau*log(prior_c), whose spread on this 30:1 imbalance is 3.42 - so tau=1.0
spent 57% of the ENTIRE expressible range on the prior correction.
The network did the only thing available to it: saturate Buy/Sell outputs to
1.0 to overcome a -3.42 training handicap. The offsets are absent at
inference, so that surplus made every bar directional. Measured across all
five still-training charts: Neutral recall 0%, directional calls on ~100% of
bars, win rate 5-7% against a ~6% base rate - no information whatsoever -
while balanced accuracy read a flattering 58-64% because two of its three
terms sat near 95%. OOS accuracy 6%.
Menon et al. assume an unbounded logit head where a 3.42 shift is negligible
against the reachable range. It is not negligible here, so the strength is
now expressed RELATIVE to the range actually available:
tau_eff = min(tau_cfg, LOGIT_ADJUST_MAX_RANGE_FRACTION * SCALE / spread)
At 20% that gives tau 0.35 on this data. Deliberately a fraction rather than
a tau ceiling: it stays correct if CLASS_LOGIT_SCALE changes, if the head
becomes unbounded, or on any symbol whose imbalance differs. The input
remains effective below the cap, so dialling it down needs no rebuild.
Simulated at a signal strength where the task is genuinely learnable, the
precision/recall frontier is monotone: tau 1.0 -> 49.6% call rate at 6.4%
precision (base rate 6.1%, i.e. worthless); tau 0.35 -> 2.0% at 15.5%;
tau 0.15 -> 0.2% at 33.3%. The capped value lands in the same regime the
pre-logit-adjustment run occupied (1-6% of bars at 20-35% win rate).
Also logs the measured priors, the spread, and whether the cap bound.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 23:20:07 -04:00
bool m_logitAdjustLogged ;
fix: the imbalance correction never ran during the auto-tune search
Neutral collapse on all four topologies by era 5 with a 2:6 barrier
(recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on
"measuring...". One root cause, and it was not the barrier.
The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is
exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1%
of Neutral coming from the vertical barrier - so the new m*k horizon
scaling is right, arguably generous.
What was broken: Train()'s era-start block wrapped UpdateClassPriors() in
`if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode,
and AutoTuneIndicators ships ON, so on a default configuration EVERY era
of the search ran with unmeasured priors. ApplyLogitAdjustment() requires
measured priors; without them it calls ClearLogitAdjustment() and returns.
So the entire search trained under PLAIN cross-entropy. With a 52.5%
majority class the optimum of plain CE is "always predict Neutral", and
that is precisely what all four models found. The panel followed: its
counters only advance on bars the model CALLED Buy or Sell, so a
collapsed model leaves them at zero and the line reads "measuring..."
forever.
This was latent, not new. It has been true for every auto-tuned run, but
it was invisible while the labels were near-balanced - last night's
accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to
collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight
to survive noise) moved Neutral to the majority and exposed it.
The guard's stated fear cannot happen. These priors are measured from the
LABEL distribution, and the tuner only perturbs indicator periods
(MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and
TP_Mode - none of which the search touches - so every candidate sees
byte-identical labels and identical priors. There is nothing to
contaminate. What the guard actually protected was the .stats write, and
that is gated separately: eval candidates never checkpoint and never
persist.
Also, because this is the THIRD quiet no-op to cost a run in this
codebase (after the fictional oversampling log line and the shadow-blend
skip):
- ApplyLogitAdjustment() now WARNS when it declines to install, instead
of silently clearing. A mechanism that cannot announce it is not
running is indistinguishable from one that is.
- The panel distinguishes "measuring..." (before era 1, nothing scored
yet - an honest warm-up) from "no directional calls yet" (eras trained,
zero calls - a finding, not a wait).
Both builds compile 0 errors / 0 warnings. No retrain forced by this
commit itself, but the collapsed models must be discarded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
//--- Latch for the counterpart warning: the correction DECLINING to install. See ApplyLogitAdjustment().
bool m_logitAdjustSkipWarned ;
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior
Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add
tau*log(prior_c) to each class logit inside the training gradient. Softmax
CE on adjusted logits is consistent for BALANCED error - the metric
checkpoint selection already ranks on - so the loss and the deploy decision
finally optimize the same thing.
The engine already computed a true softmax + categorical-CE gradient and
wrote it over the per-neuron sigmoid delta, so this is an offset added to
three logits in the two places that gradient is built (backProp scalar path
and backPropOCL). No backend, kernel or DLL change; the forward pass and
every inference path are untouched, which is the point - the network learns
to absorb the offset, so its raw argmax becomes the balanced-optimal
decision with nothing applied at inference.
Replaces rather than stacks. Minority replay is disabled while this is on,
and the post-hoc inference prior is forced off. Stacking is not a
theoretical worry: simulated on the measured 1118/1119/34298 distribution
in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced,
Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance -
and BOTH together score 45.4% with Neutral recall at 0%, worse than either
alone. Buda et al. 2018 predicts exactly that.
Motivation from the six-chart run: every topology took one direction to
~50% recall and abandoned the other, the direction chosen arbitrarily (the
batch-norm control went Buy 1% / Sell 42%, the inverse of the other five).
One era in 1,301 cleared the per-class recall floor.
Fingerprinted conditionally, so the converged 60.7% models on disk keep
their filenames and stay loadable as the fallback.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
2026-07-23 19:36:34 -04:00
//--- True class base rates (natural, un-oversampled), measured from the label distribution each era
//--- (UpdateClassPriors, EMA-blended for stability) and PERSISTED alongside the weights (.stats
//--- sidecar) so live inference - including after a restart, when no training re-runs - calibrates
//--- exactly as training did. 0 = not yet measured => AdjustedSignalFromSoftmax falls back to raw.
double m_priorBuy , m_priorSell , m_priorNeutral ;
//--- Per-era OOS "fired" confusion counts under the LIVE decision rule: a directional call is counted
2026-07-26 18:33:12 -04:00
//--- whenever the prior-corrected posterior is non-Neutral, i.e. exactly the bars on which the
//--- deployed EA would cast a directional vote. m_oosBuyFiredHits/m_oosBuyFired = live Buy precision;
//--- likewise Sell. This is the metric that predicts forward-trading performance (recall/argmax-
//--- precision above score the RAW argmax, before the base-rate correction the live decision applies).
//--- Reset each era. Whether a counted vote also clears Min_Vote_Open against the other filters'
//--- average is an aggregate question this per-bar scorer cannot see - see its use site.
2026-07-23 19:36:34 -04:00
int m_oosBuyFired , m_oosBuyFiredHits ;
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
//--- The same live-fired population as above, but BUCKETED BY CONFIDENCE TIER (ConfidenceTier(), 4
//--- buckets quartiled from the head's structural floor). Exists to answer the one question the
//--- aggregate precision cannot: whether raising Min_Vote_Open would actually buy precision, and how
//--- much coverage it would cost. Tier weights are 25/50/75/100, and for an AI-only configuration the
//--- averaged vote IS the tier weight, so these four rows map directly onto the input: a floor of 50
//--- keeps tiers 1-3, 75 keeps 2-3, 100 keeps tier 3 alone. Measuring it beats guessing at it - the
//--- floor is only worth raising if precision actually rises monotonically across the tiers, and if it
//--- does not, that is itself the finding (the model's confidence is not calibrated to correctness).
int m_oosTierFired [ 4 ] , m_oosTierHits [ 4 ] ;
feat(rank): AI models rank their own confidence tiers from held-out outcomes
Closes the caveat 4858507 shipped with: the vote is a confidence percentage,
but only to the extent the pattern weights are measured. AI tier weights sat
at their designed defaults (25/50/75/100) because AI rows only ever arrive
from LIVE journaling, of which a training run produces almost none.
AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed
every scanned bar by ConfidenceTier(), which reads dPrevSignal - and
dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an
entire era's fires were bucketed by one stale, unrelated bar's confidence and
landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0)
T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the
calibration clamp. The clamp was real and was fixed then; this is a second,
independent cause of the identical output that survived that fix untouched -
which is why the log kept reading the same afterwards. Two causes, one symptom.
Now ConfidenceTierFor(adjSig): the bar this iteration actually scored.
WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of
"fill the database during training". The user's own observation is the reason:
a classic Pattern_2 is a fixed geometric condition, so its win rate is
legitimately accumulated over years, but an AI Pattern_2 means "confidence
landed in tier 2" and tier 2 under era 100's weights is a different statement
from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation
is exactly what is wrong here - it would average together models that no
longer exist, while colliding with the per-table row cap and mixing
measured-on-holdout outcomes into the live ledger's own tables. What the DB
actually supplies is a measured win rate per pattern, and pass 3 already
computes that on held-out bars, thousands at a time. So the model ranks itself
once per era, REPLACING rather than accumulating, which makes the weights
describe the current weights by construction.
ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades
BEFORE shrinking, which here would fire on every tier every era and hand all
four the pooled rate - the tiers could never separate and the mechanism would
be inert. Shrinkage is the answer to a small sample; a floor in front of it
means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N
pseudo-observations centred on the model's pooled holdout rate, counted in
EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw
fires can be worth ~12 independent ones. Rounded to the integer, not to the
decade NormalizeWinRate() uses, which would collapse the shrunk tiers back
into one number.
NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard:
weights are computed at the END of era N, so the vote scored during era N was
cast with era N-1's weights. The deploy gate never grades a vote whose weights
were fitted on the bars it is scoring. Residual leakage remains - the same OOS
bars each era under a different model - and is stated in the code rather than
papered over.
Both DB clobber paths are closed: ApplyPatternWeight() declines once
self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by
SelfRanked() - guarding only the tiers would have let the hourly ranking pass
undo half the self-ranking.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- Set by RankTiersFromOos() the first time this model measures its own tiers on held-out
//--- bars. Gates the signal DB out of this filter's pattern weights from then on.
bool m_tiersSelfRanked ;
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era
Full-pipeline analysis after "threshold 30, attained often, nothing drawn,
still glued to buy". The log falsified the premise before any code did:
21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0%
21:42:07 swept 4999, 0 voters, drew 0
21:51:30 swept 4999, 0 voters, drew 0
21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0%
The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root
cause, three symptoms: every display path read m_arrowSignalCache, which is
wiped to sentinel at each era start and only complete again when pass 3
finishes. With eras at ~30s and a sweep at ~17s:
* ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its
else-branch deleted the arrow on every voteless bar - erasing the previous
sweep's entire output. The chart cycled populated -> blank -> populated;
the user kept catching the blank phase.
* READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every
era and fell through to dPrevSignal - the frozen purge-band edge bar that
reads Buy. 659638e fixed which bar was frozen, not the freezing.
* VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a
different fraction of half-rebuilt caches.
THE FIX, structural rather than another patch:
1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one
moment the cache is complete - and now copies it (raw signals, newest
LOOKBACK+16 bars) into member-owned snapshot state, unconditionally,
BEFORE its early return: an all-Neutral era is a snapshot worth showing,
not an absence of one. Raw signals rather than votes, so a tier re-rank
between eras reprices them at read time via LiveVoteContribution for free.
2. The sweep (SnapshotVoteAt) and the prospective readout both read
snapshots; the readout's fallback chain is live-cache -> snapshot ->
dPrevSignal, and the snapshot leg is the one that fires most of the time.
3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual
sub-threshold vote takes an arrow down. This alone ends the wipe half of
the flicker even where snapshots are missing (before the first era).
4. Arming moved from an era-counter diff (which fires at era BOUNDARIES,
i.e. precisely when caches are about to be wiped) to
g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's
snapshot just got fresher", the only event a redraw can act on. 60s rate
limit collapses the four members' burst into one sweep. Classic-only
charts arm once at start.
5. Census now reports the direction split - "922 had a voter (610 buy / 312
sell)" - so "the vote leans buy" is checkable from the log instead of
inferred from arrow colours.
Also visible in the log and worth knowing: the threshold flip-flopped
30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51
ran at 40), so part of the observed blankness was configuration, not code.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
//--- ERA-END SNAPSHOTS of the arrow cache, taken in RankTiersFromOos() at pass-3 completion -
//--- the one moment the cache is complete for the era. Everything DISPLAY-side reads these, never
//--- the live cache, because the live cache is wiped to sentinel at every era start and is
//--- therefore empty for most of every era (the draw/wipe flicker and the glued-to-one-bar
//--- readout both traced to exactly that). m_prospectiveSigSnap is the newest scored bar's raw
//--- signal (-2 = no completed era yet); the array is the newest SIGNAL_RESCAN_LOOKBACK_BARS+16
//--- cache entries, raw signals, converted to votes at read time so a tier re-rank between eras
//--- reprices them without a copy.
double m_overlaySigSnap [ ] ;
int m_overlaySnapBars ;
double m_prospectiveSigSnap ;
2026-07-23 19:36:34 -04:00
int m_oosSellFired , m_oosSellFiredHits ;
2026-07-25 01:07:21 -04:00
//--- Cumulative (compounded, persistent) DIRECTIONAL accuracy = the win-rate of the model's Buy/Sell
//--- calls: of the bars it actually called Buy or Sell, how many matched the true label. Neutral ("no
//--- trade") calls are deliberately EXCLUDED - counting them inflates the rate to ~80%+ (Neutral is the
//--- ~94% majority the model gets right for free) and tells you nothing about trade quality. Summed
//--- across every era AND carried across restarts via the .stats sidecar (WST5); never reset era-to-era,
//--- so the panel shows a stable win-rate that only firms up as more pivots confirm. Incremented at the
//--- IS (pass 2) and OOS (pass 3) hit sites, only when the PREDICTION is directional, skipped for
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Reset only by ResetWeights (a fresh model).
2026-07-25 00:02:34 -04:00
long m_cumIsCorrect , m_cumIsTotal ;
long m_cumOosCorrect , m_cumOosTotal ;
2026-07-23 19:36:34 -04:00
//--- Latest live-fired precision (%) and fire count per direction (-1 = n/a), cached at era end for
//--- the status panel/log the same way m_lastBuyRecallPct is (see its comment).
int m_lastBuyFiredPrecPct , m_lastSellFiredPrecPct ;
int m_lastBuyFired , m_lastSellFired ;
2026-07-18 02:01:06 -04:00
//--- Ceiling on the class-balance sampleWeight multiplier (see its computation in Train(), keyed off
//--- m_prevEraTrueBuyCount/Sell/Neutral). MAX_WEIGHT_DELTA (AI\Network.mqh/.cl, DirectML\WarriorCPU.cpp)
//--- bounds how far a SINGLE weight can move in one step, but it does nothing to stop many small
//--- steps in the same direction from accumulating into a large net swing across an era - with real
//--- label counts around Buy 1340 / Sell 1312 / Neutral 7013 the uncapped ratio is ~5.3x on every
//--- single Buy/Sell example, every era. At the old hardcoded 3.0x ceiling that was STILL strong
//--- enough in practice to let one era's Buy gradients overwrite the separability the previous era
//--- had just built for Sell (and vice versa) - the observed anti-correlated Buy/Sell recall whipsaw
//--- (e.g. era 8 Buy:0%/Sell:26% -> era 9 Buy:10%/Sell:57% -> era 10 Buy:1%/Sell:37%), never both
2026-07-18 23:25:54 -04:00
//--- classes improving together. Was tunable (ClassSampleWeight input, CSW_15/1.5x default) as a
//--- loss-level multiplier, but that mechanism proved structurally too weak against Adam's near-
//--- invariance to constant gradient rescaling (Kingma & Ba 2015) - see the m_isTrainQueue queueing
//--- block's oversampling comment for the full history. Class-balance correction now happens
//--- entirely via data-level oversampling (repCount in that queueing block); this member is
2026-07-22 13:33:56 -04:00
//--- currently unread by that path. The ClassSampleWeight INPUT that used to set it was removed
//--- (it had no effect); the member is kept at its constructor default in case a smaller, additive
//--- loss-level nudge is ever reintroduced on top of oversampling.
2026-07-18 02:01:06 -04:00
double m_maxClassSampleWeight ;
2026-08-01 11:27:28 -04:00
//--- FOCAL LOSS REMOVED 2026-07-31 (was m_focalGamma / the FocalLossGamma input). Lin et al. 2017's
//--- (1-pt)^gamma is sound, but it corrects the SAME axis as the logit-adjusted loss, and stacking
//--- two corrections on one axis is the failure Buda et al. 2018 warns about (cited in the queueing
//--- block). It was also running at gamma*0.125, damped by a replay flag whose path was already dead.
//--- The ladder's escape is the learning-rate warm restart; the gamma anneal was a monotone step to
//--- zero, so nothing went with it. Full nine-input audit: the class-imbalance block in Inputs.mqh.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- ZigZag repainting embargo, in bars. The real MQL5 ZigZag (CustomIndicators\ADZigZag.mq5 - a
//--- renamed, logic-untouched copy of the stock ZigZag.mq5 at its stock defaults: Depth=12,
//--- Deviation=5 points, Backstep=3) revises its most recent 1-3 legs as new bars arrive (see
//--- ADZigZag.mq5's own ExtRecalc), so a bar's ADZigZagBuffer value is only trusted once at least
//--- this many MORE bars have closed after it.
//--- SCOPE NARROWED 2026-08-01: this used to gate the training LABELS as well, back when the target
//--- was the exact confirmed pivot. The target is now the triple barrier, whose lookahead is its own
//--- horizon (m_barrierHorizonBars), so this constant survives for exactly one job - keeping the
//--- swing-context INPUT FEATURES (m_useSwingContext) from reading a leg that can still change.
2026-07-14 22:36:27 -04:00
int m_swingConfirmationBars ;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Vertical barrier of the triple-barrier label, in bars - see BARRIER_HORIZON_LADDER_COUNT. Derived
//--- once by ComputeBarrierHorizonBars() at the start of the label prebuild and then held for the run.
//--- It is ALSO the label's lookahead depth, so it is what the IS/OOS embargo and the online-learning
//--- confirmation delay must both wait out: a bar's barrier label is not knowable until this many more
//--- bars have closed after it. That job used to belong to m_swingConfirmationBars, which answered the
//--- ZigZag question ("has this leg stopped repainting") and no longer answers the label's.
int m_barrierHorizonBars ;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon
Three defects found by reading the 2026-08-01 training logs, all of which
only became visible because the relabel made the numbers mean something.
1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET.
`OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101`
-101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in
7eb48f5. MetaTrader does not validate a saved enum input against the
enum's current members, so charts saved before that kept the old
integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;`
then quietly turned it into a 1:1 barrier, and all four topologies
trained ~250 eras against a strategy nobody selected - while the log
reported "target 1.00*ATR" as though it were configured.
Since the relabel these two inputs ARE the label definition, so this
is not a bad trade setting, it is a wrong dataset. ValidateBarrier-
Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix)
on any value that is not an enum member. Members are enumerated rather
than range-checked because both enums are sparse and carry negative
sentinels, so no min/max test can tell a legal value from a deleted
one - which is the entire failure mode. The fallback survives as
belt-and-braces but now announces itself: a fallback that cannot say
it fired is indistinguishable from correct behaviour.
2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE.
`tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct
is Buy+Sell as a share of all bars. At the old exact-pivot target that
was ~6%, so "beat the base rate" read as "beat chance" and the test
looked sound. Triple-barrier labels put it at ~83%, so the gate now
demanded 83% directional precision - impossible by construction.
Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
safe to deploy" at a perfectly healthy 43-45% precision, with no
checkpoint able to ship however good it got.
Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the
score of the degenerate always-call-one-direction model this floor
exists to reject. Correct at any base rate - ~43% on the current
labels, ~3% on the old rare-pivot ones. The era line now prints
"(chance N%, edge +Mpp)" beside the selection score, because 44%
precision is excellent against a 3% chance level and worthless against
a 43% one, and reading the first as the second is what made tonight's
run look better than it was.
3. THE HORIZON IGNORED THE BARRIER GEOMETRY.
ComputeBarrierHorizonBars() returned the median ZigZag leg, which
measures how long a ~1 ATR move takes and says nothing about how long
the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales
with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled
horizon would have timed out most 1:3 trades and pushed Neutral
straight back up, re-creating the imbalance the relabel removes.
Now multiplied by slMult*tpMult, calibrated against a real measurement
rather than assumed: the accidental 1:1 run resolved at horizon 12 with
only 16.7% timeouts, so the swing median is the right scale at m*k=1.
Verifiable, not just asserted: the prebuild now counts barriers that
ended on the VERTICAL barrier and reports them as a share of Neutral.
Neutral conflates "timed out" with "stopped out" and only the first
indicts the horizon.
Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting
TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right:
no existing model was trained on the intended target.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
//--- Latch so the invalid-TP fallback in BarrierMultiples() shouts once, not once per labelled bar.
bool m_barrierFallbackWarned ;
//--- Did the LAST TripleBarrierLabel() call run out of horizon without either barrier being touched?
//--- Neutral conflates two very different outcomes - "the trade timed out" and "the stop was hit
//--- before the target" - and only the first one indicts the horizon. Counting them apart is what
//--- lets the m*k horizon scaling in ComputeBarrierHorizonBars() be VERIFIED against a real run
//--- rather than trusted: a high timeout share means the horizon is too short for the configured
//--- barrier, a high stop-out share just means the barrier is hard.
bool m_lastBarrierTimedOut ;
fix: both-won bars were labelled "do not trade" - resolve by first touch
Removing the min-reward:risk raise let the MEASURED geometry come back with
the target NEARER than the stop (SP500 H1: target 1.62*ATR at q50 of
favourable, stop 3.33*ATR at q75 of adverse). That reopened a branch the
code called unreachable: price can reach +target and -target inside one
horizon, winning in BOTH directions, and those bars fell through to Neutral.
Neutral has only three producers, both-lost is unreachable (you cannot touch
-3.33 without crossing -1.62 first, which wins the short), and timeouts logged
at 1.0% of Neutral - so ~27% of ALL bars were being handed to the model as the
abstain class when a trade either way would have collected its target. The
cleanest positives in the sample, labelled "do not trade", while the fitted
confidence threshold was being asked to find selectivity in what was left.
Resolved by FIRST TOUCH: the target reached earlier is the trade that would
have closed first. Same forward window, no extra lookahead. Same-bar ties stay
Neutral - OHLC cannot order two touches, and unlike an intrabar stop tie there
is no pessimistic side to fall to, so a guess would inject a coin-flip
direction into the target.
Also:
- count both-won and its same-bar tie subset in the prebuild line, so the
share is measured rather than inferred from arithmetic on a log line
- scope the timeout counter to IS, matching the tally it is reported as a
percentage OF; it was incremented over the whole scan and divided by an
in-sample denominator
- clear m_lastBarrierTimedOut at the top of the walk with the excursions, not
at the bottom - the two early returns published the previous bar's verdict
- mark the pass-1 label line PROVISIONAL. It prints the enum fallback because
geometry can only be derived from excursions that do not exist yet, and it
reads exactly like a config change that failed to take effect
FORCES RETRAIN. Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:35:42 -04:00
//--- Did the LAST call find BOTH directions' targets reachable inside the horizon? Impossible while
//--- reward >= risk - the case only opened up when the minimum-reward:risk raise was removed and the
//--- MEASURED geometry came back with the target (q50 of favourable travel) CLOSER than the stop (q75
//--- of adverse). It is not an edge case there: it is the whipsaw class, and on SP500 H1 it accounts
//--- for nearly all of Neutral. Published so the prebuild can count it - see
//--- m_labelPrebuildBothWonCount for why a bar that wins in either direction must not be labelled
//--- "do not trade".
bool m_lastBarrierBothWon ;
//--- Subset of the above where both targets fell inside the SAME bar, so OHLC cannot say which came
//--- first. Those stay Neutral, for the same reason intrabar ties score as the stop: the file refuses
//--- to order two touches it cannot see the order of.
bool m_lastBarrierBothWonTied ;
fix: the deploy gate was benchmarking a win rate against a label frequency
The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a
driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade
is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test.
That invariant needs reward >= risk, and the measured geometry no longer
satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but
both-won bars were stripped out of Buy and Sell so the label base rate read
37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing
models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma
against 37.5% and loses money on every single trade. Live since 217b9bc.
Root cause is that label agreement stopped being the same question as trade
profitability. Buy implies winLong, but the converse fails on every both-won
bar, and the label can only name one of two directions that both pay.
So stop asking the model whether it matched a label and start asking whether
its trade paid:
- cache winLong/winShort per bar beside the label, under the same validity
flag; published from the barrier walk before the collapse to 3 classes
- dirPrecPct now counts wins on the side actually called
- chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook
m/(m+k) would credit SP500's drift to the model
- the NMS "what would I have made" pair, the live-fired precision, and the
IS/OOS cumulative win rates all move to the same test. IS and OOS are read
side by side as the overfitting signal, so measuring one in wins and the
other in agreement would put a fixed gap between them that has nothing to do
with generalization
- the confidence threshold is FITTED on wins too, so the operating point
maximises what the gate grades
- per-class label-agreement precision is still computed and logged; it is the
right diagnostic for class separation, just not for a deploy decision
- era line renamed dir-precision -> win-rate, chance -> chance=break-even
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Did a LONG / a SHORT placed at this bar reach its target before its stop? These are the raw
//--- questions the barrier walk answers, before they are collapsed into one 3-class label, and they
//--- are what a trade's profitability actually depends on. Kept separately because the collapse is
//--- LOSSY in exactly the case that now matters: on a both-won bar the label names one direction, but
//--- BOTH trades would have paid, so scoring the other call as an error understates the model. See
//--- m_oosWinLongTotal for why the deploy gate had to stop using label agreement as its hit test.
bool m_lastWinLong ;
bool m_lastWinShort ;
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- Excursions of the bar TripleBarrierLabel() just resolved, in ATR units, published the same way
//--- m_lastBarrierTimedOut is: the walk that finds them is the walk the label already does, so they
//--- cost one max and one min per bar rather than a second pass over history.
double m_lastExcUp ; // (maxHigh - entry)/ATR over the horizon, >= 0
double m_lastExcDown ; // (entry - minLow)/ATR, >= 0
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder
Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped
stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384).
1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels
started one per bar overlap by the label's lifespan, so n calls are worth
~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample
uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count.
The tell: the operating point's null-of-the-maximum gate is family-wise and
should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73
(64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model
whose margin distribution admits few bins, sat on the null; the rest cleared
a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently
alternated between the ENDS of its own range era to era (0.10 -> 0.88 ->
0.86 -> 0.66; coverage 16% <-> 73%).
TripleBarrierLabel now records when each label became KNOWABLE - the first
winning touch, or both stops, or the timeout - and the prebuild accumulates
the mean. EffectiveSampleSize() feeds the operating point, the member deploy
gate and the ensemble vote gate. Conservative by construction (n/L is an
upper bound on the damage); gates get harder, never easier.
2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and
since 4d8cb08 reachability is measured OVER that horizon - so a wider rung
buys itself the time that makes it look reachable. Same target -> horizon ->
reach -> target loop the excursion window is kept short to avoid; fixing the
window confusion reopened it through the other door. It walked 128 -> 256 ->
384 bars and stopped at q90, the widest rung there is, with every rung
reading 39-48% against a 20% floor. A floor nothing fails selects nothing.
Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected -
the same rule ReportGeometryExpectancyScan already applied. It was printing
the shipped pair as CLAMPED and disqualified ('h384!') two lines under the
deriver that chose it: two subsystems, one geometry, opposite verdicts.
3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped
independently to the coarse first-passage grid, re-rating each candidate:
q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL
measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the
ladder compared win shares taken at ratios from 1.63 to 2.17 and read the
differences as scale. It is why the reach column came out non-monotone in
width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in
log space and the target follows the ratio off it; the pair actually measured
is returned and logged, so a collision reads as a collision.
Also: LadderWinShare guarded against the conditional (fractal) geometry path,
which fills n from m_fracLegCount while leaving idxList empty - a latent
out-of-bounds on a currently-dead path.
New log lines: mean label lifespan and effective n on the label-cache line, the
required-vs-available horizon per rung, and the grid pair the reconciliation
actually measured (its tolerance now scales with the grid skew instead of a flat
5pp).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//--- LABEL LIFESPAN, in bars: how long after entry this bar's label became KNOWABLE - the age of the
//--- barrier touch that fixed the outcome, or the full horizon when it timed out. Published for the
//--- bar just walked, exactly like m_lastExcUp.
//---
//--- WHY THIS EXISTS, and it is not a diagnostic. Triple-barrier labels on adjacent bars OVERLAP: a
//--- label that takes L bars to resolve shares L-1 bars of its outcome window with its neighbour, so
//--- consecutive labels are nowhere near independent draws. Every standard error in this codebase was
//--- computed as sqrt(p(1-p)/n) on the RAW call count, which is the formula for n independent
//--- observations - and with L in the hundreds the true error is larger by ~sqrt(L).
//---
fix(labels): correct EffectiveSampleSize clamp order, share the horizon ladder, retract a false justification
Self-review of 1540ba8 against the FULL 6,930-era log rather than the first
three minutes of it. Three corrections.
1. EffectiveSampleSize() clamped in the wrong order. MathMax(2, MathMin(eff,
rawN)) returns 2 when rawN is 1 - an effective sample LARGER than the raw
one, shrinking the SE in exactly the direction the function exists to
prevent. Floor first, cap at rawN last.
2. The horizon cap rejected on the CEILING only, and said so as though that
made the label untruncated. It does not: the horizon ladder also snaps DOWN,
so a pair needing 317 bars is granted 256 and is silently truncated without
ever being flagged CLAMPED. Added SnapHorizonToLadder() / GrantedHorizonBars()
and the scale ladder now reports "needs N gets M" per rung. Rejection stays on
the ceiling alone - matching ReportGeometryExpectancyScan's '!' exactly, which
was the point - because rejecting on the snap-down would select rungs for
landing just above a ladder point rather than for anything about the market.
ComputeBarrierHorizonBars' private copy of the ladder is gone; there is now
one copy, which is the whole reason RequiredHorizonBars was factored out.
3. RETRACTED THE JUSTIFICATION IN 1540ba8's COMMENTS. That commit claimed the
overlap correction was needed because the operating point's null-of-the-
maximum gate fired on 47/73 Perceptron eras (64%) where a family-wise test
should fire on ~5%. Those 73 fits were the first three minutes of a
six-and-a-half-hour run. Over the full run:
PAI 47/3214 = 1.5% HYB 30/1200 = 2.5%
CONV 4/63 = 6.3% LSTM 75/915 = 8.2%
All at or below the null. The gate from 7414570 is working as designed and
PAI's 47 clears were a cold-start transient never repeated in 3,141 later
fits; its threshold over the run's second half has sd 0.01. The overlap
correction is still right - sqrt(p(1-p)/n) on overlapping labels is the wrong
formula - but it fixes no observed failure, and it costs nothing today
because no model is near the deploy line.
WHAT THE FULL RUN DOES CONFIRM, unchanged: the geometry ran away exactly as
described (2.00/6.00 h128 -> 3.49/6.99 h256 -> 4.86/9.71 h384, three passes,
stopping at q90 because the quantile ladder ended), the label stayed long-skewed
at Buy 42.9% / Sell 22.6%, and no checkpoint on any of the four models ever
cleared the deployability floor. Pooled declustered win rates: PAI 31.70%,
HYB 31.02%, LSTM 32.69%, CONV 31.57% - every one 4-6pp below the 37% always-long
chance rate and 1-2.7pp below the 33.7% cost-adjusted break-even.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 07:30:19 -04:00
//--- THE JUSTIFICATION IS THE OVERLAP ITSELF, not a misbehaving gate. Recording this because the
//--- first version of this comment claimed otherwise and was WRONG: it cited the operating point's
//--- null-of-the-maximum gate firing on 47 of 73 Perceptron eras (64%) where a family-wise test
//--- should fire on ~5%. That 73 was the first three minutes of a six-and-a-half-hour run. Over the
//--- full 6,930 eras the same gate fired 47/3214 = 1.5% (PAI), 2.5% (ConvLSTM), 6.3% (Convolutional),
//--- 8.2% (LSTM) - all of them at or below the null, i.e. the gate from 7414570 is working exactly as
//--- designed, and PAI's 47 clears were a cold-start transient it never repeated in 3,141 later fits.
//--- The operating point it produces is stable too: PAI's threshold over the run's second half has a
//--- standard deviation of 0.01.
//---
//--- So this correction fixes a real statistical error that was NOT, on this evidence, causing an
//--- observable failure. It is worth having anyway - sqrt(p(1-p)/n) on overlapping labels is simply
//--- the wrong formula - and it costs nothing today because no model is anywhere near the deploy
//--- line (all four finished 4-6pp BELOW the always-long chance rate). It will matter on the day one
//--- is close, and then it will demand a much larger measured edge before certifying anything.
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder
Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped
stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384).
1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels
started one per bar overlap by the label's lifespan, so n calls are worth
~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample
uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count.
The tell: the operating point's null-of-the-maximum gate is family-wise and
should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73
(64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model
whose margin distribution admits few bins, sat on the null; the rest cleared
a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently
alternated between the ENDS of its own range era to era (0.10 -> 0.88 ->
0.86 -> 0.66; coverage 16% <-> 73%).
TripleBarrierLabel now records when each label became KNOWABLE - the first
winning touch, or both stops, or the timeout - and the prebuild accumulates
the mean. EffectiveSampleSize() feeds the operating point, the member deploy
gate and the ensemble vote gate. Conservative by construction (n/L is an
upper bound on the damage); gates get harder, never easier.
2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and
since 4d8cb08 reachability is measured OVER that horizon - so a wider rung
buys itself the time that makes it look reachable. Same target -> horizon ->
reach -> target loop the excursion window is kept short to avoid; fixing the
window confusion reopened it through the other door. It walked 128 -> 256 ->
384 bars and stopped at q90, the widest rung there is, with every rung
reading 39-48% against a 20% floor. A floor nothing fails selects nothing.
Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected -
the same rule ReportGeometryExpectancyScan already applied. It was printing
the shipped pair as CLAMPED and disqualified ('h384!') two lines under the
deriver that chose it: two subsystems, one geometry, opposite verdicts.
3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped
independently to the coarse first-passage grid, re-rating each candidate:
q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL
measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the
ladder compared win shares taken at ratios from 1.63 to 2.17 and read the
differences as scale. It is why the reach column came out non-monotone in
width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in
log space and the target follows the ratio off it; the pair actually measured
is returned and logged, so a collision reads as a collision.
Also: LadderWinShare guarded against the conditional (fractal) geometry path,
which fills n from m_fracLegCount while leaving idxList empty - a latent
out-of-bounds on a currently-dead path.
New log lines: mean label lifespan and effective n on the label-cache line, the
required-vs-available horizon per rung, and the grid pair the reconciliation
actually measured (its tolerance now scales with the grid skew instead of a flat
5pp).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//---
//--- The correction is Lopez de Prado's sample uniqueness (Advances in Financial Machine Learning,
//--- ch. 4) - the same triple-barrier framework this label already implements. With one label started
//--- per bar and mean lifespan L, average concurrency is ~L and average uniqueness ~1/L, so the
//--- effective sample size is n/L. See EffectiveSampleSize().
int m_lastLabelLifespan ;
//--- Running mean of the above over every bar the label cache has resolved this process. Rebuilt with
//--- the cache (a horizon change makes every previous measurement answer a different question), and
//--- deliberately a plain mean rather than an EMA: it is a property of the geometry, not a time series.
double m_labelLifespanSum ;
long m_labelLifespanCount ;
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in
3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever
chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of
eleven guesses is not deriving anything.
WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It
ranks pairings by how predictable their OUTCOME is - a question about direction.
The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing
absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE
scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x.
Hence the scan failing its own gate on every run, and its "winner" wandering
2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is
strongly measurable, so derive the geometry from that instead.
stop = q25 of measured ADVERSE travel (ordinary noise does not reach it)
target = q50 of measured FAVOURABLE travel (reached ~half the time, by
construction, inside the horizon)
Continuous, in ATR units, superseding the enum multiples. Reachability ("target
on X% of bars, stop on Y%") and the implied break-even are printed so the choice
is auditable rather than trusted.
FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the
horizon with the target (first-passage time grows with the band) and the
excursions are measured OVER the horizon, so target -> horizon -> excursions ->
target is a real loop - deriving once sizes the target from travel measured
under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at
3 passes, and says so if it does not settle.
Does NOT create expectancy, and the log says as much: chance precision equals
break-even at every geometry (m/(m+k) on both sides). It buys a target the
market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio
forces a target the market rarely reaches, it WARNS rather than overriding -
the ratio is the user's risk policy, so the honest move is to state its cost.
That is the collision that once rejected 100% of setups.
Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg
files written earlier today still load (their length guard finds no doubles) and
a model that carries them was trained on them and never re-derives.
Also fixes a message from e5ceed6 that claimed "this model resumed from disk"
unconditionally - it printed above a "seeding era 0" line on a brand-new model,
because the branch fires whenever the cache is not built, which is equally true
before a fresh model's first prebuild. A diagnostic that misreports its own
trigger is worse than one that says nothing: it gets quoted back as evidence.
FORCES A FULL RETRAIN (labels change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- DERIVED barrier multiples, in ATR units, taken from the measured excursion distribution rather
//--- than from an enum. Zero means "not derived yet" and BarrierMultiples() falls back to the mode
//--- constants. Continuous on purpose: the whole point is to stop snapping the geometry to {2,3} x
//--- {2,3,4,6,8,10}, a grid whose members were guesses.
double m_derivedSlMult ;
double m_derivedTpMult ;
bool m_geometryDerived ;
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- DIRECTIONAL CONFIDENCE THRESHOLD - see DIR_CONF_THRESHOLD_BINS for the rationale. The ACTIVE
//--- value, read by AdjustedSignalFromSoftmax() on every live and pass-3 decision; 0.0 means
//--- unthresholded (era 0, or an IS histogram too sparse to fit). Refitted at the end of every
//--- pass 2 from that era's own IS margins, because the margin distribution moves with the weights.
double m_dirConfThreshold ;
//--- The value that belongs to the CHECKPOINTED weights. Captured at the same instant as
//--- Net.CaptureWeights() and restored beside them, because a threshold fitted for one set of
//--- weights is meaningless against another - deploying era 40's weights under era 63's operating
//--- point would silently change both coverage and precision away from the numbers the deploy gate
//--- actually cleared.
double m_bestDirConfThreshold ;
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Margin histogram for the fit, rebuilt each era from the CALIBRATION slice (see
//--- DIR_CONF_CALIB_PCT_OF_IS). Index = bin of (winning direction's probability - its best rival's);
//--- [0] counts directional calls, [1] counts the ones whose implied trade WON. No oversampling
//--- applies here - the calibration walk visits each bar once, chronologically - so the primary-only
//--- correction the replay queue needed (see m_cumIsTotal) is structural rather than a filter.
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
long m_dirConfBinCalls [ DIR_CONF_THRESHOLD_BINS ] ;
long m_dirConfBinHits [ DIR_CONF_THRESHOLD_BINS ] ;
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
long m_dirConfPrimaryBars ; // denominator for coverage: every calibration bar scored
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- one-shot so the "histogram too sparse" explanation is stated once per run, not once per era
bool m_dirConfSparseWarned ;
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in
3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever
chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of
eleven guesses is not deriving anything.
WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It
ranks pairings by how predictable their OUTCOME is - a question about direction.
The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing
absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE
scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x.
Hence the scan failing its own gate on every run, and its "winner" wandering
2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is
strongly measurable, so derive the geometry from that instead.
stop = q25 of measured ADVERSE travel (ordinary noise does not reach it)
target = q50 of measured FAVOURABLE travel (reached ~half the time, by
construction, inside the horizon)
Continuous, in ATR units, superseding the enum multiples. Reachability ("target
on X% of bars, stop on Y%") and the implied break-even are printed so the choice
is auditable rather than trusted.
FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the
horizon with the target (first-passage time grows with the band) and the
excursions are measured OVER the horizon, so target -> horizon -> excursions ->
target is a real loop - deriving once sizes the target from travel measured
under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at
3 passes, and says so if it does not settle.
Does NOT create expectancy, and the log says as much: chance precision equals
break-even at every geometry (m/(m+k) on both sides). It buys a target the
market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio
forces a target the market rarely reaches, it WARNS rather than overriding -
the ratio is the user's risk policy, so the honest move is to state its cost.
That is the collision that once rejected 100% of setups.
Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg
files written earlier today still load (their length guard finds no doubles) and
a model that carries them was trained on them and never re-derives.
Also fixes a message from e5ceed6 that claimed "this model resumed from disk"
unconditionally - it printed above a "seeding era 0" line on a brand-new model,
because the branch fires whenever the cache is not built, which is equally true
before a fresh model's first prebuild. A diagnostic that misreports its own
trigger is worse than one that says nothing: it gets quoted back as evidence.
FORCES A FULL RETRAIN (labels change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
int m_geometryDerivePasses ; // fixed-point iteration counter, capped
fix: a restart no longer loses the measured geometry or the training window
Terminal restart, 22:25: all four resumed models sat on empty windows with
enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt:
1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's
dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the
resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as
"Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to
the training-window rule before computing its window; the pre-scan did not.
The rule is now factored into TrainWindowStart() and both use it. The scan
also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as
OnInit), and deployed models keep their watermark - for them it gates
inference recency, not a training window.
2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran
against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon
latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A
leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the
label cache wiped if it moved (labels from two horizons answer different
questions), and the geometry deriver refuses to run from it - a pair derived
over a warm-up window would get PINNED.
3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model
creation and at weights-reset - both BEFORE era 0 derives - so the measured
pair lived only in memory: every restart read back zeros, adopted nothing,
fell back to the enum barriers, and the era-0-only gate meant a resumed model
could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6.
Now: the settled pair is pinned to the .cfg the moment derivation completes
(one-shot, atomic write), and the derive gate accepts any model with no
pinned pair, not just era 0 - mid-run stability is carried by
m_geometryDerived itself, which never allows a second derivation.
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//--- The last ComputeBarrierHorizonBars() ran with FEWER confirmed ZigZag legs than the median
//--- needs, so the horizon it returned is the fallback, not a measurement. While this is true the
//--- horizon stays PROVISIONAL (m_barrierHorizonResolved is left false so the next full rebuild
//--- re-resolves it) and the geometry deriver refuses to run - deriving a permanent, .cfg-pinned
//--- barrier from excursions measured over a made-up window would pin the artifact, not the market.
//--- Observed 2026-08-09 22:25: a terminal restart ran the resumed-model pre-scan in the same
//--- second as OnInit, the ZigZag had 0 calculated legs, and the horizon latched to fallback(32) x
//--- slMult x tpMult = 384 for the whole process.
bool m_barrierHorizonLegStarved ;
bool m_horizonStarvedWarned ; // one-shot: the starved path can retry every call
//--- Has THIS process written the derived geometry into the .cfg? The .cfg was only ever written at
//--- model creation and at weights-reset - both BEFORE era 0's derivation - so the derived pair
//--- never reached disk, every resumed model read back zeros, adopted nothing, fell back to the
//--- enum barriers, and (era > 0) never re-derived: trained on 3.33/1.62 all day, relabelled at 2:6
//--- on the next restart. Set by the post-derivation save, and also by the adoption path (the pair
//--- is already on disk there).
bool m_geometryCfgSaved ;
fix(consistency): one geometry authority, one exit authority, and the MI screen finally gets a veto
Consistency pass before a fresh deployment. Three places where two systems were
choosing the same thing and one of them silently lost.
1. THE GEOMETRY SCAN'S DECISION WAS INERT - measured, not suspected.
USDJPY, 2026-08-17:
14:24:12.844 adopting barrier geometry 2:8 ... Relabelling and training on it.
14:24:12.979 triple-barrier labels - stop 1.61*ATR, target 3.21*ATR ... this is what trains
It adopted 2:8 and trained on 1.61:3.21. ReportBarrierGeometryScan wrote only
m_sl_mode/m_tp_mode, and BarrierMultiples ranks the DERIVED pair ABOVE those ints -
so on any model carrying a derived pair (every model with a .cfg, including a fresh
one whose weights are gone but whose sidecar survived) the adoption changed nothing.
Worse, had it changed something it would have been undone immediately: the adoption
sets m_labelCachePrebuilt = false, and that prebuild re-runs DeriveBarrierGeometry at
era 0, which overwrites m_derivedSl/TpMult from the excursion quantiles.
ONE AUTHORITY: the derived pair, because it is what the labels read, what the deploy
gate certifies, what g_Derived*AtrMult places on the live order, and what the .cfg
pins across restarts. The scan now writes THAT (floored by MIN_SL_ATR_MULTIPLIER, the
same floor DeriveBarrierGeometry applies so the live stop can never be wider than the
labelled one), republishes to the bridge immediately rather than at the next era end,
and forces the sidecar to be rewritten. m_geometryAdopted latches it so the derive
pass the adoption itself triggers cannot overwrite it.
The scan outranks the derive for an evidential reason, not an architectural one: its
winner cleared a permutation test against the null of the MAXIMUM over every eligible
pairing, and it scores the incumbent derived pair as a peer in that same field. The
derive is a descriptive quantile read with no significance test attached.
BEHAVIOURAL CHANGE, and the reason to flag it before a fresh test: barrier geometry
will now actually move when the scan says so. Until today it never did.
2. ONE EXIT AUTHORITY, tied to whose certificate the trade was placed under.
CheckClosePosition had two routes. The AI early-exit reads the AI vote undiluted and
is exactly what the new exit replay reproduces. The blended route thresholds
m_direction, the average over EVERY filter including classic ones whose live votes
pass 3 never computes - so it can close a position the certificate never modelled,
and no replay can ever check it.
When g_DerivedSlAtrMult > 0 the AI's measured geometry is on the order, which means
the deploy gate's certificate is the reason the trade exists. In that state the AI now
governs the exit and the blended route is suppressed. Classic-only configurations are
untouched: there the blended route is the only exit opinion and stays exactly as it
was. Nothing moves at the shipped defaults either way (Min_Vote_Close = Disabled).
3. EDGEFINDER, SECOND HALF: THE MEASUREMENT NOW STEERS.
The MI suite has always printed its verdicts and then trained the direction target
regardless of what they said. That gap IS the difference between this and the
EdgeFinder discipline: measure what the market offers, THEN aim.
m_dirEvidence is set when EITHER the feature/label mutual information OR the
normalised excursion asymmetry clears its block-permuted null - an OR, because the two
look for the same thing by different routes and requiring both would reject on the
weaker of two independent measurements. Normalised asymmetry specifically, never the
raw one, which is the volatility confound.
Deploy - solo AND ensemble - now requires it. A run without it still trains, and keeps
its checkpoint: the research value is real and the measurement can be wrong. It simply
may not go live. Reported separately from the statistical gate because the remedy is
different: a failed selection test says train differently, this says look somewhere
else. Excursion SIZE keeps clearing where direction does not, and that is a
risk-control head rather than an entry signal.
For the ensemble the check is per-chart by construction - the MI suite runs once and
shares its outcome across members - which is the honest treatment: four models finding
nothing between them is not four chances at an edge, it is four fits to the same
absent information.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:43:20 -04:00
//--- ADOPTED-GEOMETRY LATCH. True once ReportBarrierGeometryScan has crowned a pairing that cleared
//--- its family-wise null. Two measurements choose this geometry and until 2026-08-17 they fought:
//--- DeriveBarrierGeometry re-measures the SCALE off the excursion quantiles on every era-0 pass and
//--- overwrites m_derivedSl/TpMult, while the scan wrote only the legacy mode ints - which
//--- BarrierMultiples ranks BELOW the derived pair. So the scan's decision was inert, and had it not
//--- been it would have been overwritten by the next derive pass anyway.
//--- The scan wins, and this latch is how. The reason is evidential rather than architectural: the
//--- scan's winner cleared a permutation test against the null of the MAXIMUM over every eligible
//--- pairing, and it scores the incumbent derived pair as a peer in that same field. The derive is a
//--- descriptive quantile read with no significance test attached. A measurement that survived a
//--- family-wise gate outranks one that was never tested against a null.
bool m_geometryAdopted ;
//--- MEASURED DIRECTIONAL EVIDENCE, and the one thing that makes the MI suite a SCREEN rather than a
//--- commentary. Set when either the feature/label permutation test or the normalised excursion
//--- asymmetry clears its null; false when neither does. The suite has always printed these verdicts
//--- and then trained the direction target regardless of what they said - which is the whole gap
//--- between this and the EdgeFinder discipline the user runs in SQX: measure what the market offers,
//--- THEN aim. A run with no measured directional information may still train (the research value is
//--- real and the measurement can be wrong), but it must not DEPLOY: no amount of fitting creates
//--- information the screen could not find, and every direction verdict this project has closed was
//--- closed after exactly that was attempted anyway.
bool m_dirEvidence ;
string m_dirEvidenceWhy ;
fix: excursion window must not depend on the barrier it sizes
DIRECTION IS NOT THERE, and this run is what establishes it. Three symbols:
raw ASYMMETRY clears on all three (p=0.0199 / 0.0050 / 0.0050)
norm ASYMMETRY collapses on all three (p=0.3433 / 0.5075 / 0.2736),
USDCAD landing BELOW its own null
RANGE control strengthens to 3-5x its null everywhere
Divide sigma out and the apparent directional signal vanishes entirely. What
cleared was volatility leaking through an unnormalised difference. Note this
would have passed any replication test: three instruments at p=0.005 is exactly
the evidence one would accept before committing to a rebuild, and the confound
reproduces perfectly. Replication was never going to catch it - only the
normalisation could.
Two defects of mine, both surfaced by the same run.
1. THE GEOMETRY DERIVATION WAS DIVERGING, NOT CONVERGING. It produced a
14.57*ATR stop and a 29.14*ATR target that only 5.7% of bars ever reach.
Excursions were measured over the barrier horizon; the horizon scales with
the target; the target is a quantile of the excursions - so target ->
horizon -> excursions -> target ran away, and "settled" only because the
horizon ladder caps at 384 bars. A saturated runaway, which the iteration
guard could not catch because it watches for OSCILLATION.
Fixed at the root: excursions now accumulate only over m_swingMedianBars -
the UNSCALED median ZigZag leg, a property of the instrument that owes
nothing to the barrier. The barrier walk still runs the full horizon,
because that is how long the trade is held; only the MEASUREMENT used to
size the barrier is confined to a geometry-independent window.
(The Min_Risk_Reward_Ratio warning fired correctly and is what flagged it -
the diagnostic worked while the derivation behind it did not.)
2. THE CONFOUND VERDICT WAS UNREACHABLE. `sizeCleared && !asymCleared` was
tested first and is true whenever size clears - i.e. always - so the branch
that NAMES the volatility confound never printed; all three symbols showed
the generic size-not-direction message instead. Verdict chain rewritten with
the specific case first, and the dangling elses my first patch introduced
removed.
FORCES A FULL RETRAIN (the excursion window changes every derived barrier).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:57:23 -04:00
//--- Median confirmed ZigZag leg in bars, UNSCALED by the barrier. The window excursions are measured
//--- over, kept independent of the geometry so sizing the geometry from them cannot feed back.
int m_swingMedianBars ;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon
Three defects found by reading the 2026-08-01 training logs, all of which
only became visible because the relabel made the numbers mean something.
1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET.
`OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101`
-101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in
7eb48f5. MetaTrader does not validate a saved enum input against the
enum's current members, so charts saved before that kept the old
integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;`
then quietly turned it into a 1:1 barrier, and all four topologies
trained ~250 eras against a strategy nobody selected - while the log
reported "target 1.00*ATR" as though it were configured.
Since the relabel these two inputs ARE the label definition, so this
is not a bad trade setting, it is a wrong dataset. ValidateBarrier-
Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix)
on any value that is not an enum member. Members are enumerated rather
than range-checked because both enums are sparse and carry negative
sentinels, so no min/max test can tell a legal value from a deleted
one - which is the entire failure mode. The fallback survives as
belt-and-braces but now announces itself: a fallback that cannot say
it fired is indistinguishable from correct behaviour.
2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE.
`tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct
is Buy+Sell as a share of all bars. At the old exact-pivot target that
was ~6%, so "beat the base rate" read as "beat chance" and the test
looked sound. Triple-barrier labels put it at ~83%, so the gate now
demanded 83% directional precision - impossible by construction.
Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
safe to deploy" at a perfectly healthy 43-45% precision, with no
checkpoint able to ship however good it got.
Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the
score of the degenerate always-call-one-direction model this floor
exists to reject. Correct at any base rate - ~43% on the current
labels, ~3% on the old rare-pivot ones. The era line now prints
"(chance N%, edge +Mpp)" beside the selection score, because 44%
precision is excellent against a 3% chance level and worthless against
a 43% one, and reading the first as the second is what made tonight's
run look better than it was.
3. THE HORIZON IGNORED THE BARRIER GEOMETRY.
ComputeBarrierHorizonBars() returned the median ZigZag leg, which
measures how long a ~1 ATR move takes and says nothing about how long
the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales
with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled
horizon would have timed out most 1:3 trades and pushed Neutral
straight back up, re-creating the imbalance the relabel removes.
Now multiplied by slMult*tpMult, calibrated against a real measurement
rather than assumed: the accidental 1:1 run resolved at horizon 12 with
only 16.7% timeouts, so the swing median is the right scale at m*k=1.
Verifiable, not just asserted: the prebuild now counts barriers that
ended on the VERTICAL barrier and reports them as a share of Neutral.
Neutral conflates "timed out" with "stopped out" and only the first
indicts the horizon.
Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting
TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right:
no existing model was trained on the intended target.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
int m_labelPrebuildTimeoutCount ;
fix: both-won bars were labelled "do not trade" - resolve by first touch
Removing the min-reward:risk raise let the MEASURED geometry come back with
the target NEARER than the stop (SP500 H1: target 1.62*ATR at q50 of
favourable, stop 3.33*ATR at q75 of adverse). That reopened a branch the
code called unreachable: price can reach +target and -target inside one
horizon, winning in BOTH directions, and those bars fell through to Neutral.
Neutral has only three producers, both-lost is unreachable (you cannot touch
-3.33 without crossing -1.62 first, which wins the short), and timeouts logged
at 1.0% of Neutral - so ~27% of ALL bars were being handed to the model as the
abstain class when a trade either way would have collected its target. The
cleanest positives in the sample, labelled "do not trade", while the fitted
confidence threshold was being asked to find selectivity in what was left.
Resolved by FIRST TOUCH: the target reached earlier is the trade that would
have closed first. Same forward window, no extra lookahead. Same-bar ties stay
Neutral - OHLC cannot order two touches, and unlike an intrabar stop tie there
is no pessimistic side to fall to, so a guess would inject a coin-flip
direction into the target.
Also:
- count both-won and its same-bar tie subset in the prebuild line, so the
share is measured rather than inferred from arithmetic on a log line
- scope the timeout counter to IS, matching the tally it is reported as a
percentage OF; it was incremented over the whole scan and divided by an
in-sample denominator
- clear m_lastBarrierTimedOut at the top of the walk with the excursions, not
at the bottom - the two early returns published the previous bar's verdict
- mark the pass-1 label line PROVISIONAL. It prints the enum fallback because
geometry can only be derived from excursions that do not exist yet, and it
reads exactly like a config change that failed to take effect
FORCES RETRAIN. Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:35:42 -04:00
//--- Bars where BOTH targets were reached, and the same-bar subset that could not be ordered. Counted
//--- rather than inferred: the share was first arrived at by subtracting timeouts from Neutral and
//--- reasoning that both-lost is unreachable, which is sound but is still arithmetic on a log line.
//--- These two numbers make the label composition readable directly, and the tie count is the one that
//--- says whether first-touch resolution is doing real work or just relabelling noise.
int m_labelPrebuildBothWonCount ;
int m_labelPrebuildBothWonTieCount ;
2026-07-14 22:36:27 -04:00
//--- safety valve: Train()'s do-while loop has no other bound on how many eras it will run
//--- before giving up, so a config that can't reach the convergence objective (e.g. too few
//--- swing-confirmed examples for the min recall bar to be reachable) would otherwise loop
2026-07-17 21:28:59 -04:00
//--- forever, permanently keeping the era-progress status label up instead of the normal per-tick
2026-07-22 13:33:56 -04:00
//--- info line and burning CPU nonstop. When the cap is hit, the operator is prompted (see
//--- PromptContinuePastEraCap): CONTINUE resets the era counter and keeps training; STOP deploys
//--- the best checkpoint found so far (FinalizeTrainRun) and terminates training. Headless
//--- (tester/optimizer) runs can't prompt, so they take the STOP branch automatically.
2026-07-14 22:36:27 -04:00
int m_maxErasPerRun ;
//--- Train() runs its per-bar loop synchronously, and MQL5 is single-threaded per chart - a
//--- multi-minute era would otherwise starve the terminal's chart-event queue for that whole
//--- stretch, including the control panel's own click/drag hit-testing (Panel\ControlPanel.mqh),
//--- which depends entirely on CHARTEVENT_MOUSE_MOVE being delivered promptly. Train() is
//--- therefore chunked: each call does at most ~TRAIN_TIME_BUDGET_MS of work, then returns and
//--- picks back up exactly where it left off on the next call (re-triggered via the same "New
//--- Bar" custom-event scheduling ScheduleTrainingIfNeeded() already used) - the members below
//--- are what makes that resumable across calls.
bool m_trainRunActive ; // true: a run (schedule -> convergence/stop) is in progress, possibly spanning many Train() calls
bool m_eraResumePending ; // true: yielded mid-bar-loop last call - resume the SAME era, don't start a new one
int m_resumeBars ;
int m_resumeTotalIter ;
int m_resumeOosCutoff ;
int m_resumeBarIndex ;
bool m_resumeAddLoop ;
2026-07-18 11:33:21 -04:00
//--- Pass 2 of the era loop: bar indices pass 1 (sequential feedForward/scoring) queued as
//--- IS-eligible for backProp, trained on in a freshly shuffled order instead of pass 1's own
//--- fixed chronological (oldest-to-newest) visitation order. Root cause this targets: Adam's
//--- momentum (b1, AI\Network.mqh) has an effective memory window of ~1/(1-b1) steps, which lands
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- close to the length of the contiguous same-class label runs the target produces (triple-barrier
//--- labels make those runs LONGER than the exact-pivot ones this was first measured against, since
//--- adjacent bars share most of their forward window and usually resolve the same way, so the
//--- argument holds a fortiori) - replaying those runs in the SAME order every single era let momentum lock onto
2026-07-18 11:33:21 -04:00
//--- whichever run it was currently passing through, with the network's end-of-era state
//--- disproportionately reflecting whatever it last walked through (recency), not a globally
//--- balanced fit. Symptom observed in practice: IS error climbing era-over-era (0.16->0.55+ over
//--- 25 eras on the same fixed dataset) instead of settling, and OOS Buy/Sell recall whipsawing
//--- between near-0% and 70-100% despite near-equal Buy/Sell label counts. Shuffling is the
//--- standard SGD fix - see the m_isPass2Active declaration below for how it's sequenced against
//--- pass 1 within Train()'s existing resumable-chunk machinery.
int m_isTrainQueue [ ] ;
int m_isTrainQueueCount ;
2026-07-18 23:03:51 -04:00
//--- Parallel to m_isTrainQueue (same index, kept in lockstep through the Fisher-Yates shuffle
2026-07-18 23:25:54 -04:00
//--- below) - the per-occurrence weight to apply when this queued slot is trained on in pass 2,
//--- decided once at queue time (see the queueing block's oversampling comment) rather than
//--- recomputed in pass 2. Currently always 1.0: class-balance correction is carried entirely by
//--- repCount (how many times a bar was duplicated into the queue), not by any per-occurrence
//--- weight scaling - see the queueing block's comment for why stacking a second correction here
//--- caused two separate training collapses. Kept as a real per-slot array rather than a literal
//--- 1.0 so a future, smaller/safer supplemental weight could be reintroduced without re-touching
//--- the queueing or shuffle code.
2026-07-18 23:03:51 -04:00
double m_isTrainQueueWeightScale [ ] ;
2026-07-25 19:34:24 -04:00
//--- Parallel to m_isTrainQueue too (same index, same lockstep swap in the shuffle): true on exactly
//--- ONE occurrence per duplicated bar - the rep-0 slot written at queue time. Its only job is to keep
//--- the reported IS accuracy (m_cumIsCorrect/m_cumIsTotal) measured on the NATURAL class distribution
//--- while backProp still trains on the oversampled one.
//--- Why: the queue duplicates each Buy/Sell bar up to repCount times (~21x at the observed 30.7:1
//--- imbalance), so counting every occurrence measured IS accuracy over a set that is ~58% directional,
//--- while the OOS counter measures the real ~6% directional distribution. Both excluded Neutral and
//--- both used the identical formula, so they LOOKED directly comparable - and reported things like
//--- "IS 77% / OOS 12%", which reads as catastrophic overfitting when the two numbers were simply
//--- scored against base rates an order of magnitude apart (77% vs a 58% baseline is a 1.33x lift;
//--- 12% vs a 6.1% baseline is 1.97x - the OOS side was actually the STRONGER one).
//--- Counting primaries only puts both metrics on the same footing, so the IS/OOS gap once again means
//--- what everyone reads it as meaning: generalisation. Training itself is completely unaffected -
//--- every occurrence still backprops exactly as before; this flag is read only by the counter.
bool m_isTrainQueuePrimary [ ] ;
2026-07-28 17:42:12 -04:00
//--- Pass 2's cursor into the (already shuffled) m_isTrainQueue - lets pass 2 itself yield/resume
//--- mid-queue under the same TRAIN_TIME_BUDGET_MS chunk budget pass 1 already yields under.
int m_isTrainCursor ;
//--- true: pass 1 (sequential) has finished for this era and pass 2 (shuffled backProp) is either
//--- running or has yielded mid-queue - Train() skips straight past pass 1's loop on resume when
//--- this is set. Reset to false only at a fresh era's start (never mid-run).
bool m_isPass2Active ;
//--- true: pass 2 has already run to natural completion for this era (m_isPass2Active's own
//--- false state is ambiguous between "not started yet" and "already finished" - both look
//--- identical to a plain `if(!m_isPass2Active)` check). Needed because pass 3 (OOS scoring) can
//--- itself yield/resume across multiple Train() calls same as passes 1/2 do; without this flag,
//--- every resume into an unfinished pass 3 fell through the `if(!m_isPass2Active)` guards on both
//--- pass 1 and pass 2 and re-ran the ENTIRE shuffled queue again (pass 1 itself was a no-op on
//--- resume since its own loop cursor `i` was already exhausted, but pass 2 re-shuffled and replayed
//--- from scratch every single time) - silently multiplying the predicted-class counts (and the
//--- extra, unintended backProp() calls) once per resume, for as long as pass 3 kept needing more
//--- than one chunk to finish. Reset to false only at a fresh era's start, alongside m_isPass2Active.
bool m_isPass2Done ;
//--- Pass 3: chronological, OOS-region-only re-walk that happens AFTER pass 2 has actually trained
//--- on this era's IS data - see m_isTrainQueue's declaration comment for why OOS scoring can no
//--- longer just happen inline during pass 1 (that would score every era's OOS window against
//--- weights from BEFORE this era's training, one full era stale - and for era 0 specifically,
//--- against the still-untrained cold-start network, which is why era 0's OOS recall used to show
2026-07-18 12:10:37 -04:00
//--- a meaningless 100% Neutral / 0% Buy / 0% Sell every time). Cursor walks i downward from
//--- m_oosScoreStartIndex to 0, mirroring pass 1's own iteration bounds/order for whichever bars
//--- satisfy isOOS - order matters here (unlike pass 2) since dOosForecast/dOosError are recursive
//--- EMAs over the visitation sequence, not order-independent.
bool m_isPass3Active ;
int m_oosScoreIndex ;
int m_oosScoreStartIndex ;
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Pass 2.5: the CALIBRATION walk. Sits between pass 2 (train) and pass 3 (grade) because that is
//--- the only position where the operating point can be fitted on data that is neither trained on nor
//--- graded - see DIR_CONF_CALIB_PCT_OF_IS for the measurement that forced it out of pass 2. Chunked
//--- and resumable on exactly the same pattern as passes 1-3 (cursor + active/done flags, both reset
//--- only at a fresh era's start), because at ~5.7k bars it will not finish inside one 120 ms budget.
//--- The band's bounds are recomputed from oosCutoff on every call rather than stashed in resume
//--- state: they are a pure function of (totalIter, oosCutoff, horizon), all three of which the resume
//--- path already restores, so deriving them cannot drift out of step with pass 1's queueing decision.
bool m_isCalibActive ;
bool m_isCalibDone ;
int m_calibIndex ;
int m_calibStartIndex ;
feat: excursion-size head (Stage 1, measurement only)
Direction is closed - normalised asymmetry fails on three instruments
with a working positive control, and the classifier's own best-of-999
era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is
a different question and RANGE clears at ~4x its null.
Checked the denomination before building on that, since the source memo
warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is
predictable" is a claim about travel RELATIVE to current ATR, not a
restatement of "ATR is autocorrelated". It is exactly the part a fixed
multiple (stop 3.31*ATR, target 1.64*ATR) discards.
A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches
ladder rung k) upward and downward. Survival parameterisation rather than
regressing the multiple, because it needs nothing new from CNet: sigmoid
outputs and the per-neuron delta the `total != 3` branch already applies
(a quantile head would need a linear activation and a pinball gradient in
Network.mqh, Network.cl and the DirectML path, on a class four topologies
share). Targets are free - m_ladderUpAt already records first-touch age
per rung with 0 meaning never reached.
Separate net, not extra outputs on the classifier: more outputs would
change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push
the count off 3 - the exact condition backProp uses to select the joint
softmax gradient the 3-class head depends on. The classifier is
bit-for-bit unaffected and this is removable without trace.
STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the
constant per-rung base rate - the baseline a fixed ATR multiple already
assumes - with both predictors fitted IS and evaluated OOS, so neither
gets a look at the test set. Positive skill justifies Stage 2 (drive
SL/TP and sizing off ExcursionQuantile, which is defined and deliberately
uncalled). Zero or negative means ATR already carries everything and
Stage 2 must not be built.
Trains only on primary occurrences: the replay queue oversamples for
CLASS balance, and a direction-balanced sample is a biased SIZE sample.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION-SIZE HEAD. A separate small CNet rather than extra outputs on the classifier: adding
//--- outputs would change m_outputNeuronsCount (branched on in a dozen places), the .nnw shape and
//--- the weights fingerprint, and would push the output count off 3 - which is the exact condition
//--- backProp uses to select the joint softmax gradient the 3-class head depends on. Kept separate,
//--- the classifier is bit-for-bit unaffected and this whole instrument is removable without trace.
//--- NOT PERSISTED in Stage 1: it is a measurement, it retrains within an era or two, and a second
//--- weights file is surface area that only earns its place once the skill score justifies Stage 2.
CNet * m_excNet ;
bool m_excHeadFailed ; // one-shot: creation failed, do not retry every bar
//--- Allocated once, reused every bar. getResults takes CArrayDouble*& and allocates when handed a
//--- NULL, so locals would mean an allocation per bar across ~32k bars an era.
CArrayDouble * m_excTgt ;
CArrayDouble * m_excOut ;
long m_excBaseHits [ 2 * BARRIER_LADDER_COUNT ] ;
long m_excBaseTotal ; // rows the base rates were estimated from
double m_excBrierHead [ 2 * BARRIER_LADDER_COUNT ] ;
double m_excBrierBase [ 2 * BARRIER_LADDER_COUNT ] ;
int m_excScored ; // held-out bars scored this era
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth
Three findings from the 2026-08-11 audit:
1. The excursion head's trailing-quantile ring was deliberately never cleared
between eras ("a rolling estimate of the market, not of the era") - but
pass 3 re-walks the SAME OOS window every era, so at each walk's restart
the ring still held the outcome masks of the newest OOS bars from the
previous walk: the chronological FUTURE of the bars about to be scored.
For the first ~window+horizon pushes of every era the "trailing" incumbent
was partly a leading one - conservative for the gate (an informed incumbent
is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts.
The ring now clears at era-score reset; the warm-up bars simply don't score
the trail race, which the m_excTrailN gating already accounts for.
2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage)
against the incumbent's subset sum - valid only if head skill is uniform
across the OOS walk, while the trail-scored subset systematically excludes
each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/
m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593
made every scored bar disjoint). The dead trio is replaced by
m_excBrierHeadT: the head's Brier accumulated only on the bars the warm
incumbent also scored, so the race now compares both predictors on an
identical bar set.
3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a
cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the
sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so
BufferTempData cached an all-zero Wyckoff block as a success for the whole
bar frame: the one path the f6150ee only-cache-successes rule cannot see,
because it never fails (the ba13eef class, arriving through values that
never fail; a resumed model's era-0 prebuild starts milliseconds after
OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means
async warm-up (transient reject, retried), while deep bars beyond the
buffered depth keep the sanitize loop's neutral-fill so degraded history
still trains. Also fixed m_featureCacheValid's declaration comment, which
still described the pre-f6150ee cached-miss semantics.
Compile: 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
//--- Since e2c9593 every scored bar IS a disjoint window (the score step strides by the horizon),
//--- so m_excBrierHead/m_excOosHits are already the disjoint tally and m_excScoredD just counts it.
//--- The old parallel *D arrays from the two-tally era were declared and reset but never
//--- accumulated - dead weight found by the 2026-08-11 audit and replaced by this one:
//--- the head's Brier accumulated ONLY on the bars the trailing incumbent also scored, so
//--- skillTrail compares the two predictors on the SAME bars instead of pro-rating the head's
//--- full-block sum by coverage (which assumed head skill is uniform across the OOS walk while
//--- the trail-scored subset systematically excludes the warm-up bars).
double m_excBrierHeadT [ 2 * BARRIER_LADDER_COUNT ] ;
fix: the excursion gate would have passed Stage 2 on an artifact I made
Second-opinion review killed the +4.2% far-rung result, correctly, and
the mechanism is my own bug. A head trained toward {0.05,0.9} converges
to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1,
POSITIVE where p < 1/3, growing monotonically as the rung gets farther.
Against a baseline frozen at the IS rate, an upward-biased head scores
positive Brier skill whenever the OOS rate merely sits above the IS rate.
Predicted signature: huge negatives near, ~zero at p=1/3, growing
positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs
were not the clean end of a distorted measurement, they were the other
face of the same artifact. Everything before 25aca83 is void.
The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4
topologies x N eras, reported per era - a best-of-~300 with no interval
and no multiplicity control, which is the shape of the four traps already
documented here. It now needs FOUR things at once:
DECISION RUNGS only the rungs ExcursionQuantile actually reads at the
live geometry (target 1.62, stop 3.31 ATR), fixed
before looking. Skill at 5 ATR is skill about a
distance no order is placed at - and the TARGET side
currently interpolates 1.5/2.0, which measured -2.2%
and -1.3%.
DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64
horizon bars, so ~16k scored bars is ~250 independent
ones and every SE over the full set is ~8x understated.
VS ORACLE the best constant achievable ON THE SCORED BLOCK,
closed form from H and n (Brier = H*(1-H/n)). A head
that learned only a LEVEL nearer the OOS rate than the
frozen IS constant scores positive against the old
baseline and <= 0 here. This is the control that
separates per-bar skill from base-rate drift.
MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing
constrained 8 independent sigmoids to obey that, and
ExcursionQuantile returns the FIRST crossing - so a
tangled curve is misread exactly where the head is
least sure. Counted and reported, not silently used.
The pass message now also states what a pass would and would not buy:
expectancy is -costs at zero directional edge whatever the stop distance,
and under prop DD limits LOWER variance also lowers P(reach target before
limit), so "better drawdown" is a choice of failure mode, not a win.
Still owed before any Stage 2: a race against a trailing-quantile
incumbent and a vol-feature logistic. Beating a frozen global constant is
the weakest admissible bar for replacing a global constant.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
int m_excScoredD ;
//--- OOS positives per rung. Feeds the ORACLE control: the best constant achievable ON THE SCORED
//--- BLOCK, in closed form. Separates "predicts per bar" from "learned a level nearer the OOS rate
//--- than the frozen IS constant", which scores positive while carrying no per-bar information.
long m_excOosHits [ 2 * BARRIER_LADDER_COUNT ] ;
//--- Bars whose predicted survival curve rose with distance. P(reach k) must be non-increasing in k;
//--- nothing constrains 8 independent sigmoids to obey that, and ExcursionQuantile reads the first
//--- crossing, so a tangled curve is misread exactly where the head is least certain.
int m_excMonoViol ;
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x
Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other"
30->337s). My estimate had been "single-digit percent". The cost is
per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is
19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k
forward/backward calls x several layer submits each, and its 760-wide
layer exceeds the CPU DLL's inline threshold so each one pays a real
handoff. The classifier's own net time tripled too, from contention with
a second pool on an already-full box.
Three changes, all backend-neutral because they remove submits rather
than tune threads:
SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar
of their horizon, so 16k consecutive bars were always ~250 independent
observations - the full-sample tally was never worth more than the
disjoint one, it just quoted an n that was ~64x too large. Dropping it
costs nothing statistically and removes 63 of every 64 forward passes.
The two parallel tallies collapse into one, which is also less code.
The trailing ring still advances on every bar: it needs the outcome
SEQUENCE, and that is array lookups, not a forward pass.
TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and
strongly autocorrelated - neighbouring bars carry near-identical
excursion information - so per-bar training buys resolution the target
does not have. Strided on ATTEMPTS, not acceptances, so a stretch of
unlabelled bars cannot silently change the spacing.
OWN TIMING COLUMN. The head's passes were landing in the era line's
"other" bucket, which is how a 3.6x regression read as an unexplained
jump in the one column nobody attributes. A cost that cannot be seen in
the timing line cannot be traded off against anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
long m_excTrainTick ; // stride counter, on attempts not acceptances
ulong m_excUs ; // head's own microseconds this era - see the era line
feat: race the excursion head against a trailing-quantile incumbent
Beating a frozen global constant is the weakest admissible bar for
replacing a global constant. The honest incumbent is a rolling rung
frequency: it adapts to the volatility regime - exactly what the head
claims to predict - and needs no model, no 760 inputs and no training.
Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one
ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon`
entries are held back UNRESOLVED: a bar's rung outcomes are only known
one horizon later, so using them would be lookahead and would flatter the
incumbent into an opponent the head could never fairly beat. Pass 3 walks
oldest-to-newest, so "pushed more than horizon bars ago" is exactly
"resolved by now". Each push is O(rungs), not O(window).
The head's decision-rung Brier is pro-rated to the trailing estimate's
coverage before the ratio, since the incumbent only scores bars where its
window is warm.
This line is worth reading on its own, independently of the head: if the
trailing quantile beats the global constant, that is a cheap risk-control
win available with no machine learning at all - and it is the same number
either way, so the run answers both questions in one pass.
The ring is deliberately NOT reset per era - it estimates the market, not
the era, and re-warming 500 bars every era would leave the incumbent
unusable over the first chunk of every scoring pass, handing the head a
free win on exactly those bars.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
//--- TRAILING CLIMATOLOGY (see EXCURSION_TRAIL_WINDOW). Ring of per-bar outcome bitmasks - 32 rungs
//--- fit one ulong, so the whole rolling history is one array of longs. Sized horizon + window: the
//--- newest `horizon` entries are NOT yet resolved (a bar's outcome is only known one horizon later,
//--- and using it would be lookahead), the next `window` entries are the rolling sample the estimate
//--- is taken over, and anything older falls out.
ulong m_excTrailRing [ ] ;
int m_excTrailHead ; // next write position
int m_excTrailCount ; // entries pushed so far, capped at the ring size
long m_excTrailHits [ 2 * BARRIER_LADDER_COUNT ] ;
long m_excTrailN ; // resolved bars currently inside the window
double m_excBrierTrail [ 2 * BARRIER_LADDER_COUNT ] ;
long m_excTrailScored ; // bars scored while the trailing estimate was usable
2026-07-14 22:36:27 -04:00
datetime m_lastBarTime ;
2026-07-17 23:21:12 -04:00
//--- This model's own learning-rate trajectory. `eta` (AI\Network.mqh) is a single file-scope
//--- global shared by every CNet in the process - PAI/CONV/LSTM each train independently and
//--- asynchronously (their own Train() calls are interleaved via separate chart-event/timer
//--- scheduling, not synchronized), but all three previously read AND wrote that one global, so
//--- one model's era-end regression/recovery decay/recovery of `eta` (see Train()'s era-end
//--- block) silently changed the learning rate the OTHER two models' very next backProp() call
//--- used too - an unintended coupling between what's supposed to be three independent training
//--- trajectories. Train() now restores `eta` from this member right before touching it, and
//--- saves it back before returning (both at the mid-chunk yield and at the natural end of the
//--- call) - MQL5 gives one chart's EA a single execution thread and each Train() call runs to
//--- completion or to its own yield point before another event is dispatched, so this save/
//--- restore is enough to isolate each model's schedule with no change needed to the neuron-level
//--- or OpenCL/DirectML call signatures that read the global directly.
double m_modelEta ;
2026-07-18 14:56:41 -04:00
//--- Ceiling the era-end recovery bump (Train()'s isBetterEra block) restores `eta` toward - used
//--- to be the raw `lr` constant unconditionally, which is only correct for ADAM. SGD's own starting
//--- rate is the separate SgdLearningRate input (AI\Network.mqh) - clamping SGD's recovery bump to
//--- plain `lr` (AdamLearningRate) would silently cap it back down to Adam's ceiling after the
//--- first regression+recovery cycle, undoing the whole point of giving SGD its own configured
//--- rate. Computed once at construction from this model's own m_optimizationAlgo, matching
//--- m_modelEta's per-instance isolation (see that member's comment).
double m_etaCeiling ;
2026-07-14 22:36:27 -04:00
int m_erasSinceCooldown ; // eras completed since the last cooldown reset - replaces the old per-call-only "erasThisCall"
CArrayDouble m_oosWindow ; // run-scoped OOS stability window (used to be a Train()-local CArrayDouble)
double m_bestOosForecast ;
2026-07-21 00:03:45 -04:00
//--- Balanced accuracy (macro-recall: mean of Buy/Sell/Neutral OOS recall) of the era the current
//--- checkpoint was taken from. This is the metric the checkpoint SELECTION ranks on now, in place
//--- of the blended dOosForecast - blended accuracy is dominated by Neutral (~96% of bars), so among
//--- otherwise-acceptable eras it silently preferred the MOST Neutral-leaning weights, deploying a
//--- model that scores well on paper but under-calls Buy/Sell. Balanced accuracy weights every class
//--- equally, so "best" now means "most accurate across all three classes" - exactly what a 3-class
//--- signal product wants. Kept SEPARATE from m_bestOosForecast (which still snapshots the blended
//--- value at the same checkpoint) because FinalizeTrainRun()/the restore-on-regression branch reset
//--- dOosForecast to m_bestOosForecast, and that must stay the blended EMA the rest of the machinery
//--- expects. The directional recall FLOOR (directionalRecallOK) still gates convergence unchanged;
//--- this only fixes which era gets deployed among the candidates. -1 until the first ranked era.
double m_bestBalancedOos ;
2026-07-15 21:47:37 -04:00
//--- whether the era m_bestOosForecast/the checkpoint was taken from also cleared the per-class
//--- directional recall floor (see directionalRecallOK below) - part of the "best" ranking itself,
//--- not just a side note, so blended accuracy alone can never outrank a directionally-useful era
//--- (see the checkpoint/eta-decay comment in Train()'s era-end block for why that matters).
bool m_bestPassedRecall ;
fix: a one-sided era can no longer become the best checkpoint
Measured on HYBRID, era 29 of the first win-scored run: the model collapsed
to always-Buy and was crowned "new best selection score 67.1%". Under
win-based scoring that is not a coincidence - the always-call-the-drift-side
model IS the chance reference, so it scores exactly chance (P(winLong) ~ 67%
on SP500), while every honest two-sided era scores 63-66% because shorts win
less often against the drift. Raw score ranking therefore actively prefers
the degenerate model, every regression restores back to it, and live NMS
collapses its near-constant signal to ~25 trades per era - observed as
"hybrid barely trades".
bothSidesLive already blocked one-sided eras from DEPLOYING (tradeableOK,
371f8aa), but among not-yet-deployable eras the score alone ranked - the same
early phase the coverage credit was added for, failing the same way through a
different door.
The ranking key is now three lexicographic tiers: deployable > two-sided >
score. A one-sided era cannot displace a two-sided best regardless of score -
by construction its score is a property of the data's drift, not the model -
and a two-sided era displaces a one-sided best no matter how much lower it
scores. m_bestBothSidesLive is snapshotted with the checkpoint and reset with
the rest of the best-tracking state.
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 21:19:44 -04:00
//--- Was the checkpointed era calling BOTH directions? Middle tier of the ranking key - see
//--- isBetterEra. Exists because of a measured failure (2026-08-09, HYBRID era 29): under win-based
//--- scoring an always-Buy model scores EXACTLY chance (P(winLong), ~67% on drifting SP500) while
//--- honest two-sided eras score 63-66% (shorts win less often against the drift), so raw score
//--- ranking crowned the degenerate model, every regression restored back to it, and live NMS
//--- collapsed its constant signal to ~25 trades per era ("barely trades"). bothSidesLive already
//--- blocked it from DEPLOYING via tradeableOK, but among not-yet-deployable eras the score alone
//--- decided - the same phase the coverage credit exists for.
bool m_bestBothSidesLive ;
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint
The 23:42 restart left all four charts grinding ~25x slower than the 18:01
baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and
NOTHING could say why from outside: pass 1 logs nothing, its status paint sat
inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed
FIRST - painted nothing either, the VPS has no debugger for a thread stack,
and the hourly new-bar cache invalidation cancels and restarts an unfinished
era, so a slow era can stay invisible FOREVER. Externals gave: four chart
threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That
narrows it to "MQL5-side per-item work in the era passes" and no further.
So training now explains itself:
- TrainHeartbeat: one line per 4096 processed items, only after an era has
already run 60s, at most 6 lines per era - a healthy era stays exactly as
quiet as before. Reports position and the cumulative split: feature-window
builds vs net forward/backprop vs everything else. Hooked into all three
passes.
- The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back
Y, other Z)" whenever an era exceeded 120s.
- Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so
the panel shows "learning (era N)" instead of sitting on the idle writer's
"Getting ready..." for the entire IS sweep. The label is throttled
internally; painting per bar costs nothing.
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
//--- SLOW-ERA HEARTBEAT (2026-08-10). The era passes are silent by construction - pass 1 logs
//--- nothing and (until tonight) painted nothing for IS bars, pass 2/3 only paint - so when a
//--- restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86s),
//--- the outside view was "Getting ready...", four pegged cores, and an empty log for 20+ minutes,
//--- with the hourly new-bar cache invalidation then cancelling and restarting the unfinished era
//--- forever. Nothing external (thread stacks need a debugger the VPS lacks) can say WHERE the time
//--- goes, so training itself must: cumulative per-era timers around the two candidate costs
//--- (feature-window builds, net forward/backprop) and a heartbeat line that only speaks when an
//--- era is genuinely slow - a healthy run stays exactly as quiet as before.
uint m_eraStartTick ;
ulong m_passFeatUs ; // cumulative BuildFeatureWindow time this era, microseconds
ulong m_passNetUs ; // cumulative feedForward/backProp time this era, microseconds
int m_passHeartbeatPrints ;
2026-08-10 07:44:03 -04:00
uint m_lastHeartbeatTick ;
diag: an era that discards itself now says so instead of scanning forever
add_loop is exactly "at least one bar produced a usable feature
window". When it stays false, pass 2, pass 3, the era counter, the
checkpoint and every log line in the era-end block are ALL skipped:
Train() returns having done nothing, m_eraResumePending is still false,
and the next call restarts the SAME era from bar 0. That is an
infinite 0->100% "scan" loop that prints absolutely nothing - the only
remaining silent restart path in Train(), and it matches the reported
symptom exactly.
Pass 1 now counts usable vs unusable windows and reports at the pass
boundary, which demonstrably executes:
- total failure routes through ReportTrainStall (already capped at
one line a minute, and carries the run-state flags) naming the
counts, the required window width and the bar count
- success prints how long the scan took and how many samples it
handed to pass 2, but only once the era has passed 10s - a fast
era stays as quiet as before, a slow one distinguishes "advancing"
from "sweeping the same bars forever"
A PARTIAL failure is normal and deliberately does not shout: pass 1
walks oldest-to-newest and the deepest bars predate the indicators'
warm-up, so those windows fail and are cached as misses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
//--- How many of pass 1's bars produced a usable feature window, and how many did not. These decide
//--- whether the era does ANY work: add_loop is just "m_passWindowOk > 0", and when it is false
//--- pass 2, pass 3, the era counter and the checkpoint are ALL skipped, so Train() returns having
//--- done nothing and the next call restarts the same era from bar 0 - an infinite, wholly silent
//--- 0->100% scan loop. Counting both sides makes a partial failure (indicators unavailable over
//--- the oldest bars, which is normal and harmless) distinguishable from a total one.
int m_passWindowOk ;
int m_passWindowFail ;
fix: the recall gate was unsatisfiable and the LR decay was a spiral
Both made the run structurally unable to succeed, independently of any
signal in the data. Found by reading the 13:01 log.
RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall
each >= 40%. First-touch resolution (ce52654) collapsed Neutral from
the ~94% majority it was under exact-pivot labels to a same-bar-tie
residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model
to identify 40% of coin-flip ties before it could converge. Measured:
CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on
every era. No model could ever satisfy it; every run was destined for
the plateau ladder or the era cap.
Only the DIRECTIONAL floors are load-bearing for the anti-collapse job
the gate exists to do: an all-Neutral model shows Buy and Sell recall
at 0% and is blocked by them. Neutral's own floor guarded the mirror
bias (over-calling Buy/Sell at Neutral's expense), which was real at
94% prevalence and is not at 0.65% - there, almost never calling
Neutral is correct rather than biased.
Prevalence-guarded rather than hardcoded off, so it returns by itself
if a future label rule makes Neutral substantial again. Deliberately
NOT extended to Buy/Sell: exempting a thin directional class reopens
the era-44-46 hole, which directionalRecallMeasured only half-covers -
it checks those classes were MEASURED, not that they passed.
ETA DECAY. A regressing era restored the checkpoint, reset the
optimizer and cut eta - all on the FIRST regression. The next era then
started from an identical state with a smaller step, regressed again,
and got the same treatment. The loop is self-sustaining and cannot
discover anything, because rolling the weights back is exactly what
removes the exploration that would end it.
Measured on PAI: eras 2-11 every one a regression against era 1, eta
0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras,
~45s each, reproducing era 1 exactly and unable to do anything else.
Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the
standard ReduceLROnPlateau formulation. A single bad era is noise, and
an improving era clears the counter so alternating runs never
accumulate into a decay.
Build tag -> gate-patience-v3. It had not moved in six commits, which
is why the running binary could not be identified from its own log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- Consecutive regressing eras since the last new best - the patience counter for the checkpoint
//--- restore / eta decay (see ETA_DECAY_PATIENCE_ERAS). Reset by any era that improves.
int m_consecutiveRegressions ;
2026-08-10 07:22:29 -04:00
void TrainHeartbeat ( const string tag , int done , int total , const string shortLabel ) ;
//--- Progress of the pass currently running, and its name, for the simple panel. Published by each
//--- pass through TrainHeartbeat rather than derived in the UI, because the UI cannot know which
//--- pass owns the counters it can see - deriving it from pass 2's cursor is what made an entire
//--- pass-1 scan display "100%".
int m_passProgressPct ;
string m_passLabel ;
fix: prebuild and era sized different windows; diag: Train() names its branch
TWO things, one incident.
1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised
to 0, and NEVER ASSIGNED - the assignment existed before the God-class split
and the split dropped it, leaving a dead member. Harmless while nothing read
it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset
dtStudied from it. Train() then computed the window as
max(StartTrainBar, floor) while the prebuild computed max(0, floor), where
StartTrainBar is the non-zero datetime OnChartEventHandler passes through from
the "New Bar" event. The two therefore disagreed about `bars`, so
EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches,
and re-armed a full 38k-bar prebuild - instead of training. Restored the
assignment so both sides evaluate the identical expression.
2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six
early-return branches above the era loop and every one of them is silent. Four
charts burned a core each for 15 minutes with an empty journal: the pass
heartbeats (694b756) proved the era loop was never reached, no prebuild
completion line appeared either, and nothing external can see inside a single
MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS
has no debugger. That is an undiagnosable state, and it is the thing to fix,
not just the bug of the day.
ReportTrainStall() now names the branch Train() is taking whenever no era has
completed for 3 minutes, at most once a minute per signal, with the state that
decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and -
for the cache-invalidation branch specifically - BOTH bar counts, since two
sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a
healthy run: an era completing resets the clock.
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
//--- STALL REPORTER. Train() is a state machine with several early-return branches ABOVE the era
//--- loop (OOS simulation walk, label prebuild, history sync, warm-up, cache invalidation), and
//--- every one of them is silent. On 2026-08-10 four charts burned a core each for 15 minutes with
//--- no journal output at all: the pass heartbeats proved the era loop was never reached, and no
//--- prebuild completion line appeared either, so the work was in a branch that cannot say its own
//--- name. Externals (per-thread CPU, file writes) cannot see inside one MQL5 thread, and the VPS
//--- has no debugger - so the state machine has to report itself. m_lastEraCompleteTick is set at
//--- every era end; when Train() is entered and that is stale, ReportTrainStall() names the branch
//--- it is about to take, once per STALL_REPORT_INTERVAL.
uint m_lastEraCompleteTick ;
uint m_lastStallReportTick ;
void ReportTrainStall ( const string branch ) ;
2026-07-14 22:36:27 -04:00
bool m_haveOosCheckpoint ;
bool m_oosStable ;
bool m_objectiveMet ;
feat: gate deployment on the null of the MAXIMUM, not the per-era null
EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM
over every era a run ranks. A 2-sigma one-sided test passes on noise with
probability 0.0228 per era, so over N eras the chance at least one clears
it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The
gate was near-certain to open on a long run whatever the data held.
It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance -
+1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the
call counts these runs produce that is p_family 0.92..0.9999.
Every OTHER best-of-N decision here already carries this correction, and
every one REJECTS on this data: the barrier-geometry winner (null of the
maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI
lag profile (null of the maximum over 21 lags). The one decision that
ships a model to a live account had none.
BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to
deploy:
z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n)
p_single = P(Z >= z)
p_family = 1 - (1-p_single)^N
against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN
snapshotted precision/chance/call-count, not the latest era's, because
the model that ships is the one that has to clear the bar.
N counts CANDIDATE eras (coverage measurable, at least one directional
call) - an era that called nothing directional could never have become
the best, so counting it would make the gate stricter than the search
that actually happened.
Conservative on purpose: consecutive eras share OOS bars and differ by
one gradient step, so they are nowhere near N independent draws and the
true family-wise error is below this bound. This gate decides what trades
real money and the house posture is reject-unless-demonstrated.
Effect at 2900 directional calls / N=112: required edge goes 1.76pp ->
2.92pp. A real edge clears it; +1.5pp does not.
Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and
the m_trainingComplete assignment - which must stay identical or the flag
persisted into the .nnw disagrees with the decision to stop, and a reload
runs inference on a model the ladder refused.
NOT applied to the two operator paths (era-cap deploy, panel Deploy
button). Those stay the operator's call; ReportSelectionGateVerdict()
logs the verdict beside them so an authorised deploy can never later be
misread as a validated one.
NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather
than pulling in Math\Stat. Verified against reference values to 6dp:
Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are
ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1".
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- RAW inputs to the family-wise deployment gate, snapshotted at the same instant as the checkpoint
//--- so the test re-runs on the era that will actually ship rather than on whatever the latest era
//--- happened to score. m_bestBalancedOos alone cannot serve: it is precision already multiplied by
//--- the coverage credit, and the significance test needs the unweighted precision, the chance rate it
//--- is measured against, and the call count that sets its standard error. -1 until the first ranked era.
double m_bestDirPrecPct ;
double m_bestChancePrecPct ;
int m_bestDirCalls ;
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade
NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject().
It never touched dPrevSignal, and dPrevSignal is what LongCondition() /
ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its
arrow and still opened a position.
Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars,
so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows
were drawn. Roughly one arrow per eight positions the EA would take.
And the survivors are not a random eighth. Rule 2 of the declustering
keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is
systematically the best member of each run. A chart showing the best of
every eight decisions and hiding the rest reads far better than the model
is - the same best-of-N selection error already corrected in the geometry
scan, the indicator tuner, the lag profile and the deploy gate, this time
on the display layer, where it is most likely to mislead the person
deciding whether to trade.
Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a
"may trade" flag consulted at each read site: that leaves exactly ONE
definition of what the model decided this bar, so the arrow, the panel's
"Current signal", the confidence feeding sizing/SL/TP/trailing, the
refresh tally and the order itself cannot drift apart again.
Also reports the consequence instead of hiding it. Every OOS counter on
the era line still scores every directional call - a population ~8x larger
than what now trades - so the line carries a second figure:
| TRADED (declustered) NN% on N calls (edge +Npp)
replaying the identical rule over pass 3 (which walks OOS bars oldest to
newest, the same order the live sweep sees). Its cursors are separate
members from the live ones so a training pass can never disturb the live
chart's declustering.
Deliberately NOT switched into selectionScore yet. Declustering cuts
coverage from ~64% of bars to ~8%, well under
MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint
undeployable overnight - the minRR collision and the recall-floor catch-22
twice over. The floor gets re-derived from these measurements first.
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
//--- DECLUSTERED OOS tally: the calls that survive NMS, i.e. the ones that actually become positions
//--- now that live NMS gates the trade (see RefreshLatestSignal). Every other OOS counter here scores
//--- a population ~8x larger than the EA trades, so this pair is what "would I have made money" reads.
//--- Era-scoped, reset with the rest of the OOS counters. The three *Nms* cursors are the replay state
//--- for the same rule PruneDirectionalClusters and NmsLiveAccept use - kept separate from the LIVE
//--- cursors so a training pass can never disturb the live chart's declustering.
int m_oosNmsFired ;
int m_oosNmsHits ;
int m_oosNmsLastBuyIdx ;
int m_oosNmsLastSellIdx ;
int m_oosNmsKeptIdx ;
double m_oosNmsKeptConf ;
ENUM_SIGNAL m_oosNmsKeptDir ;
feat: gate deployment on the null of the MAXIMUM, not the per-era null
EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM
over every era a run ranks. A 2-sigma one-sided test passes on noise with
probability 0.0228 per era, so over N eras the chance at least one clears
it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The
gate was near-certain to open on a long run whatever the data held.
It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance -
+1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the
call counts these runs produce that is p_family 0.92..0.9999.
Every OTHER best-of-N decision here already carries this correction, and
every one REJECTS on this data: the barrier-geometry winner (null of the
maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI
lag profile (null of the maximum over 21 lags). The one decision that
ships a model to a live account had none.
BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to
deploy:
z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n)
p_single = P(Z >= z)
p_family = 1 - (1-p_single)^N
against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN
snapshotted precision/chance/call-count, not the latest era's, because
the model that ships is the one that has to clear the bar.
N counts CANDIDATE eras (coverage measurable, at least one directional
call) - an era that called nothing directional could never have become
the best, so counting it would make the gate stricter than the search
that actually happened.
Conservative on purpose: consecutive eras share OOS bars and differ by
one gradient step, so they are nowhere near N independent draws and the
true family-wise error is below this bound. This gate decides what trades
real money and the house posture is reject-unless-demonstrated.
Effect at 2900 directional calls / N=112: required edge goes 1.76pp ->
2.92pp. A real edge clears it; +1.5pp does not.
Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and
the m_trainingComplete assignment - which must stay identical or the flag
persisted into the .nnw disagrees with the decision to stop, and a reload
runs inference on a model the ladder refused.
NOT applied to the two operator paths (era-cap deploy, panel Deploy
button). Those stay the operator's call; ReportSelectionGateVerdict()
logs the verdict beside them so an authorised deploy can never later be
misread as a validated one.
NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather
than pulling in Math\Stat. Verified against reference values to 6dp:
Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are
ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1".
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- How many eras the maximum was taken over - the N in the Sidak correction. Counts CANDIDATE eras
//--- (coverage measurable and at least one directional call), not every era, because an era that
//--- called nothing directional was never in the running to become the best and must not inflate N.
//--- Run-scoped: reset with the rest of the best-checkpoint tracking at the top of a fresh run.
int m_deployCandidateEras ;
//--- Upper-tail standard normal, Q(z) = P(Z >= z). Abramowitz & Stegun 26.2.17, |error| < 7.5e-8 -
//--- self-contained rather than pulling in MQL5's Math\Stat, which this project has never included and
//--- which drags a chain of headers behind it for the sake of one function.
double NormalUpperTail ( double z ) ;
//--- THE GATE. Re-tests the checkpoint that is about to deploy against the null of the MAXIMUM over
//--- m_deployCandidateEras eras, and reports the pieces so the log can show its working. See
//--- DEPLOY_FAMILY_WISE_ALPHA. Returns false (refuse) whenever the inputs are missing.
bool BestCheckpointSurvivesSelection ( double & zObs , double & pFamily , int & nTried ) ;
//--- Logs that verdict WITHOUT enforcing it, for the two deploy paths that are explicit operator
//--- decisions (the era cap and the panel's Deploy button). Those stay the operator's call; this just
//--- makes sure the log never lets an authorised deploy read as a validated one.
void ReportSelectionGateVerdict ( string context ) ;
2026-07-25 15:55:56 -04:00
//--- Plateau ladder state (see the PLATEAU_* constants). m_erasSinceBestBalanced counts eras since
//--- the last NEW BEST balanced accuracy; m_plateauStage is how far up the escalation it has climbed.
//--- Both are run-scoped and deliberately NOT persisted, matching m_modelEta/m_bestBalancedOos, which
//--- also restart fresh - a resumed run re-earns its patience rather than resuming mid-escalation.
int m_erasSinceBestBalanced ;
int m_plateauStage ;
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it
Points 3 and 4 of the four-point plan.
1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute.
The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it
fires, every one of those eras has been evaluated out of sample, so all of them sit
in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training
longer therefore does not merely cost time - it RAISES the bar the eventual winner
has to clear.
The new stop reads the TRAINING error, which the gate never looks at. When the
optimiser has stopped improving on data it can see, more eras will not find a better
model; they will only enlarge the OOS family. Ending there shrinks the correction,
and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted
an out-of-sample number.
That distinction is the whole point and it is the one this project has got wrong four
times: stop on IS and the family really is smaller; stop on OOS and those eras were
searched and still count. Both stops now exist; only this one buys a lower bar.
Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training
error is noisy per era - mini-batch order alone moves it - and ending a run that is
still learning costs far more than a few wasted eras. Improvement is RELATIVE
(IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and
it only acts when a checkpoint exists, since otherwise it would end a run with
nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot
early-stop on its first era against a previous run's best.
2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER.
The family-wise permutation gate already establishes that the RANKING is not noise.
It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is
biased upward by construction, being the largest of K noisy draws. The adoption
message quotes that raw maximum and compares it against the incumbent, so the number
a reader plans on is the inflated one.
The penalty is now measured, not assumed: the same permutation draws that produce the
p-value also produce, per draw, the MAXIMUM excess across all candidates under pure
noise. The mean of those maxima is exactly what a best-of-K selection is expected to
report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE
penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88),
and it needs no normality assumption because the draws ARE the null distribution.
Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large
effect is nearly untouched and a marginal one collapses toward zero.
Reported, not gated. The adoption decision still turns on the permutation p-value,
which is the right test for "is the ranking real"; the shrunk number is there so the
magnitude quoted beside it is one worth planning on. Closes the first of the two
EdgeFinder ports identified on 2026-08-12.
NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement
steer the target" is already true where it matters most - ReportGeometryExpectancyScan
ADOPTS the winning barrier geometry under the family-wise gate rather than advising
it, and the MI excursion suite publishes a verdict per instrument per config. What is
still missing is steering the TRAINING TARGET itself (direction vs excursion) off
those verdicts, and that is a design change rather than a surgical one - direction is
a closed verdict while excursion SIZE keeps clearing, so the honest version of that
change is a target-selection policy, not a flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
//--- IN-SAMPLE early-stop state (see IS_ERROR_IMPROVE_FRAC). Best training error seen this run and
//--- eras since it last improved. -1 = nothing measured yet.
double m_bestIsError ;
int m_erasSinceBestIsError ;
fix(plateau): the IS-error early stop was inert for every ensemble member
15 hours of training, and the stop that exists to END a run announced itself
1,299 consecutive times without ending anything:
SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299
eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398
SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras
XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras
CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors
the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` -
on EVERY era, purely so each member's status line shows the collective stage. A
display mirror was silently overwriting a decision, so the stop re-armed and
re-fired the next era, forever.
This is the worst possible direction for this particular bug. Every one of those
1,299 eras was scored out of sample and joined the family the deploy gate
corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make
that family SMALLER; instead the run spent fifteen hours raising its own bar.
- m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run.
Nothing in the ladder may reset it. The stop condition and the two solo deploy
conditions read the latch, not the mirrored stage.
- The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across
participating members (same participation test the era barrier uses, so an
excluded or finished member cannot veto). One member still learning can still
move the combined vote, and the vote is what the gate certifies.
- Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage.
The block that actually ends the run sits under `dueStage > g_ensPlateauStage`,
so assigning the stage directly makes that test false and the deploy never
happens - the same inert-write shape as the bug being fixed. Caught before
committing; raising dueStage carries it through the ladder's own path (warm
restarts skipped, family-wise vote test, measurement screen, joint checkpoint)
unchanged.
- g_ensIsPlateauAnnounced: announce once per run, not once per era.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
//--- LATCHES when the IN-SAMPLE error stops improving, and it is a separate flag from m_plateauStage
//--- for one measured reason: EnsembleEraVerdict mirrors the shared ladder onto every member with
//--- `mm.m_plateauStage = g_ensPlateauStage` on EVERY era, purely so each member's status line reads
//--- the collective stage. That display mirror was overwriting a DECISION. The IS stop set
//--- m_plateauStage = PLATEAU_STAGE_DEPLOY, the next verdict stomped it back, and the stop re-fired
//--- the era after - measured 2026-08-18: SP500 ConvLSTM printed IN-SAMPLE ERROR PLATEAU for 1,299
//--- consecutive eras and reached era 1,398 without ever ending. That is the exact opposite of what
//--- the stop is for: every one of those eras was scored out of sample and joined the family the
//--- deploy gate corrects over, so the run spent fifteen hours RAISING its own bar.
//--- One-way: set once, cleared only by a fresh run. Nothing in the ladder may reset it.
bool m_isErrorPlateaued ;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric
Four of the six findings from research/training_pipeline_audit_2026-08-09.md
(F4 mini-batching and F6 feature re-encode deliberately deferred - see the
report's implementation-status section for why):
- F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%,
which is 15-bit - provably non-uniform on every full-history era over 32,768
queued samples. New 30-bit ShuffleRandomIndex().
- F2: plateau warm restarts were a no-op whenever eta already sat at its
ceiling (the normal state of a non-regressing plateau) - the ladder was just
a 24-era countdown. Restarts now overshoot to 5x the ceiling
(PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience
window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has
real range.
- F3: checkpoint restores put weights back but kept the rejected trajectory's
Adam moments, so the optimizer immediately pushed back toward the rolled-back
state (the restore->regress->restore oscillation). CNet::ResetOptimizerState()
zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta
untouched) on every mid-run restore, every boosted restart, and the
deploy-time restore that online learning continues from.
- F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk,
so the selection metric the checkpoint ranking and deploy gate read is a pure
function of the checkpoint instead of partly measuring BN drift. Defensive
unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and
the OOS continual-learning simulation stay adaptive by design.
Compiled clean (0 errors, 0 warnings) via the staged-tree recipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Eras remaining in the current warm-restart boost window (see PLATEAU_RESTART_BOOST): set to
//--- PLATEAU_PATIENCE_ERAS by each boosted restart, decremented by the era-end anneal that walks eta
//--- back to the ceiling, cleared by any new best. Run-scoped and not persisted, same as the ladder
//--- state above.
int m_restartBoostErasLeft ;
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- m_focalGammaRuntime removed 2026-07-31 with focal loss itself - see the removal note at the
//--- former m_focalGamma above. The plateau ladder keeps its learning-rate warm restart, which was
//--- always the actual escape; the gamma anneal beside it stepped monotonically to zero anyway.
2026-07-14 22:36:27 -04:00
uint m_syncWaitStartTick ; // 0 = not waiting on history sync; else GetTickCount() when the wait began
//--- 3 no-op passes on a fresh start (see InitNeuralNetwork()/ResetWeights()), each its own separately-
//--- scheduled Train() call (not a tight in-process loop), so the broker/terminal's history sync gets
//--- several real, wall-clock-separated chances to finish before the era loop commits to a bar count.
int m_warmupPassesRemaining ;
//--- fractal/swing-confirmation/trend-context Buy/Sell label cache: the label at a given now-relative
//--- bar index only depends on price/ATR history, never on model state, so recomputing it every era
//--- (as opposed to once per real bar close) is pure waste. Rebuilt fresh whenever `bars` changes
//--- (see the invalidation check in Train()) rather than incrementally appended, since MQL5 timeseries
//--- indices are relative to "now" and shift by one on every new candle - a full rebuild on change
//--- sidesteps needing datetime-keyed/incremental bookkeeping entirely.
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- Filled in the same pass as the label caches below and gated by the SAME m_labelCacheHasValue, so
//--- a bar either has both or neither and no third validity flag can drift out of step with them.
double m_excUpCache [ ] ;
double m_excDownCache [ ] ;
feat: first-passage ladder + expectancy scan - price every geometry, not just the chosen one
Corrects the premise of the previous plan. Break-even is NOT a ceiling.
If the model shifts the win probability on the bars it selects from
p0 = m/(m+k) to p0 + d, then
EV = (p0+d)*k - (1-p0-d)*m = d*(k+m)
because p0*k - (1-p0)*m is zero by construction. The stop:target RATIO
is expectancy-neutral - a punishing break-even is exactly repaid by the
payoff - and only the real edge d and the TOTAL WIDTH (k+m) move EV.
Width matters because the spread is charged once per trade however wide
the barriers are, so a narrow barrier spends much of its own range on
costs. DeriveBarrierGeometry's own comment already said the ratio buys
nothing; the objective just never followed from it.
Blocker this had to solve first: m_excUpCache/m_excDownCache hold only
MAXIMUM travel each way, and a maximum cannot say which side was
reached FIRST - so any geometry other than the walked one was
undecidable on precisely the bars where both barriers were touched,
~28% of the sample.
- BARRIER_LADDER: per bar, the first-touch AGE for 8 travel distances
in each direction, filled during the walk the labels already run.
Cursors keep it O(1) amortised per walked bar rather than 16
comparisons. Levels are travel FROM ENTRY, not barrier prices, so one
ladder serves both directions and the spread is applied analytically
when a level converts back to an SL/TP multiple - storing prices
would need four ladders and bake today's spread into the cache.
Sized, invalidated and validity-gated with the label caches.
- ReportGeometryExpectancyScan: every ladder pair priced exactly off
that cache - width in ATR and in SPREADS (cost efficiency, knowable
without knowing d), break-even, both base rates, the share of bars
resolved inside the horizon, and EV per unit of edge. Compares the
widest resolvable pair against the quantile rule's pick.
MEASUREMENT ONLY - the quantile rule still chooses. Nothing here can
measure d, and width buys nothing if the wider target is less
predictable. Base rates are printed beside each break-even because a
persistent gap is DRIFT and must not be credited to the model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 12:59:18 -04:00
//--- First-passage ladder, flat (idx * BARRIER_LADDER_COUNT + level). Value = bars AFTER the entry
//--- bar at which travel first reached BARRIER_LADDER[level] in that direction; 0 = never within the
//--- horizon. Sized and invalidated with the label caches and gated by the same m_labelCacheHasValue,
//--- so a bar has all of it or none. See BARRIER_LADDER for why this exists and why the levels are
//--- travel-from-entry rather than barrier prices.
int m_ladderUpAt [ ] ;
int m_ladderDownAt [ ] ;
//--- Scratch for the bar TripleBarrierLabel is currently walking, published the same way m_lastExcUp
//--- is and copied into the caches by AdvanceBarrierLabelState under the label's validity flag.
int m_lastLadderUpAt [ BARRIER_LADDER_COUNT ] ;
int m_lastLadderDownAt [ BARRIER_LADDER_COUNT ] ;
//--- Reports expectancy for every ladder pair - see the definition. Measurement only; it does not
//--- (yet) choose the geometry.
void ReportGeometryExpectancyScan ( void ) ;
2026-07-14 22:36:27 -04:00
bool m_labelCacheBuy [ ] ;
bool m_labelCacheSell [ ] ;
fix: the deploy gate was benchmarking a win rate against a label frequency
The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a
driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade
is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test.
That invariant needs reward >= risk, and the measured geometry no longer
satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but
both-won bars were stripped out of Buy and Sell so the label base rate read
37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing
models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma
against 37.5% and loses money on every single trade. Live since 217b9bc.
Root cause is that label agreement stopped being the same question as trade
profitability. Buy implies winLong, but the converse fails on every both-won
bar, and the label can only name one of two directions that both pay.
So stop asking the model whether it matched a label and start asking whether
its trade paid:
- cache winLong/winShort per bar beside the label, under the same validity
flag; published from the barrier walk before the collapse to 3 classes
- dirPrecPct now counts wins on the side actually called
- chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook
m/(m+k) would credit SP500's drift to the model
- the NMS "what would I have made" pair, the live-fired precision, and the
IS/OOS cumulative win rates all move to the same test. IS and OOS are read
side by side as the overfitting signal, so measuring one in wins and the
other in agreement would put a fixed gap between them that has nothing to do
with generalization
- the confidence threshold is FITTED on wins too, so the operating point
maximises what the gate grades
- per-class label-agreement precision is still computed and logged; it is the
right diagnostic for class separation, just not for a deploy decision
- era line renamed dir-precision -> win-rate, chance -> chance=break-even
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Per-bar outcome of each DIRECTION taken on its own, cached beside the label under the same
//--- m_labelCacheHasValue flag. Not derivable from the label: Buy implies winLong, but the converse
//--- fails on every both-won bar, and those are ~27% of the sample under the measured geometry. This
//--- is what the deploy gate scores against - see m_oosWinLongTotal.
bool m_winLongCache [ ] ;
bool m_winShortCache [ ] ;
2026-07-14 22:36:27 -04:00
bool m_labelCacheHasValue [ ] ;
int m_labelCacheBars ; // 0 = no cache built yet
datetime m_labelCacheAnchorTime ; // m_Time.GetData(0) at last (re)build - 2nd invalidation key
void ComputeLabelForBar ( int i , int bars , bool & buy , bool & sell ) ;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
void AdvanceBarrierLabelState ( int i , int bars ) ;
//--- The triple-barrier verdict for one bar - the training TARGET. See BARRIER_TIE_GOES_TO_STOP.
//--- `idx` is a now-relative index; the scan walks FORWARD in time, i.e. toward index 0, and needs
//--- m_barrierHorizonBars of them to exist, so callers must keep idx >= m_barrierHorizonBars.
//--- Returns Neutral for any bar it cannot resolve (no ATR, ran out of history), which is the same
//--- answer as "no setup" and keeps the caller free of a third outcome to handle.
ENUM_SIGNAL TripleBarrierLabel ( int idx ) ;
//--- Resolves the SL/TP ATR multiples the label uses from the EA's live SL_Mode/TP_Mode. Split out
//--- because the INTELLIGENT modes scale with AI confidence, which does not exist at label time -
//--- see the definition for why the label uses their zero-confidence base instead.
void BarrierMultiples ( double & slMult , double & tpMult ) ;
fix(geometry): the reachability floor measured the WRONG WINDOW - my bug from bc57aca, and it cost real width
RECONCILED: the derivation reported "target reached on 17.7% of bars" while the
label cache reported Buy on 35.9%. Nothing was broken. They measure different
windows, and both are correct:
EXCURSION window ~12 bars (the SWING MEDIAN) - what m_excUpCache accumulates
over. Deliberately short: sizing a barrier off travel
measured over a horizon that itself scales with the barrier
is circular, and it ran away to 14-31*ATR on EURUSD/USDCAD
in 2026-08-07. That guard is correct and stays.
BARRIER horizon 64 bars - what the LABEL walk and the first-passage ladder
run over, and how long the EA actually holds the trade.
So `up >= target` is a 12-bar question and `label == Buy` is a 64-bar one, and
the second can freely exceed the first. TripleBarrierLabel gates the excursion
accumulation on `idx - t <= excWindow` while the barrier walk and the ladder run
the full horizon - the split is explicit and intentional.
THE BUG IS MINE. bc57aca's scale ladder tested reachability with `up[i] >= tp`,
i.e. it asked the 12-bar question about a 64-bar trade. That understates
reachability by ~2x, which is why EVERY wide rung was rejected and the geometry
fell back to the tightest rung at 1.61/3.21. The data supported considerably
wider; the test was just asking the wrong question.
FIX: LadderWinShare() reads the answer off the first-passage ladder - target
touched strictly before the stop, over the full horizon, tie to the stop. That
is the identical question the label walk asks, so the ladder share and the Buy
rate should now agree to within rung discretisation. Both legs snap to the
SMALLEST rung at or above the requested multiple (harder target, harder stop) so
the floor stays conservative.
Expect the scale ladder to select a WIDER rung on the next relabel. On this
data the excursion test read 17.7% at q50 where the true full-horizon share is
35.9%, so rungs that scored 8.1% and 2.8% were likely well above the floor.
ALSO:
- Window reconciliation now PRINTED every derivation: excursion travel share,
ladder win share, and the label cache's Buy share side by side, with the
ladder-vs-label gap flagged if it exceeds rung discretisation. Those two must
agree; if they ever stop agreeing, one of them is wrong and the line says so.
- Renamed tpReach/slReach -> tpTravel/slTravel and relabelled the log line. They
describe the EXCURSION window and are near-tautological there (a q50 stop is
exceeded by ~50% of bars); calling them "reached within the horizon" is what
made the two quantities look like one.
- BARRIER_MIN_TP_REACH_PCT is now BARRIER_MIN_REACH_FRACTION_OF_BE (0.60) x
break-even instead of a hardcoded 20.0. Break-even for 1:RR is 100/(1+RR), so
the absolute floor silently tightened as RR rose - 0.60x at RR=2 but 0.80x at
RR=3, penalising the user for asking for a bigger target. Evaluates to exactly
20.0% at the shipped RR=2, so this is a no-op today and correct if the knob
moves.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:51:08 -04:00
//--- Long win-share at (sl, tp) from the first-passage ladder - the FULL horizon, not the excursion
//--- cache's much shorter reference window. See the definition for why that distinction is load-bearing.
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder
Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped
stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384).
1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels
started one per bar overlap by the label's lifespan, so n calls are worth
~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample
uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count.
The tell: the operating point's null-of-the-maximum gate is family-wise and
should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73
(64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model
whose margin distribution admits few bins, sat on the null; the rest cleared
a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently
alternated between the ENDS of its own range era to era (0.10 -> 0.88 ->
0.86 -> 0.66; coverage 16% <-> 73%).
TripleBarrierLabel now records when each label became KNOWABLE - the first
winning touch, or both stops, or the timeout - and the prebuild accumulates
the mean. EffectiveSampleSize() feeds the operating point, the member deploy
gate and the ensemble vote gate. Conservative by construction (n/L is an
upper bound on the damage); gates get harder, never easier.
2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and
since 4d8cb08 reachability is measured OVER that horizon - so a wider rung
buys itself the time that makes it look reachable. Same target -> horizon ->
reach -> target loop the excursion window is kept short to avoid; fixing the
window confusion reopened it through the other door. It walked 128 -> 256 ->
384 bars and stopped at q90, the widest rung there is, with every rung
reading 39-48% against a 20% floor. A floor nothing fails selects nothing.
Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected -
the same rule ReportGeometryExpectancyScan already applied. It was printing
the shipped pair as CLAMPED and disqualified ('h384!') two lines under the
deriver that chose it: two subsystems, one geometry, opposite verdicts.
3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped
independently to the coarse first-passage grid, re-rating each candidate:
q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL
measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the
ladder compared win shares taken at ratios from 1.63 to 2.17 and read the
differences as scale. It is why the reach column came out non-monotone in
width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in
log space and the target follows the ratio off it; the pair actually measured
is returned and logged, so a collision reads as a collision.
Also: LadderWinShare guarded against the conditional (fractal) geometry path,
which fills n from m_fracLegCount while leaving idxList empty - a latent
out-of-bounds on a currently-dead path.
New log lines: mean label lifespan and effective n on the label-cache line, the
required-vs-available horizon per rung, and the grid pair the reconciliation
actually measured (its tolerance now scales with the grid skew instead of a flat
5pp).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//--- effSl/effTp report the rung pair actually measured, which is NOT the pair asked for: the ladder is
//--- discrete. Callers that compare rungs to each other must log them - see the definition.
double LadderWinShare ( const int & idxList [ ] , int n , double sl , double tp ,
double & effSl , double & effTp ) ;
//--- Bars this (sl, tp) pair needs before its label means "target before stop" rather than "target
//--- before stop OR 384 bars, whichever comes first". Identical arithmetic to
//--- ComputeBarrierHorizonBars' swingMedian * slMult * tpMult, factored out so the scale ladder and
//--- the horizon resolver cannot drift apart - the 2026-08-17 runaway shipped a geometry needing 613
//--- bars while the expectancy scan was flagging that same pair as CLAMPED and disqualified.
int RequiredHorizonBars ( double slMult , double tpMult ) ;
fix(labels): correct EffectiveSampleSize clamp order, share the horizon ladder, retract a false justification
Self-review of 1540ba8 against the FULL 6,930-era log rather than the first
three minutes of it. Three corrections.
1. EffectiveSampleSize() clamped in the wrong order. MathMax(2, MathMin(eff,
rawN)) returns 2 when rawN is 1 - an effective sample LARGER than the raw
one, shrinking the SE in exactly the direction the function exists to
prevent. Floor first, cap at rawN last.
2. The horizon cap rejected on the CEILING only, and said so as though that
made the label untruncated. It does not: the horizon ladder also snaps DOWN,
so a pair needing 317 bars is granted 256 and is silently truncated without
ever being flagged CLAMPED. Added SnapHorizonToLadder() / GrantedHorizonBars()
and the scale ladder now reports "needs N gets M" per rung. Rejection stays on
the ceiling alone - matching ReportGeometryExpectancyScan's '!' exactly, which
was the point - because rejecting on the snap-down would select rungs for
landing just above a ladder point rather than for anything about the market.
ComputeBarrierHorizonBars' private copy of the ladder is gone; there is now
one copy, which is the whole reason RequiredHorizonBars was factored out.
3. RETRACTED THE JUSTIFICATION IN 1540ba8's COMMENTS. That commit claimed the
overlap correction was needed because the operating point's null-of-the-
maximum gate fired on 47/73 Perceptron eras (64%) where a family-wise test
should fire on ~5%. Those 73 fits were the first three minutes of a
six-and-a-half-hour run. Over the full run:
PAI 47/3214 = 1.5% HYB 30/1200 = 2.5%
CONV 4/63 = 6.3% LSTM 75/915 = 8.2%
All at or below the null. The gate from 7414570 is working as designed and
PAI's 47 clears were a cold-start transient never repeated in 3,141 later
fits; its threshold over the run's second half has sd 0.01. The overlap
correction is still right - sqrt(p(1-p)/n) on overlapping labels is the wrong
formula - but it fixes no observed failure, and it costs nothing today
because no model is near the deploy line.
WHAT THE FULL RUN DOES CONFIRM, unchanged: the geometry ran away exactly as
described (2.00/6.00 h128 -> 3.49/6.99 h256 -> 4.86/9.71 h384, three passes,
stopping at q90 because the quantile ladder ended), the label stayed long-skewed
at Buy 42.9% / Sell 22.6%, and no checkpoint on any of the four models ever
cleared the deployability floor. Pooled declustered win rates: PAI 31.70%,
HYB 31.02%, LSTM 32.69%, CONV 31.57% - every one 4-6pp below the 37% always-long
chance rate and 1-2.7pp below the 33.7% cost-adjusted break-even.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 07:30:19 -04:00
//--- Bars it would actually GET: the above, clamped to [MIN, MAX] and snapped DOWN to the ladder.
//--- These are two different numbers and conflating them is its own trap - the ceiling clamp is what
//--- ReportGeometryExpectancyScan disqualifies with '!', but the snap-down truncates as well and is
//--- silent about it (a pair needing 317 bars is granted 256). The scale ladder rejects on the
//--- ceiling only, matching the scan, and REPORTS the granted figure so the shortfall is visible
//--- rather than inferred.
int SnapHorizonToLadder ( int rawBars ) ;
int GrantedHorizonBars ( double slMult , double tpMult ) ;
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder
Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped
stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384).
1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels
started one per bar overlap by the label's lifespan, so n calls are worth
~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample
uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count.
The tell: the operating point's null-of-the-maximum gate is family-wise and
should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73
(64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model
whose margin distribution admits few bins, sat on the null; the rest cleared
a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently
alternated between the ENDS of its own range era to era (0.10 -> 0.88 ->
0.86 -> 0.66; coverage 16% <-> 73%).
TripleBarrierLabel now records when each label became KNOWABLE - the first
winning touch, or both stops, or the timeout - and the prebuild accumulates
the mean. EffectiveSampleSize() feeds the operating point, the member deploy
gate and the ensemble vote gate. Conservative by construction (n/L is an
upper bound on the damage); gates get harder, never easier.
2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and
since 4d8cb08 reachability is measured OVER that horizon - so a wider rung
buys itself the time that makes it look reachable. Same target -> horizon ->
reach -> target loop the excursion window is kept short to avoid; fixing the
window confusion reopened it through the other door. It walked 128 -> 256 ->
384 bars and stopped at q90, the widest rung there is, with every rung
reading 39-48% against a 20% floor. A floor nothing fails selects nothing.
Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected -
the same rule ReportGeometryExpectancyScan already applied. It was printing
the shipped pair as CLAMPED and disqualified ('h384!') two lines under the
deriver that chose it: two subsystems, one geometry, opposite verdicts.
3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped
independently to the coarse first-passage grid, re-rating each candidate:
q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL
measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the
ladder compared win shares taken at ratios from 1.63 to 2.17 and read the
differences as scale. It is why the reach column came out non-monotone in
width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in
log space and the target follows the ratio off it; the pair actually measured
is returned and logged, so a collision reads as a collision.
Also: LadderWinShare guarded against the conditional (fractal) geometry path,
which fills n from m_fracLegCount while leaving idxList empty - a latent
out-of-bounds on a currently-dead path.
New log lines: mean label lifespan and effective n on the label-cache line, the
required-vs-available horizon per rung, and the grid pair the reconciliation
actually measured (its tolerance now scales with the grid skew instead of a flat
5pp).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//--- Independent-observation count behind `rawN` overlapping triple-barrier labels. See
//--- m_lastLabelLifespan for the measurement and for what an uncorrected n did to the operating point.
double EffectiveSampleSize ( double rawN ) ;
//--- Mean bars-to-resolution over the label cache, or 1.0 before anything has been measured (which
//--- makes EffectiveSampleSize the identity, i.e. the old behaviour, rather than a guess).
double MeanLabelLifespan ( void ) ;
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective
The last run could not have demonstrated an edge either way, and nothing in the
log said so. Four changes so it does.
1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets
every era; m_oosSamples only resets on a full model reset. So 'always-long %'
decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and
0.0% at era 2219. This is the SAME bug already found and fixed for
logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left
in the one line whose whole job is to be the reference every other number is
read against. Correct at era 1, wrong everywhere after - including the '62%
zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is
finally readable.
2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate
'short by a hair' from 'short by an amount no strategy could cover'. The era
line now prints the required win rate, the SE, the effective n and the
lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars
and L=75.6 there are ~63 independent observations, putting the bar near 66% at
typical coverage.
3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages
at every ladder level, so each candidate geometry's resolution time is
readable without training on it - L-vs-width becomes a measurement across the
whole ladder in ONE run rather than a second chart. Each rung reports L,
n_eff, min provable edge and min provable EV.
4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability
are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio,
so min provable EV ~ width^2 while the cost saving from width is only linear.
Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that
clears reachability) is right once an edge is known; MEASURE (narrowest that
keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it
still has to be shown. The direction does not depend on the exponent, and
item 3 makes the exponent checkable.
Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a
rejected rung reports the previous rung's lifespan as its own; per-rung
detectability is labelled IS-sample based (the deriver may not see the holdout),
so absolute figures are optimistic while the ranking is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
//--- Last era's deploy-gate arithmetic, published purely so the era line can state the bar rather
//--- than leave it implicit. -1 = not computed this era. See where they are set in Train() for why a
//--- gate has to publish its own threshold: an unreachable one is otherwise indistinguishable from a
//--- merely unmet one, and this project spent two days reading the first as the second.
double m_lastEdgeFloorPct ;
double m_lastPrecSE ;
double m_lastEffN ;
//--- Mean bars-to-resolution at the rung LadderWinShare() last measured - a candidate geometry's
//--- lifespan, readable without training on it. See the scale ladder's detectability column.
double m_lastRungLifespan ;
feat(gate): cross-instrument pooled certification
The deploy bottleneck is CERTIFICATION, not training. A 4,738-bar OOS window at
L=75.6 holds ~63 independent observations; certifying a 3pp edge at 2 sigma needs
~1,036. More bars of the same symbol barely help - they overlap. Other symbols do
not.
WHAT POOLS. Not win rates: symbols have different derived geometries, different
break-evens and different drifts, so averaging raw rates across them is
meaningless. What pools is each symbol's EXCESS OVER ITS OWN CHANCE RATE,
combined by inverse-variance weighting (fixed-effects meta-analysis). Each symbol
keeps its own model, geometry and chance rate; only the evidence is combined.
THE CORRELATION PROBLEM, bracketed rather than assumed away. SP500 and NAS100 are
~0.9 correlated and pooling them as independent inflates the evidence. Nothing
here can measure that without sharing return series, so instead of guessing a
correction the gate reports both ends:
SE_INDEP = sqrt(1/SUM(1/var_i)) all members independent
SE_CORR = SUM(w_i * sqrt(var_i)) all members perfectly correlated
The truth is always between. THE GATE USES SE_CORR, so a pass cannot be an
artifact of correlated instruments - that bound already assumes the worst. The
ratio is logged as the diversification credit the gate declines to claim, so the
cost of that conservatism is visible instead of hidden.
SCOPE, deliberately limited: the pooled result is REPORTED, never folded into
tradeableOK. The local gate certifies the model that actually trades this symbol;
the pool answers the different question of whether the strategy has an edge at
all. Letting a cross-symbol result license a local deploy would ship a model that
never cleared its own bar - so it cannot.
Mechanics: one file per instrument (no concurrent-write path to get wrong), every
FileOpen carrying FILE_SHARE_READ|FILE_SHARE_WRITE, records skipped rather than
reinterpreted on a version mismatch, 12h staleness cutoff so a stopped chart
cannot vote, and pooling refused below 3 instruments. Poolability requires
matching timeframe and ratio; differing SYMBOL is the entire point. Publishing is
unconditional - a pool that only hears from winners is a selection effect, not a
meta-analysis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:45:37 -04:00
//--- CROSS-INSTRUMENT CERTIFICATION - see AIBase\PooledGate.mqh for why the deploy bottleneck is
//--- certification rather than training, and why only the evidence pools while each symbol keeps its
//--- own model, geometry and chance rate.
void PublishPoolRecord ( double chancePct , double winPct , double effN ) ;
int ReadPooledEvidence ( double & pooledExcessPp , double & seIndep , double & seCorr ,
string & detail ) ;
bool PooledGatePasses ( string & report ) ;
bool m_poolWriteWarned ; // one-shot: a pool that cannot be written must say so
//--- Last era's pooled verdict, cached for the era line. The pool is REPORTED, not folded into
//--- tradeableOK: the local gate certifies THIS symbol's model, which is what actually trades, while
//--- the pool answers the different question of whether the strategy has an edge at all. Letting a
//--- cross-symbol result license a local deploy would ship a model that never cleared its own bar.
bool m_lastPoolPasses ;
string m_lastPoolReport ;
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed
The derivation read the stop from q75 of ADVERSE travel and the target from q50
of FAVOURABLE travel. Over one horizon those distributions are broadly the same
shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no
matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1
payoff needing 64.3%. That was never a measurement, it was two mismatched
constants.
The reachability line printed beside it - "target on 50.0% of bars, stop on
25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm
anything, and it read as validation.
WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width;
ratio is EV-neutral (a driftless walk reaches +m before -k with probability
k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread
is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per
unit of travel. So:
RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%.
SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST,
taking the first rung whose implied 2x target is still reached often
enough to be a trainable class.
That last clause is the difference from the min-reward:risk raise removed in
2026-08-09, which forced target = 2 x stop with NO reachability test, landed on
6.66*ATR reachable on 3.3% of bars, and trained the model to predict something
that essentially never happened. Same ratio; the scale now retreats until the
data says the target is attainable. Every rung is logged.
LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's
"best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it
pinned to the top rung. A recommendation landing exactly on the edge of its own
search space is a boundary, not a finding: it cannot tell "5 ATR is optimal"
from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else
needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and
the horizon constraints (decided >= 60%, reachability floor) now bind instead of
a constant.
THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked
the configured pair up in its integer grid, and DeriveBarrierGeometry produces
CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so
cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2
scores -1.00000", which reads as a catastrophic score and actually means "never
evaluated". Worse, the grid skipped target<stop entirely because it "inverts the
trade's whole premise" - while the derivation was shipping exactly that. The
incumbent is now always scored as a peer (never crowned; it is already in force
and is not an enum pairing the scan could adopt).
BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless
SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was
62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp
better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and
a loss at (SL + spread), matching the expectancy scan's convention exactly so
the two reports cannot disagree.
It also feeds FitDirConfThreshold, which is the correctness half: the operating
point subtracts break-even from precision, so the frictionless figure made every
candidate threshold look better by the width of the spread - 2.2pp against a
measured edge of 2.3pp, i.e. very nearly all of it.
Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD".
Forces a full relabel and retrain. Requested.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
//--- Break-even WITH the spread, which is the bar a model actually has to clear. The frictionless
//--- SL/(SL+TP) is what every report quoted until 2026-08-17, and on SP500 H4 it read 64.3% while the
//--- MEASURED zero-skill rate was 62.1% - a 2.2pp gap that is exactly the cost, and that made every
//--- model look 2.2pp better than it was. A win nets (TP - spread), a loss costs (SL + spread).
//--- Falls back to frictionless when m_spreadAtr is unset (a loaded model that has not re-derived).
double CostAdjustedBreakEvenPct ( void ) ;
//--- Spread in ATR units, averaged over the IS bars - measured in ReportGeometryExpectancyScan, which
//--- is the only place with both the ATR series and the label cache in hand. 0 = not yet measured.
double m_spreadAtr ;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Median confirmed-ZigZag-leg length over the training window, snapped to the horizon ladder.
int ComputeBarrierHorizonBars ( int bars ) ;
//--- Resolves m_barrierHorizonBars exactly once per process, from live buffers. Needed on BOTH paths,
//--- which is the whole reason it is not simply inlined in the prebuild: a DEPLOYED model never enters
//--- Train(), so it never reaches StartLabelCachePrebuild() - yet OnlineLearnStep() reads the horizon
//--- as its confirmation delay. Left unresolved there it would sit at BARRIER_HORIZON_FALLBACK and, on
//--- any symbol whose real horizon is longer, backprop bars whose barriers had not actually resolved -
//--- silent lookahead in the one place that writes to a live, trading model.
void EnsureBarrierHorizon ( int bars ) ;
bool m_barrierHorizonResolved ;
2026-07-14 22:49:14 -04:00
//--- full per-bar INPUT feature vector cache (everything BufferTempData() computes: ATR-normalized
//--- OHLC, time-of-day encoding, volume delta, AD indicator buffers, ...). Same rationale as the
//--- label cache above - a given now-relative bar index's feature vector only depends on price/
//--- indicator history, never on model state, so recomputing it is pure waste: BufferTempData()
//--- used to rebuild every bar from scratch once per historyBars-wide window it appears in, AND
//--- again on every subsequent era on top of that. Flat array (idx*m_neuronsCount + feature),
//--- not a true 2D array - MQL5 can't dynamically resize an inner dimension. Shares the label
//--- cache's invalidation trigger (see EnsureBarCachesCapacity()) since both are keyed on the exact
//--- same now-relative index frame.
double m_featureCache [ ] ;
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth
Three findings from the 2026-08-11 audit:
1. The excursion head's trailing-quantile ring was deliberately never cleared
between eras ("a rolling estimate of the market, not of the era") - but
pass 3 re-walks the SAME OOS window every era, so at each walk's restart
the ring still held the outcome masks of the newest OOS bars from the
previous walk: the chronological FUTURE of the bars about to be scored.
For the first ~window+horizon pushes of every era the "trailing" incumbent
was partly a leading one - conservative for the gate (an informed incumbent
is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts.
The ring now clears at era-score reset; the warm-up bars simply don't score
the trail race, which the m_excTrailN gating already accounts for.
2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage)
against the incumbent's subset sum - valid only if head skill is uniform
across the OOS walk, while the trail-scored subset systematically excludes
each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/
m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593
made every scored bar disjoint). The dead trio is replaced by
m_excBrierHeadT: the head's Brier accumulated only on the bars the warm
incumbent also scored, so the race now compares both predictors on an
identical bar set.
3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a
cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the
sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so
BufferTempData cached an all-zero Wyckoff block as a success for the whole
bar frame: the one path the f6150ee only-cache-successes rule cannot see,
because it never fails (the ba13eef class, arriving through values that
never fail; a resumed model's era-0 prebuild starts milliseconds after
OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means
async warm-up (transient reject, retried), while deep bars beyond the
buffered depth keep the sanitize loop's neutral-fill so degraded history
still trains. Also fixed m_featureCacheValid's declaration comment, which
still described the pre-f6150ee cached-miss semantics.
Compile: 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
bool m_featureCacheHasValue [ ] ; // true once idx has a CACHED SUCCESS (f6150ee: only
// successes are ever cached - a miss is never stored,
// in any form; see BufferTempData's comment)
bool m_featureCacheValid [ ] ; // paired flag, always true when HasValue is true -
// kept for the (currently unreachable) cached-miss
// shape so the cache layout survives f6150ee
fix: a resumed model cached a cold ATR as permanent, so it never trained
BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true
with m_featureCacheValid[idx]=false - and the cache never re-tries a
miss. So a single feature read taken before the terminal had finished
calculating the indicator buffers marked those bars unusable for the
rest of the process, even though the data arrived milliseconds later.
MT5 fills an indicator's buffers asynchronously after the handle is
created, and a cold ATR returns 0 for EVERY index, not just its warm-up
tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the
price features would be meaningless), so the whole window failed, and
the whole cache was poisoned.
Only resumed models were hit, because only they read features that
early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3:
a fresh start sits through three separately-scheduled Train() calls
before anything touches a feature, which is exactly what those passes
are for. A resumed one skips them and TuneIndicatorsAndTrain drives
StartLabelCachePrebuild and the MI report from the first chart event.
Its rationale - "a restart already has a proven-synced history" - holds
for HISTORY and not for INDICATORS, which are recreated every process
start.
Downstream: BuildFeatureWindow failed on every bar of every era, so
add_loop never went true, so pass 2, pass 3, the era counter and the
checkpoint were all skipped and pass 1 swept 0->100% forever. The
"0 samples" MI report line at startup was the same failure, four
seconds earlier, already visible in the log.
- a miss is now cached only when it is PERMANENT; the two "not ready
yet" guards mark m_featureFailTransient and are recomputed on the
next visit. Steady-state cost is ~ind_Periods bars per era, not 54k.
- an era that discards itself now drops the feature cache before
restarting, so any remaining cause of this state self-heals instead
of looping.
Deleting the .nnw "fixed" this only by turning the model back into a
fresh one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:41:31 -04:00
//--- Set by BufferTempDataCompute when it rejected a bar because the data had not ARRIVED yet
//--- (price buffer EMPTY_VALUE, or an ATR the terminal has not finished calculating) as opposed to
//--- the bar being genuinely unusable. BufferTempData reads it to decide whether the miss may be
//--- cached - see both for why a wrongly-cached miss is unrecoverable and what it cost.
bool m_featureFailTransient ;
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- WHICH BLOCK rejected the bar, and at which series index. Cleared per bar alongside
//--- m_featureFailTransient and written only by a guard that actually returns false, so the healthy
//--- path costs one string clear against ~52 feature computations and 60-odd Add() calls.
//---
//--- This is the line that four sessions of this bug did not have. The pass-1 stall report could say
//--- "lookback slot 0 REJECTED (window had 24 of 832 values)" and no more, so which guard fired was
//--- INFERRED - by counting 4 price + 5 swing + 4 range + 4 volume + 6 time + 1 ATR = 24 and
//--- concluding feature 25 must be the MA. That arithmetic happened to be right and the conclusion
//--- drawn from it ("the MA handle is short") was wrong five times running, because a value count
//--- names a POSITION in the vector and the position does not tell you whether the block was cold,
//--- capped, invalid or reading off the end. Name the block, and print the handle depths beside it.
string m_featureFailBlock ;
int m_featureFailIdx ;
diag: name the cause when every feature window fails, and enforce the width contract
Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable
feature window, windows ok=0 failed=54681" and nothing else. That line
reads identically for a cold ATR, a conditionally-missing optional
feature block and an out-of-range index, so it cannot be diagnosed
without one restart per hypothesis.
Two changes:
1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit
exactly m_neuronsCount values on EVERY bar. A block that emits its
values on some bars and skips them on others (indicator, panel or
series unavailable for that bar) does not merely shorten the window -
it SHIFTS every feature after it into the wrong slot, and the net
then trains on silently misaligned inputs that still look like a
valid window to everything downstream. Now rejected, rolled back and
reported once, naming the optional blocks (XA / SPR / swing context)
as the ones carrying an availability test. Worth having independently
of the current stall.
2. BuildFeatureWindow records WHICH lookback slot rejected and how much
of the window was assembled, and the pass-1 stall report renders it:
"slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up
or history-edge read; "every lookback bar ACCEPTED and the window was
still short: 640 of 760" is a missing 6-value block.
No behaviour change on a healthy run: the width check is an equality
that already holds, and the diagnostics render only inside the
total-failure branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 10:31:56 -04:00
//--- Why the LAST BuildFeatureWindow failed, so the pass-1 stall report can name a cause instead of
//--- a count. Slot = which lookback position rejected (-1 = none did and the window was still
//--- short); Total = how many values had been assembled when it gave up.
int m_windowFailSlot ;
int m_windowFailTotal ;
bool m_featureWidthWarned ; // one-shot: the width contract is a structural fault
fix(altdata): median-fill instead of zero-fill, and a one-shot feature-vector autopsy
ALT-DATA AUDIT. The files themselves are healthy - all six symbols, 6,073 daily
rows, 2010-01-01 to 2026-08-17, no constant or degenerate columns, sane tails
(mac_cpi/mac_unemp flat ~47d is monthly data behaving correctly). The problem is
not the data, it is what happens where the data ISN'T.
CAltDataPanel::Features() returned an all-ZERO vector for any bar older than the
file's first row, and left blank cells at 0 too. Both were deliberate ('the block
is additive context and must degrade, never reject the bar') and that reasoning
holds for the CHANGE columns - but half these features are LEVELS: vix, ivol,
mac_y10, mac_cpi, mac_unemp, eia_util. For a level, 0 is not a missing reading,
it is an impossible one far outside the series' range. VIX does not visit zero.
And the spike lands in exactly the wrong place. Every alt file starts 2010-01-01
while the charts run far deeper - USDJPY H4 reaches ~1994, roughly HALF its
history - so 'alt block is all zeros' is precisely the predicate 'this bar is
older than 2010'. The IS/OOS split is chronological, so that predicate covers
~half of IS and none of OOS: an in-sample feature guaranteed to be useless
out-of-sample, and a bimodal input for the first BatchNorm to normalise. Not a
lookahead leak - a distribution corruption, which is quieter and was never
reported anywhere.
Now filled with the column MEDIAN over the covered range. A constant cannot leak
whatever its source - it takes the same value on every pre-coverage bar, so it
carries no information about which of those bars won - which is what makes a
median computed over later data legitimate here. Median not mean because the
series are skewed. Blank cells get the same treatment (eia_stk_idx1y alone has
181 blanks in 6,073 rows) and the count is now logged at load.
THE BACKOFF WAS ALREADY THERE AND WAS DEAD. Training.mqh arms m_coldSweepTick on
m_featureFailTransient, but only the open/ATR guards ever set that flag, so
f0cf659's cold ADMovingAverage looked PERMANENT and the sweep re-ran at full
speed forever. Setting the flag in the indicator guards revives the mechanism
that was already designed for this; no second backoff was needed and the one I
first wrote has been removed in favour of it.
SELF-HEALING, as asked. ReportFeatureHealth() runs once, the first time pass 1
produces usable windows, samples ~400 bars spread across the whole training range
and names every feature slot that is CONSTANT or mostly-zero, tagging alt-block
slots as alt[i]. Both of today's failures were the same shape - a block silently
produces nothing while every downstream number stays plausible - and neither an
accuracy figure nor a model can tell 'this feature is always 0' from 'this
feature is genuinely 0 here'. Evenly spaced sampling so a block that dies only in
deep history is caught as surely as one dead everywhere. A report, not a gate:
a rare-flag feature can be legitimately constant, and refusing to train would
turn a diagnostic into an outage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 12:53:51 -04:00
//--- One-shot feature-vector autopsy - see ReportFeatureHealth() for the two silent 2026-08-17
//--- failures it exists to catch. Runs the first time pass 1 produces usable windows.
void ReportFeatureHealth ( int bars ) ;
bool m_featureHealthReported ;
2026-07-14 22:49:14 -04:00
bool BufferTempDataCompute ( int idx ) ;
2026-07-19 11:04:38 -04:00
//--- Nearest confirmed (non-repainting) ZigZag pivot at or after fromIdx - see this method's
//--- definition comment and m_useSwingContext's declaration comment for the repainting-embargo
//--- rationale callers must apply to fromIdx before calling this.
bool FindConfirmedZigZagPivot ( int fromIdx , int & pivotIdx , double & pivotPrice , bool & pivotIsLow ) ;
2026-07-14 22:49:14 -04:00
bool EnsureBarCachesCapacity ( int bars ) ;
2026-07-14 22:36:27 -04:00
//--- Eager label-cache pre-build + true-label tally, run once per fresh start (see
2026-08-01 11:27:28 -04:00
//--- m_warmupPassesRemaining) BEFORE era 0's real training loop begins. Without it, era 0 trains with
//--- m_prevEraTrueBuyCount/Sell/Neutral all still 0, so UpdateClassPriors() has no measured
//--- distribution and era 0 alone gets ZERO correction for label imbalance. Pre-scanning the whole IS
//--- window upfront (reusing the same cache/ComputeLabelForBar() the era loop uses) lets era 0 start
//--- from real measured class base rates.
2026-07-14 22:36:27 -04:00
bool m_labelCachePrebuilt ; // true once the one-time pre-scan has completed
bool m_labelPrebuildActive ; // true while a chunked pre-scan is in progress
bool m_prebuildSeedPending ; // true: era 0's era-start reset must NOT stomp the
// prebuild-seeded m_prevEraTrue* counts with the
// still-empty live tally (see Train()'s era-start block)
int m_labelPrebuildBars ;
int m_labelPrebuildOosCutoff ;
int m_labelPrebuildIndex ;
int m_labelPrebuildBuyCount ;
int m_labelPrebuildSellCount ;
int m_labelPrebuildNeutralCount ;
void StartLabelCachePrebuild ( void ) ;
void AdvanceLabelCachePrebuild ( void ) ;
//--- Evaluation-only continual-learning OOS simulation: once the core model converges, a CLONE of its
//--- weights (never the production Net itself) walks forward through the OOS window bar-by-bar,
//--- scoring each bar with its current weights THEN learning from it - simulating how the model would
//--- adapt in live/forward trading. This must never feed back into Net or the real OOS convergence
//--- metric (dOosForecast/m_oosSamples), so it's tracked in entirely separate members and the clone is
//--- discarded (never Save()'d) once each walk completes.
CNet * m_simOosNet ; // NULL when no simulation is active
bool m_simOosRunActive ;
int m_simOosCutoff ; // oosCutoff snapshot from the run that converged
int m_simOosBarIndex ; // resume point, m_simOosCutoff-1 down to 0
double m_simOosForecast ; // smoothed accuracy - separate from dOosForecast
int m_simOosSamples ;
void StartOosContinualSimulation ( int bars , int oosCutoff ) ;
void AdvanceOosSimulationChunk ( void ) ;
2026-08-16 21:08:41 -04:00
//--- ONE-SHOT pattern-database backfill (user request 2026-08-16): "the DB needs to be filled
//--- during training so I do not have to run a backtest before deploying to live trading". Runs
//--- once, right after FinalizeTrainRun() has restored the DEPLOYED weights, walking the OOS window
//--- chronologically oldest->newest (same bounds pass 3 uses) and registering every fired call into
//--- the SAME per-pattern/direction tables live voting writes to (RegisterSignal/PatternTableName) -
//--- so UpdateSignalsWeights() has real win-rate history to rank on the instant the model goes live,
//--- instead of only starting to accumulate it from real trades placed after deployment.
bool m_dbBackfillActive ;
bool m_dbBackfillDone ; // one-shot per deployment - never re-armed by a later call
int m_dbBackfillIndex ; // resume point, descends to 2 (mirrors pass 3's m_oosScoreIndex)
int m_dbBackfillStartIndex ;
fix(gate): move the ranking slice to the OLD end - it walled off the recent chart
NOT COMPILED - user compiles.
User: "there is quite some trading going on, but absolutely nothing on the recent
area of the chart, like there is a hard wall starting around november 2025."
That wall is 7caf2f6's ranking slice, and it was placed at the wrong end. Chart
arrows are only ever drawn on bars pass 3 GRADES, and the slice reserved the
NEWEST 20% of the OOS window plus a label-horizon purge. At the live sizing -
~4,860 OOS bars, 128-bar horizon - that is ~1,100 H4 bars withheld from grading,
about ten months back from today, exactly where the wall appears.
The invisible cost was worse than the visible one: it handed the deploy gate the
OLDEST 80% of the OOS window and withheld the most recent regime from the single
decision that has to generalise forward.
Both fixed by putting the reserve at the oldest end instead:
[0, oosScoreHi) OOS - graded by pass 3 (NEWEST, arrows restored)
[oosScoreHi, rankLo) purge - one label horizon
[rankLo, oosCutoff) RANKING - backfill only, graded by nobody
[oosCutoff, calibLo) purge
[calibLo, calibHi) CALIBRATION
... IS
Of the three consumers competing for those bars, recency is worth least to the
ranking: it is an ORDERING of confidence tiers, far less regime-sensitive than an
absolute win rate, while the gate's power and the operator's read of the chart
both want the newest data. The slice keeps every property that made it worth
carving - never graded, never selected on, never seen by the gate, purged on both
sides - so the backfilled rows are still honestly out-of-sample.
RankSliceHiIndex is replaced by RankSliceLoIndex + OosScoreHiIndex; pass 3 now
excludes the slice at the TOP of its walk and descends to 2 as it always did.
The backfill walks [RankSliceLoIndex, oosCutoff) via a new m_dbBackfillStopIndex,
clamped at both ends so a degenerate slice yields an empty walk rather than one
that wanders into graded bars. Verified no reference to the old helper survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 22:58:55 -04:00
int m_dbBackfillStopIndex ; // inclusive floor - the ranking slice's newest bar
2026-08-16 21:08:41 -04:00
int m_dbBackfillBars ;
int m_dbBackfillFired ; // rows written, for the completion log line
fix: the DB backfill could never run, and HEAD did not compile
Four defects in 64c5dd5/1a05e63, found by review + a baseline compile.
Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable.
1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were
declared `virtual bool ... override`, but CAppDialog declares both as
`virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151
on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was
never a success flag to forward. Verified: 0 errors, 0 warnings.
2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual
simulation (that one has been dead since it was written). Both are armed
at the instant convergence is declared, and both advance only from inside
Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick
ArmStudyEvent site sits in the `else` of a branch taken whenever
m_trainingComplete is set and m_trainRunActive is clear - which is exactly
the state FinalizeTrainRun() leaves behind one line before they are armed.
Train() was never called again, so the walks sat at their start index
forever: no "simulation complete" line, and not one row written to the DB
this feature exists to fill. Only a manual Resume/Retrain unstuck them.
Both flags now keep the model schedulable.
3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed.
Ensemble members deploy at Train() ENTRY and return immediately (so no era
is wasted), which skips the era-end block the backfill was started from.
All four members were a no-op for a second, independent reason. Armed on
the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff.
4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key,
no duplicate check - and m_dbBackfillDone is in-memory, so every later
attach that retrained to convergence wrote a second full set of rows for
the same bars. The ranking would count one bar once per model that ever
deployed, weighting superseded opinions as heavily as the live one. A
.dbfill marker stamps the deployed era; written only on completion (an
interrupted walk redoes itself rather than ranking a partial window) and
deleted with the other sidecars on reset-weights.
Also: WarmBlocking's timeout was silent, which restored the exact silent
pin failure it was added to prevent - it now says so in the journal, and
returns true for "no reference pairs to wait for" so the warning stays rare
enough to be read.
Not addressed, needs a decision: the backfill scores the OOS window with the
checkpoint that was SELECTED as best on that same window, then writes those
win rates into the table filter weights rank on - the selection set consumed
twice, undiscounted, while the deploy gate right next to it applies a
family-wise correction for exactly that effect. The rows are also simulated
triple-barrier outcomes at today's spread sharing a table with realised
fills. The completion log line now states both plainly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:25:51 -04:00
long m_dbBackfillEra ; // era stamped into the .dbfill marker on completion
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
void StartPatternDatabaseBackfill ( int bars , int totalIter , int oosCutoff ) ;
2026-08-16 21:08:41 -04:00
void AdvancePatternDatabaseBackfill ( void ) ;
2026-07-14 22:36:27 -04:00
//--- same resumability problem one level up: TuneIndicatorsAndTrain()'s own trial loop calls
//--- Train() per trial and used to assume each call ran an entire trial to completion synchronously
int m_tuneTrialIndex ; // -1 = no multi-trial tuning run in progress
double m_tuneBestOosForecast ;
bool m_tuneLastTrialWasWin ;
bool m_tuneHaveBestCheckpoint ;
datetime m_tuneStartTrainBar ;
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//=== Filter-based indicator auto-tuner (see TuneIndicatorsByFilter) =============================
//--- Replaced a genetic + successive-halving search on 2026-08-01. That search scored every candidate
//--- by TRAINING a throwaway network on it, which cost 1152 eras (9-48 h depending on topology) before
//--- the real model started, and its short screening rungs could not separate candidates at all. The
//--- filter scores candidates by the mutual information between the resulting FEATURES and the LABELS -
//--- arithmetic over the feature cache, no training - so it finishes in seconds and its cost is
//--- independent of topology. See TuneIndicatorsByFilter() for the measurements and the honest limit.
bool m_tuneFilterDone ; // the one-shot filter pass has run for this model
//--- Mutual information between one feature column and the 3-class label, and the whole-vector score.
double FeatureColumnMI ( const double & vals [ ] , const int & labels [ ] , int n ) ;
diag(autotune): one label shuffle cannot settle the no-edge question
The permutation baseline added in 018afb1 came back on all four charts as
0.00401 nats against floors of 0.00267 / 0.00298 / 0.00318 - three draws
whose spread is as wide as the excess being judged, because one shuffle
is one sample from the null, not the null. That is not enough to retire a
topology on.
Now MI_NOISE_PERMUTATIONS draws, reported as mean +/- sd with a z-score,
plus two numbers the mean over 26 columns cannot express:
- the STRONGEST single feature's MI, against its own shuffled value.
One informative column among 25 useless ones is precisely the case
the mean hides, and precisely the case worth finding.
- the excess as a percentage of H(Y). At these sample sizes a z-score
can be comfortably significant while the effect is worthless, so
"is it real" and "is it big enough to matter" are asked separately
and answered separately.
The verdict line also now states the measure's limit every time rather
than only when the news is bad: this is a MARGINAL, PER-BAR statistic and
the network reads m_historyBars bars jointly, so it can prove signal
exists but never that it does not. It rules out a per-feature edge - and
therefore any indicator retuning - not an edge that lives in a
combination or across time.
Compiles 0 errors / 0 warnings, standard and Market.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:32:12 -04:00
//--- Returns the MEAN per-feature marginal MI. Side-effects two more numbers that the mean alone
//--- cannot express, both read by TuneIndicatorsByFilter's report - MQL5 forbids a default value on a
//--- reference parameter, so members rather than out-params:
//--- m_miBestColumn - the STRONGEST single feature's MI. A mean over 26 columns hides one good
//--- column among 25 useless ones, which is exactly the case worth catching.
//--- m_miLabelEntropy - H(Y) in nats for the sampled labels, so MI can be quoted as a FRACTION of
//--- what there is to know. "0.001 nats" means nothing on its own; "0.1% of the
//--- label's entropy" is a magnitude anyone can act on.
fix(autotune): MI scorer read an array nobody filled; add the permutation floor
THE TUNER WAS A SILENT NO-OP. Every chart logged
auto-tune complete - 17 candidate settings scored in ~139s,
feature/label mutual information 0.0000 -> 0.0000 nats (no improvement)
0.0000 is not a weak result, it is a broken measurement: finite-sample MI
is biased UPWARD, so even pure noise scores above zero. Cause:
ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the
bar's features to TempData and never touches m_featureCache - only the
caching wrapper BufferTempData() writes that array. It then read
m_featureCache, which ReInitADIndicators had just invalidated. Every
column came back constant, FeatureColumnMI returned 0 for all of them,
and all 17 candidates tied at exactly zero. 139 s per chart to return the
settings it started with.
Now reads the values back out of TempData, where they actually land. And
an exactly-zero best score is called out as a fault rather than reported
as "no improvement", because that is what it is.
ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has
been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats -
at these sample sizes the same order as any real edge in this domain - so
a raw MI figure is uninterpretable on its own. Shuffling the labels
destroys every genuine association while leaving sample size, binning and
class proportions intact, so the score it produces IS this dataset's
noise floor, measured rather than approximated. The log now reads
feature/label information - X nats against a shuffled-label floor of Y
and says outright whether the features carry usable information about the
target. It needs no training, no topology and no convergence, so unlike
every accuracy number in this codebase it cannot be confounded by an
optimizer or an objective. If the score sits on the floor, no change of
architecture can help - which is the question the last three days of
zero-edge results have been circling.
DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by
any amount. At ~11,000 directional calls the standard error of the
precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma
noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires
EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the
actual call count, so the bar scales with the evidence instead of needing
a hand-picked constant.
Recorded with it, because it is why chance is the right reference at all:
under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k),
and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k).
The label's own base rate IS the break-even rate, at every SL/TP setting.
So "beats chance" and "is profitable" are the same test, and no choice of
SL/TP can manufacture an edge - only prediction can.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
double ScoreCurrentParamsByMI ( bool shuffleLabels = false ) ;
diag(autotune): five permutations was still a coin flip - use a real test
The 5-draw z-score shipped an hour ago disproved itself on its first run.
All four charts scored the IDENTICAL 0.00401 nats on identical features
and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two
"AT THE NOISE FLOOR", two "a real association", same data. The entire
swing came from estimating the null's spread from five draws, where the
standard deviation of the standard-deviation estimate is ~35%: the
denominator was noisier than the effect it was judging.
Replaced with an empirical permutation test. 200 draws, p counted by rank
with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never
reported as exactly zero - no normality assumption and no spread to
estimate. The strongest single column is tested against the null
distribution OF THE MAXIMUM, which corrects for scoring 26 features at
once by construction and is far less conservative than Bonferroni.
Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI
and runs ONCE for the whole test - every draw reuses that sample and costs
a relabel plus 26 histogram passes, not 2000 feature extractions. The
coordinate sweep still calls the combined form, which is correct there:
each candidate changes the indicator settings, so its features really do
have to be re-extracted.
The verdict line keeps both questions apart and prints both answers: the
p-value for "is it real", the excess as a percentage of H(Y) for "is it
big enough to trade". At n=2000 those can disagree, and collapsing them
into one word is how a worthless effect gets called a discovery.
Compiles 0 errors / 0 warnings. Build tag permtest-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
//--- The same work split in two, so the permutation test can extract the sample ONCE and reuse it for
//--- every null draw. BuildMiSample returns the sample count (or -1); ScoreMiSample shuffles `labels`
//--- in place when asked, so the observed statistic must always be taken before the first draw.
diag(autotune): a positive control, and a scan that separates "no signal"
from "signal knocked out of step"
Four architecturally different networks landed on the same precision -
Buy 23-25% against a 25.4% base rate, Sell 19-22% against 22.0% - while
making completely different calls (HYBRID votes Sell on 69% of bars, PAI
on 41%). Precision equal to the base rate is what INDEPENDENCE looks
like, and precision under independence is fixed by the label
distribution, not by the architecture, so all four converging on it is
arithmetic rather than coincidence. Accuracy meanwhile tracks coverage
exactly as independence predicts (31.1/30.3/25.0 predicted vs
31.8/28.9/24.6 observed for PAI/CONV/HYB).
But "no information in the data" and "information destroyed upstream of
every topology" produce that identical picture, and the MI test alone
cannot tell them apart either. Two additions:
POSITIVE CONTROL. Three "measurements" in this codebase have turned out
to be silent no-ops that produced plausible numbers - the MI scorer
reading an array nobody filled, the eval-mode guard that switched off the
imbalance correction, the alternation gate whose premise was never true.
So the estimator now has to prove it responds to a signal known to be
present before any floor reading is believed: the label of a neighbouring
sample row, ~19 bars away and far inside the 128-bar barrier horizon, so
the two outcome windows overlap heavily and MUST be associated. Same
binning, same estimator. Near the floor => every MI figure is void.
ALIGNMENT SCAN. Re-scores against the label taken from bar i+k for k in
-5..+5. A peak at k != 0 is a feature/label misalignment - an off-by-one
in the label index, a horizon applied to the wrong bar, a feature window
that lags what it claims - which would destroy the information before any
topology saw it and would look identical in every accuracy number this EA
prints. A flat profile says the features simply do not carry this target.
The sampled range is trimmed by |k| at both ends so a shift is measured
rather than an edge effect, and both bars must carry a real label.
Also: BuildMiSample publishes its stride instead of the report
recomputing that arithmetic (it would drift), and the control sizes its
buffers from its own sample count rather than the caller's.
Compiles 0 errors / 0 warnings, standard and Market.
Build tag mi-control-align-v1. Redeploy only - no retrain, no model
deletion; the diagnostic runs on resumed models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:12:10 -04:00
//--- labelBarOffset != 0 takes the LABEL from bar i+offset while the features still come from bar i,
2026-08-02 08:12:47 -04:00
//--- which is what the alignment scan needs. The sampled range is trimmed by MiShiftPad() at both
//--- ends - a FIXED amount, never by |offset| - so every build enumerates the same bars in the same
//--- order and two builds can be compared row by row. Returns -1 if the offset exceeds the pad.
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- `target` selects WHICH outcome the features are scored against (MI_TARGET_*). Anything other
//--- than the barrier class is continuous and is discretised into 3 equal-frequency bins at the end
//--- of the build, so every downstream consumer sees the same 3-class shape it already handles.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far
BuildMiSample samples features from ONE bar. So every "MI is at the noise
floor" result this codebase has produced - including yesterday's p=0.18 on
SP500 H1 - described the ENTRY BAR's 31 features only, while the network is
fed 20 bars of them. If information lived at lag 7 and not lag 0, the report
would have said "no signal" while the model could still learn. The diagnostic
we have been making decisions on had a blind spot exactly the width of the
input vector.
Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as
the existing labelBarOffset and is not interchangeable with it. Shifting the
LABEL changes which trade is predicted, so at any non-zero offset the
features sit inside the labelled window and the score is lookahead - that is
precisely what the alignment scan measures and correctly reports (4.7x more
knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label
pinned to the entry bar, so every row stays causal.
ReportFeatureLagProfile() then scores k = 0..historyBars against the same
block-permutation null and reports the deepest lag that clears it - the
lookback the data supports, versus the 20 that was picked by hand and never
measured. The null is redrawn PER LAG: finite-sample MI bias moves with the
realised class counts and bin occupancy, and different rows survive the
validity checks at each lag, so one shared floor would be right for lag 0 and
wrong everywhere else. Draw count is reduced accordingly (40, not 200) since
cost is draws x historyBars; this figure decides a lookback, never a trade.
MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that
makes two builds comparable row by row.
Read-only - no input, topology or label change, so no retrain. Both builds
0/0. Build tag lag-profile-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
int BuildMiSample ( double & cols [ ] , int & labels [ ] , int labelBarOffset = 0 ,
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
int featureBarOffset = 0 , int target = MI_TARGET_BARRIER ) ;
//--- Is an "optimal SL/TP" head learnable? Scores the features against excursion magnitude and
//--- asymmetry instead of the barrier class - a different question, see the definition.
void ReportExcursionInformation ( void ) ;
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in
3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever
chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of
eleven guesses is not deriving anything.
WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It
ranks pairings by how predictable their OUTCOME is - a question about direction.
The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing
absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE
scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x.
Hence the scan failing its own gate on every run, and its "winner" wandering
2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is
strongly measurable, so derive the geometry from that instead.
stop = q25 of measured ADVERSE travel (ordinary noise does not reach it)
target = q50 of measured FAVOURABLE travel (reached ~half the time, by
construction, inside the horizon)
Continuous, in ATR units, superseding the enum multiples. Reachability ("target
on X% of bars, stop on Y%") and the implied break-even are printed so the choice
is auditable rather than trusted.
FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the
horizon with the target (first-passage time grows with the band) and the
excursions are measured OVER the horizon, so target -> horizon -> excursions ->
target is a real loop - deriving once sizes the target from travel measured
under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at
3 passes, and says so if it does not settle.
Does NOT create expectancy, and the log says as much: chance precision equals
break-even at every geometry (m/(m+k) on both sides). It buys a target the
market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio
forces a target the market rarely reaches, it WARNS rather than overriding -
the ratio is the user's risk policy, so the honest move is to state its cost.
That is the collision that once rejected 100% of setups.
Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg
files written earlier today still load (their length guard finds no doubles) and
a model that carries them was trained on them and never re-derives.
Also fixes a message from e5ceed6 that claimed "this model resumed from disk"
unconditionally - it printed above a "seeding era 0" line on a brand-new model,
because the branch fires whenever the cache is not built, which is equally true
before a fresh model's first prebuild. A diagnostic that misreports its own
trigger is worse than one that says nothing: it gets quoted back as evidence.
FORCES A FULL RETRAIN (labels change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- Sets the ATR multiples from the measured MFE/MAE quantiles instead of the mode enums. Returns
//--- false (and leaves the configured pair standing) when there are too few resolved excursions.
bool DeriveBarrierGeometry ( void ) ;
diag: MI feature-lag profile - close the blind spot in every MI verdict so far
BuildMiSample samples features from ONE bar. So every "MI is at the noise
floor" result this codebase has produced - including yesterday's p=0.18 on
SP500 H1 - described the ENTRY BAR's 31 features only, while the network is
fed 20 bars of them. If information lived at lag 7 and not lag 0, the report
would have said "no signal" while the model could still learn. The diagnostic
we have been making decisions on had a blind spot exactly the width of the
input vector.
Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as
the existing labelBarOffset and is not interchangeable with it. Shifting the
LABEL changes which trade is predicted, so at any non-zero offset the
features sit inside the labelled window and the score is lookahead - that is
precisely what the alignment scan measures and correctly reports (4.7x more
knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label
pinned to the entry bar, so every row stays causal.
ReportFeatureLagProfile() then scores k = 0..historyBars against the same
block-permutation null and reports the deepest lag that clears it - the
lookback the data supports, versus the 20 that was picked by hand and never
measured. The null is redrawn PER LAG: finite-sample MI bias moves with the
realised class counts and bin occupancy, and different rows survive the
validity checks at each lag, so one shared floor would be right for lag 0 and
wrong everywhere else. Draw count is reduced accordingly (40, not 200) since
cost is draws x historyBars; this figure decides a lookback, never a trade.
MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that
makes two builds comparable row by row.
Read-only - no input, topology or label change, so no retrain. Both builds
0/0. Build tag lag-profile-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
//--- LAG PROFILE: how far back the features still say anything about the entry they precede. Prints
//--- MI at feature lag k = 0..m_historyBars against the same block-permutation null, and returns the
//--- deepest lag that clears it - i.e. the lookback the data actually supports, rather than the 20 that
//--- was picked by hand and never measured.
//--- WHY THIS WAS MISSING AND WHY IT MATTERS: BuildMiSample samples ONE bar. Every "MI is at the noise
//--- floor" verdict this codebase has produced therefore described the ENTRY BAR's features only, while
//--- the network is fed m_historyBars of them. If information lived at lag 7 and not lag 0 the report
//--- would have said "no signal" while the model could still learn - so the diagnostic we have been
//--- deciding on had a blind spot exactly the width of the input vector.
int ReportFeatureLagProfile ( void ) ;
2026-08-02 08:12:47 -04:00
//--- Bars trimmed from each end of every MI sample. Must cover the largest offset any caller asks
//--- for: the alignment scan's MI_ALIGN_MAX_SHIFT and the positive control's horizon/4. Expressed
//--- once here so the control cannot drift out of agreement with the range it is sampled over -
//--- which is precisely the failure this replaced.
int MiShiftPad ( void ) const
{
diag: MI feature-lag profile - close the blind spot in every MI verdict so far
BuildMiSample samples features from ONE bar. So every "MI is at the noise
floor" result this codebase has produced - including yesterday's p=0.18 on
SP500 H1 - described the ENTRY BAR's 31 features only, while the network is
fed 20 bars of them. If information lived at lag 7 and not lag 0, the report
would have said "no signal" while the model could still learn. The diagnostic
we have been making decisions on had a blind spot exactly the width of the
input vector.
Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as
the existing labelBarOffset and is not interchangeable with it. Shifting the
LABEL changes which trade is predicted, so at any non-zero offset the
features sit inside the labelled window and the score is lookahead - that is
precisely what the alignment scan measures and correctly reports (4.7x more
knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label
pinned to the entry bar, so every row stays causal.
ReportFeatureLagProfile() then scores k = 0..historyBars against the same
block-permutation null and reports the deepest lag that clears it - the
lookback the data supports, versus the 20 that was picked by hand and never
measured. The null is redrawn PER LAG: finite-sample MI bias moves with the
realised class counts and bin occupancy, and different rows survive the
validity checks at each lag, so one shared floor would be right for lag 0 and
wrong everywhere else. Draw count is reduced accordingly (40, not 200) since
cost is draws x historyBars; this figure decides a lookback, never a trade.
MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that
makes two builds comparable row by row.
Read-only - no input, topology or label change, so no retrain. Both builds
0/0. Build tag lag-profile-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
//--- Also covers m_historyBars, because the lag profile shifts the FEATURES that far back and every
//--- build must still enumerate the identical bar set (see BuildMiSample's fixed-pad note - padding
//--- by the requested offset instead is what voided the positive control on 2026-08-02).
return MathMax ( ( int ) MathMax ( m_historyBars , 0 ) ,
MathMax ( MI_ALIGN_MAX_SHIFT , MathMax ( m_barrierHorizonBars , 1 ) / 4 ) ) ;
2026-08-02 08:12:47 -04:00
}
diag(autotune): five permutations was still a coin flip - use a real test
The 5-draw z-score shipped an hour ago disproved itself on its first run.
All four charts scored the IDENTICAL 0.00401 nats on identical features
and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two
"AT THE NOISE FLOOR", two "a real association", same data. The entire
swing came from estimating the null's spread from five draws, where the
standard deviation of the standard-deviation estimate is ~35%: the
denominator was noisier than the effect it was judging.
Replaced with an empirical permutation test. 200 draws, p counted by rank
with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never
reported as exactly zero - no normality assumption and no spread to
estimate. The strongest single column is tested against the null
distribution OF THE MAXIMUM, which corrects for scoring 26 features at
once by construction and is far less conservative than Bonferroni.
Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI
and runs ONCE for the whole test - every draw reuses that sample and costs
a relabel plus 26 histogram passes, not 2000 feature extractions. The
coordinate sweep still calls the combined form, which is correct there:
each candidate changes the indicator settings, so its features really do
have to be re-extracted.
The verdict line keeps both questions apart and prints both answers: the
p-value for "is it real", the excess as a percentage of H(Y) for "is it
big enough to trade". At n=2000 those can disagree, and collapsing them
into one word is how a worthless effect gets called a discovery.
Compiles 0 errors / 0 warnings. Build tag permtest-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
double ScoreMiSample ( const double & cols [ ] , int & labels [ ] , int n , bool shuffleLabels ) ;
2026-08-01 14:01:32 -04:00
//--- The permutation test + verdict, split out of the tuner so it is NOT gated on era 0 with it - see
//--- the definition. Read-only; runs once per attach, whether or not the sweep did.
void ReportFeatureLabelInformation ( void ) ;
fix: make the indicator tuner actually measure, and gate what it installs
ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates
returned exactly 0.00359 nats): the tune loop re-inits the indicators and then
scores, with no RefreshData() between.
ReInitADIndicators() does its part - Create() builds a NEW handle carrying the
new parameters, and the feature cache is flagged stale so features really are
recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and
only Refresh() copies data out of a handle into those. So every candidate was
scored on values still held from the PREVIOUS handle. My earlier guess in the
diagnostic ("suspect the feature cache") was wrong: the cache invalidation works.
Two things land together, because neither is safe alone:
1. RefreshData() after the re-init, so a candidate is scored on its own features.
2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and
the maximum of N draws from a null beats its incumbent almost every time - so
"it beat the incumbent" installs noise. This selector is the highest-stakes of
the three found in this audit because it ACTS: it overwrites the user's
configured indicator settings and forces BuildFreshTopology(), so the network
then trains on whatever the noise picked. Fixing (1) without (2) would have
made a dormant bug actively harmful.
The gate draws the winner's own permutation null once, then corrects the p-value
for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather
than the max-of-N resample used by the geometry scan because each candidate here
has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only
the one null. Exact under independence, mildly anti-conservative under positive
dependence - stated in the comment rather than hidden. A rejected winner restores
the configured settings, which best[] cannot do since the descent mutates it.
Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate()
calculates asynchronously, so if the spread is STILL zero the handles simply are
not done and the tuner needs to yield between candidates rather than score them
back to back - a state machine like the label prebuild. That distinction is now
readable from the log instead of requiring another guess.
No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 09:31:06 -04:00
//--- Smallest BarsCalculated() across the ENABLED tunable indicators, or -1 when none is on. A handle
//--- created by IndicatorCreate() calculates asynchronously, so a candidate scored before its handle
//--- has caught up is scored on an empty or partial buffer. The tuner reports this so "the parameter
//--- change did not reach the features" can be told apart from "it reached them but they weren't ready".
int TunableBarsCalculated ( void ) ;
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- Same number, plus HOW MANY tunable indicators were actually consulted. The two have to travel
//--- together: -1 is returned both when nothing tunable is enabled (no cap, healthy) and when an
//--- enabled handle answers INVALID (nothing readable at any index, fatal) - and the depth gate used
//--- to collapse those into one silent `return want`, which is precisely why a build carrying the
//--- full depth instrumentation logged nothing at all through a 38-minute two-chart outage.
int TunableBarsCalculated ( int & enabled ) ;
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first
THE ANSWER, off the instrumentation added in be39674, first run:
ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable
indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ...
Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982
MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not
short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem;
the previous session's five theories were all answering the wrong question.
And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on
that same chart at 15:24:18 and went on to train normally (feature health, 51
features, excursion head) reading the same indicator. Only ConvLSTM's handle -
the last member constructed - was dead. WHY is still not established. All four
members request ADMovingAverage with identical params, so MT5 hands them the SAME
refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over
exactly that shared handle; that is the obvious suspect and it is NOT yet proven,
so this commit does not act on it.
1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths.
"MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982
(h12)" says it is the SAME handle and someone released it; "(h-1)" says it was
never created. That is the difference between a refcount bug and a creation
failure and it is one field. This is the measurement the shared-handle suspicion
needs before anyone acts on it.
2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT.
A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles()
re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD
indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does
NOT release first: -1 means the terminal no longer knows the handle, so there is
nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could
decrement whatever now owns that number. 30s cooldown, because every ServableBars()
consumer reaches it including live inference on every tick. The feature cache is
dropped with it, and the log names before/after depths.
Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars
against a buffer that answers EMPTY_VALUE at every index - then discarding the era
and doing it again - is not a recovery.
3. THE SWEEP NOW HOLDS ON A DEAD HANDLE.
ServableBars() keeps answering `want` (its contract; live inference and online
learning have their own refusal paths and a 0 there reads as "no history at all").
SettledBars() - the training sweep's entry, the one caller that can afford to wait -
returns 0 instead, so Train() holds and reports rather than burning a full-history
pass it is guaranteed to throw away. A recreated handle is cold, so it primes
through the existing settle path on the next call. If the repair fails the member
holds indefinitely and says so every minute, and be39674's barrier liveness escape
releases the rest of the ensemble after 12 minutes - which is the correct
degradation and is exactly what the log shows happening.
4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST.
Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of
Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the
next free row on each member's FIRST PublishStatus() call, so the order was a race -
the members busy sweeping published before the ones sitting idle at the era barrier,
and be39674 sharpened it by (correctly) making a held member stop writing the terse
line. Rows are now keyed to m_ensembleIndex, the registration/construction order,
which is fixed for the life of the chart. Claimed on every publish rather than once,
so it is idempotent and refreshes the tag for a member whose ID was not final when it
first published (the config-tag suffix is appended during InitIndicators, after
EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded
from the model count, so a member that has not published yet leaves no gap and shifts
nobody.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
//--- Re-Create any ENABLED tunable indicator whose handle the terminal no longer recognises
//--- (BarsCalculated() < 0). Returns true when something was actually rebuilt.
//---
//--- 2026-08-17, and this is the measured fault rather than a theory: XAUUSD H4, ConvLSTM reported
//--- `Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982` - one dead handle, everything
//--- else at full depth, while LSTM on the SAME chart read the same indicator fine and trained. So
//--- the buffer was not short and the history was not missing: this member's handle had simply
//--- stopped existing, some seconds after another member ran the 34-candidate indicator auto-tune
//--- (whose inner loop is Create-then-IndicatorRelease over handles that all four members share,
//--- because identical params return the same refcounted handle). WHY it goes invalid is NOT
//--- established - IndicatorDepthReport() now prints handle NUMBERS so the next occurrence answers
//--- that directly - but the recovery does not depend on knowing: a member that cannot read its own
//--- indicator must rebuild it, not sweep 50k bars against a buffer that answers EMPTY_VALUE at
//--- every index and then discard the era and do it again.
bool RepairDeadIndicatorHandles ( void ) ;
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate
1dda479 clamped the training sweep. It left five other paths asking the indicators
for a depth they cannot serve, and on a live account the quiet ones are worse than
the stall was - a stalled chart is visible, a chart trading on a degraded feature
window is not.
ServableBars(want, context) is now the single gate, and all six go through it:
training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small
positive BarsCalculated is warm-up, which m_coldSweepTick owns)
label prebuild clamp - labels come from price/ADZigZag and would survive a
capped MA, but ResizeBuffers sizes EVERY buffer and a failed
CopyBuffer leaves m_MA EMPTY for the next reader, so this path
could silently re-break the block Train()'s clamp just fixed
live inference HOLD. Below `need` the swing block takes its degraded path and
inference runs on a different feature distribution than the model
was fitted on. This EA sizes real positions off that output, so
no signal beats a mismatched one
online learning HOLD, same reason and worse - this path WRITES to a live trading
model, so a mismatched (features, label) pair is not a wrong arrow,
it is a wrong weight update that compounds every bar
chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest
"Max bars in chart" is also 5000, so this one is genuinely
reachable; uncapped it repaints the window all-Neutral
research export clamp before the emptiness test, so a capped symbol exports the
depth it has rather than writing a CSV with a dead feature block -
an artefact that looks complete and is silently wrong
Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars
(16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so
the failure mode is unreachable rather than merely unlikely.
Not changed: a genuinely SHORT price history still takes the old degraded path at
every site. That is pre-existing behaviour and narrowing it would mute charts that
trade today, so it stays a separate decision rather than a side effect of this fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
//--- `want`, clamped to what the indicators can actually serve. THE single gate in front of every
//--- ResizeBuffers() call site (train, live inference, chart rescan, research export).
//---
//--- Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it in its own
//--- context, bounded by Tools > Options > Charts > "Max bars in chart" (TERMINAL_MAXBARS), and
//--- CopyBuffer() past that limit does not return a short read - it FAILS, so CDoubleBuffer keeps
//--- NOTHING and EVERY index answers EMPTY_VALUE. m_MA (CiCustom, ADMovingAverage) is the only
//--- custom indicator whose feature block REJECTS the bar on that (m_ADZigZag, also CiCustom,
//--- neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins, served on demand at any depth) - so the
//--- symptom is feature 25 of every single bar failing while the 24 price features under it are
//--- fine. Measured 2026-08-17 with contention ruled out (SP500 was at era 2552 throughout):
//--- SP500 16,234 bars -> trains XAUUSD 33,982 bars -> 0 of 33,966 windows, forever
//--- XTIUSD 16,611 bars -> trains USDJPY 50,179 bars -> 0 of 50,163 windows, forever
//--- Returns `want` untouched when nothing is capping, so the no-cap path costs one min().
int ServableBars ( int want , string context ) ;
feat(depth): prime -> settle -> sweep, and name which handle is short
"Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in
1dda479 was wrong. Two other candidate causes are falsified too: the price series
is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new
H4 bar), and the handles have been stable since 13:07 with zero windows for the 17
minutes after, so it is not download-in-progress and not handle churn. The MA period
tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not
indicator cost either.
What IS verified stays verified: CopyBuffer past the calculated depth fails outright
rather than short-reading, so the buffer holds nothing and every index reads
EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag
neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25
of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated:
16k-bar charts train, 34k/50k get zero windows forever.
So the WHY is still open, and this fix does not depend on it. Per the user's protocol:
the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated()
every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever
it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned
depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two
training paths, which snapshotted a value that may still have been climbing; the clamp
remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b).
The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature
scan starves the indicator threads the request just woke, which is how the failure
sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the
0->100% oscillation on the panel.
Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and
stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the
cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
//--- ServableBars() with a WAIT in front of it, for the paths that can afford one (the training sweep
//--- and the label prebuild). Returns >0 = the depth to use, or 0 = "not settled, come back later".
//---
//--- A short depth is not necessarily a final one: the terminal calculates and extends indicator
//--- history asynchronously in response to being ASKED, so the request itself is the primer and the
//--- answer can keep climbing for a while after it. Clamping to the first short reading therefore
//--- risks pinning training to a partial history that was still growing. So: prime, then poll until
//--- the count stops changing, and only then decide. Critically, NO SWEEP happens while waiting -
//--- a 50k-bar feature sweep is what starved the indicator threads being waited on, which is how the
//--- old failure sustained itself. Bounded by DEPTH_SETTLE_TIMEOUT_MS so a depth that never settles
//--- still trains on what it has rather than waiting forever.
int SettledBars ( int want , string context ) ;
//--- Per-indicator BarsCalculated(), for the cap/priming/stall lines. The 2026-08-17 outage cost a
//--- long diagnosis because the logs proved WHICH FEATURE died (feature 25, the MA block) but never
//--- which HANDLE was short - so the cause had to be inferred, and was guessed wrong twice before
//--- this existed. It answers that directly.
string IndicatorDepthReport ( void ) ;
2026-08-01 14:01:32 -04:00
bool m_miReportDone ;
2026-08-02 12:25:20 -04:00
//--- Eras the MI report has waited for the cross-asset panel to exist, so it describes the SAME
//--- feature vector training uses. Bounded, so a terminal that never syncs the reference symbols
//--- still gets its diagnostics rather than silently getting none.
int m_miReportDeferrals ;
feat(labels): measure which barrier is predictable at entry, don't guess
The alignment scan settled the shape of the problem: 4.7x more is
knowable 5 bars into a 128-bar window than at the entry the model
actually trades. A 6xATR target reached over 128 bars is decided
overwhelmingly by what happens DURING the window, so whatever the entry
state knows is buried under 128 bars of later noise. That is a property
of the TARGET, and it is why four different architectures all landed on
precision exactly equal to the base rate - no topology can undo it.
So measure the target. For each SL/TP pairing a user can actually select,
relabel the same sampled bars and score how much the SAME features say
about THAT outcome at entry. Seconds, no training, no topology, and it
runs on the diagnostic path that already exists.
Ranked on excess over its OWN null as a share of its OWN H(Y), never on
raw nats: each geometry has a different class balance, hence a different
finite-sample bias and a different amount of information there to find,
so raw MI would rank the most BALANCED label rather than the most
PREDICTABLE one. The break-even win rate m/(m+k) is printed beside each
so the ranking is read next to the bar the model must clear.
Stated in the output because it is the easy thing to get wrong: chance
precision EQUALS break-even at every geometry, so a tighter target does
not hand you expectancy. It buys predictability - less noise piled on top
of what the entry state knows - which is the one thing changing topology
cannot do.
Read-only by construction: it relabels a sampled copy via
TripleBarrierLabel(), never writes the label cache (which belongs to the
configured geometry), and restores the horizon and overrides it borrowed.
The overrides apply only when BOTH are positive, so a half-set pair can
never silently relabel a live run.
Compiles 0 errors / 0 warnings, standard and Market. Build tag
geometry-scan-v1. Redeploy only - no retrain to READ the ranking.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:32:04 -04:00
//--- Ranks every selectable SL/TP pairing by how much the SAME features say about THAT barrier
//--- outcome at ENTRY time - see the definition. Read-only: it relabels a sampled copy, never the
//--- label cache, and restores the barrier state it borrowed.
void ReportBarrierGeometryScan ( void ) ;
//--- Scan overrides consulted by BarrierMultiples(). Both > 0 or neither applies; 0 = off. Live only
//--- for the duration of ReportBarrierGeometryScan, and nothing persisted is keyed on them.
double m_barrierScanSlMult ;
double m_barrierScanTpMult ;
//--- true => BuildMiSample computes each label with TripleBarrierLabel() instead of reading the cache,
//--- because a hypothetical geometry's labels are by definition not cached.
bool m_barrierScanLiveLabels ;
fix(labels): the geometry scan rewarded the labels it should reject
First run named 3:10 on all four charts, at 2.3x the configured 2:6. That
answer was wrong and the fault was the ranking statistic.
3:10 wants a horizon of ~swingMedian*30 (~320 bars) and gets
BARRIER_HORIZON_MAX. Clamped, most trades never resolve, the unresolved
remainder all lands in Neutral, and H(Y) collapses. The old statistic
divided the excess BY H(Y) - so a collapsing denominator made the most
degenerate label look like the most predictable one. Every geometry from
2:6 upward was already showing the clamped h128, and the two widest
scored highest, which is the fingerprint of the artefact rather than of
signal.
Two fixes:
Rank on the raw excess in nats. Subtracting each geometry's OWN measured
null already removes the class-balance bias, which is the only thing the
normalisation was ever needed for.
Disqualify clamped geometries outright rather than ranking them down. The
deployed EA holds until SL or TP with no bar limit, so a truncated label
trains the model on a question the strategy never asks. They are still
printed, marked '!', so the disqualification is visible instead of a
silent omission - and the scan now says so explicitly when nothing
eligible is left, because "the limit is the feature set, not the target"
is itself the finding in that case.
The scan also reports each geometry's directional share and timeout share
now. A label nobody can trade is not a candidate however well it scores,
and that has to be visible in the same line as the score.
Compiles 0 errors / 0 warnings. Build tag geometry-scan-v2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:46:25 -04:00
//--- timed-out labels seen during one geometry's live relabel - see the scan's dir/to columns.
int m_barrierScanTimeouts ;
//--- set by ComputeBarrierHorizonBars: this geometry needs MORE time than BARRIER_HORIZON_MAX allows,
//--- so its label truncates a trade the EA would hold to SL/TP. Disqualifies it from the scan.
bool m_barrierHorizonClamped ;
diag(autotune): one label shuffle cannot settle the no-edge question
The permutation baseline added in 018afb1 came back on all four charts as
0.00401 nats against floors of 0.00267 / 0.00298 / 0.00318 - three draws
whose spread is as wide as the excess being judged, because one shuffle
is one sample from the null, not the null. That is not enough to retire a
topology on.
Now MI_NOISE_PERMUTATIONS draws, reported as mean +/- sd with a z-score,
plus two numbers the mean over 26 columns cannot express:
- the STRONGEST single feature's MI, against its own shuffled value.
One informative column among 25 useless ones is precisely the case
the mean hides, and precisely the case worth finding.
- the excess as a percentage of H(Y). At these sample sizes a z-score
can be comfortably significant while the effect is worthless, so
"is it real" and "is it big enough to matter" are asked separately
and answered separately.
The verdict line also now states the measure's limit every time rather
than only when the news is bad: this is a MARGINAL, PER-BAR statistic and
the network reads m_historyBars bars jointly, so it can prove signal
exists but never that it does not. It rules out a per-feature edge - and
therefore any indicator retuning - not an edge that lives in a
combination or across time.
Compiles 0 errors / 0 warnings, standard and Market.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:32:12 -04:00
double m_miBestColumn ;
double m_miLabelEntropy ;
diag(autotune): a positive control, and a scan that separates "no signal"
from "signal knocked out of step"
Four architecturally different networks landed on the same precision -
Buy 23-25% against a 25.4% base rate, Sell 19-22% against 22.0% - while
making completely different calls (HYBRID votes Sell on 69% of bars, PAI
on 41%). Precision equal to the base rate is what INDEPENDENCE looks
like, and precision under independence is fixed by the label
distribution, not by the architecture, so all four converging on it is
arithmetic rather than coincidence. Accuracy meanwhile tracks coverage
exactly as independence predicts (31.1/30.3/25.0 predicted vs
31.8/28.9/24.6 observed for PAI/CONV/HYB).
But "no information in the data" and "information destroyed upstream of
every topology" produce that identical picture, and the MI test alone
cannot tell them apart either. Two additions:
POSITIVE CONTROL. Three "measurements" in this codebase have turned out
to be silent no-ops that produced plausible numbers - the MI scorer
reading an array nobody filled, the eval-mode guard that switched off the
imbalance correction, the alternation gate whose premise was never true.
So the estimator now has to prove it responds to a signal known to be
present before any floor reading is believed: the label of a neighbouring
sample row, ~19 bars away and far inside the 128-bar barrier horizon, so
the two outcome windows overlap heavily and MUST be associated. Same
binning, same estimator. Near the floor => every MI figure is void.
ALIGNMENT SCAN. Re-scores against the label taken from bar i+k for k in
-5..+5. A peak at k != 0 is a feature/label misalignment - an off-by-one
in the label index, a horizon applied to the wrong bar, a feature window
that lags what it claims - which would destroy the information before any
topology saw it and would look identical in every accuracy number this EA
prints. A flat profile says the features simply do not carry this target.
The sampled range is trimmed by |k| at both ends so a shift is measured
rather than an edge effect, and both bars must carry a real label.
Also: BuildMiSample publishes its stride instead of the report
recomputing that arithmetic (it would drift), and the control sizes its
buffers from its own sample count rather than the caller's.
Compiles 0 errors / 0 warnings, standard and Market.
Build tag mi-control-align-v1. Redeploy only - no retrain, no model
deletion; the diagnostic runs on resumed models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:12:10 -04:00
//--- bars between two consecutive MI sample rows, set by BuildMiSample - see its note.
int m_miStrideBars ;
fix(diag): the symbol sweep was measuring its own sampling, not the market
Twelve cells came back with higher-timeframe "signal" 5-9x anything on
H1, at p=0.005. It was an artifact, and the sweep's own columns gave it
away: excess tracked the sampling STRIDE almost monotonically, and the
three D1 cells - stride collapsed to 1-5 bars against a 128-bar horizon,
i.e. ~99% window overlap - were the three highest. Three flaws, all the
same family: comparing numbers without the spread that belongs to them.
1. THE NULL ASSUMED INDEPENDENCE THE LABELS DO NOT HAVE. Triple-barrier
labels overlap; two rows less than one horizon apart share most of their
outcome window. A free Fisher-Yates shuffle destroys that dependence
along with the association, making the null far narrower than the truth
and handing out significance that isn't there - Lopez de Prado ch. 4
arriving through the back door of the significance test. Now permutes
contiguous BLOCKS of at least one horizon, so the null keeps the
autocorrelation and the p-value means what it says. It degrades honestly:
severe overlap leaves few blocks, the null widens, nothing reaches
significance. The block count is now printed, because THAT - not the row
count - is the sample size a p-value rests on, and a warning fires under
30 blocks so "not significant" is not misread as "no signal" when it
means "not enough independent history to tell".
2. THE POSITIVE CONTROL'S STRENGTH DEPENDED ON THE DATASET. It paired
each row's label with the NEXT SAMPLE ROW's, whose distance is the
stride - so on M5, where stride ran 160-717 bars against a 128-bar
horizon, it was pairing two windows that never overlap. All three M5
cells duly reported a FAILED estimator and voided their own results with
nothing wrong. A control whose strength varies with the cell cannot
certify the cell. Now pinned to a quarter of the horizon, where ~75%
overlap is guaranteed by construction.
3. THE LOOKAHEAD VERDICT HAD NO MARGIN. It flagged 7 of 12 cells on gaps
of 0.00008-0.00040 nats against a measured null sd of ~0.00030 - noise,
every one. Now requires 3 sd, the same discipline the deploy floor
applies to precision.
Compiles 0 errors / 0 warnings, standard and Market. Build tag
blockperm-v1. Supersedes every number from the sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 15:11:40 -04:00
//--- independent label blocks the permutation null was built from (rows within one barrier horizon
//--- move together, so THIS - not the row count - is the sample size the p-value really rests on).
int m_miNullBlocks ;
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
void TuneIndicatorsByFilter ( void ) ;
feat: add unified MA type support to indicator tuner
Add `MA_TYPE_PRESETS` enum covering advanced (ALMA, DEMA, ZLEMA, T3, Kalman) and standard (SMA, EMA, SMMA, LWMA) moving averages. Integrate `maType` and `bestMaType` into `CADIndicatorTuner` struct, update flatten/unflatten routines, and bump `AD_TUNE_PARAM_COUNT` to 33. This allows the auto-tuner to search over MA type alongside period, improving feature discovery.
2026-07-23 15:02:09 -04:00
//================================================================================================
2026-07-14 22:36:27 -04:00
void FinalizeTrainRun ( void ) ;
2026-07-25 16:39:11 -04:00
//--- The "this is now THE model" persistence sequence, shared by every deploy path so they can't
//--- drift apart: weights (carrying the current m_trainingComplete flag), the pure-MQL5 inference
//--- self-check, the calibration sidecar, and the EMA shadow. Callers set m_trainingComplete first -
//--- the flag is written INTO the .nnw here. Used by FinalizeTrainRun() (ladder/era-cap/stop deploys)
//--- and by DeployNow()/RetrainDeployed() (the panel buttons).
void PersistDeployedModel ( void ) ;
2026-07-22 13:33:56 -04:00
//--- On hitting the per-run era cap: asks the operator whether to keep training (true) or deploy
//--- the best checkpoint and stop (false). Headless (tester/optimizer) can't show a dialog, so it
//--- returns false. See m_maxErasPerRun's declaration comment.
bool PromptContinuePastEraCap ( double bestOos ) ;
2026-07-14 22:36:27 -04:00
//--- variables
//--- training control, driven by the control panel (Warrior_EA.mq5); Train()/OnTickHandler
//--- poll these rather than being torn down/rebuilt, so pausing/stopping never loses in-memory state
bool m_trainingPaused ; // true: Train() blocks between eras until unpaused
bool m_trainingStopRequested ; // true: OnTickHandler stops scheduling new training passes
2026-07-24 11:52:19 -04:00
//--- true for ANY Strategy Tester run - a single backtest AND every optimization pass (MQL_TESTER):
//--- the run must NEVER train. It seeds the agent-local cache from the deployed production model (see
//--- InitNeuralNetwork) and runs pure inference over the window - what a buyer expects from "backtest
//--- my deployed EA", and what makes optimizing TRADING parameters (SL/TP, filters, MM) fast and
//--- comparable (the AI is held fixed, not retrained per config). Training + online continual learning
//--- happen only on a live chart, where this is false.
feat: add unified MA type support to indicator tuner
Add `MA_TYPE_PRESETS` enum covering advanced (ALMA, DEMA, ZLEMA, T3, Kalman) and standard (SMA, EMA, SMMA, LWMA) moving averages. Integrate `maType` and `bestMaType` into `CADIndicatorTuner` struct, update flatten/unflatten routines, and bump `AD_TUNE_PARAM_COUNT` to 33. This allows the auto-tuner to search over MA type alongside period, improving feature discovery.
2026-07-23 15:02:09 -04:00
bool m_inferenceOnly ;
2026-07-27 11:28:23 -04:00
//--- true only when the current Net weights came from a saved .nnw on disk, not from a freshly-built
//--- random topology. Used to let an inference-only tester replay a seeded model even if that model's
//--- persisted trainingComplete flag is still false, while still blocking the "no model found, built a
//--- fresh topology" path from placing random-weight trades.
bool m_modelLoadedFromDisk ;
feat(ai): real conv receptive field + the reference's channel pool
CONV's convolution used window = step = one bar, which is a per-bar
projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never
mixed information across time, so "convolutional" described the layer type
and nothing about what it computed. Same finding that sank HYBRID's LSTM.
Pooling was removed on 2026-07-29 for being misconfigured against the conv
output's memory layout. That removal was right; leaving the conv at a
one-bar window was not. The two belong together: the NeuroNet_DNG reference
(references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels
byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with
pool(window=4, step=4), and the pool only earns its place because a conv
with a real receptive field sits above it.
The input is bar-major (BufferTempData appends m_neuronsCount contiguous
features per bar), so a flat window of k*m_neuronsCount spans exactly k
bars - the receptive field needed NO kernel change. The conv output is
position-major, so window == step == window_out is a clean
max-over-channels, which is what the reference does and what the existing
pool kernels already implement correctly.
New chain at H1 defaults (420 = 20 bars x 21):
conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152
pool w=8 s=8 -> 19
conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars)
We deliberately stop before the reference's SECOND pool: a channel pool
emits one scalar per position, so a trailing pool would hand the dense stack
18 values and force it to fan out 18 -> 64. That is a bottleneck below every
learnable layer - the same class of mistake the 2026-07-29 removal was about.
Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor
tracked sliding POSITIONS, but a conv's real width is units_count *
window_out. Any pool stacked on a conv would therefore have sized against a
width window_out times too small and silently built the wrong shape. Both
branches now read the built layer's actual Neurons(), which is what the
batch-norm branch already did for the same reason.
Also closes the architecture-pinning trap: a .nnw persists the window each
conv was built with, so an existing CONV/HYBRID model would have loaded
cleanly and gone on training under the OLD architecture. The conv weight
tensor is (window+1)*window_out, so this cannot be repaired in place -
EnforceTopologyContract now detects it, reports both shapes, and retrains.
Conv chain shape is derived in one place (ConvReceptiveFieldBars /
ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions /
ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup
config line, so what is built and what is logged cannot drift.
Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
//--- Set by EnforceTopologyContract() when a just-loaded .nnw was built by a superseded architecture
//--- that cannot be repaired in place (currently: a different conv receptive field, whose weight
//--- tensor is a different SHAPE). InitNeuralNetwork discards the load and retrains. A .nnw persists
//--- the architecture, not just the weights - see AI\Network.mqh CNet::FirstConvWindow.
bool m_topologySuperseded ;
2026-07-24 11:52:19 -04:00
//--- true once ValidateCpuInference() has confirmed this model's pure-MQL5 forward pass matches the
//--- compute backend's within tolerance (see CNet::SetCpuInference). Persisted in the .stats sidecar
//--- so an inference-only backtest can run DLL-free; if false (e.g. a conv/LSTM topology not yet
//--- ported, or a validation miss) the backtest falls back to the DLL. Measured at deploy on the
//--- chart (where a backend exists to compare against), never in the tester itself.
bool m_mqlInferenceValidated ;
2026-07-14 22:36:27 -04:00
string m_fileName ;
string m_folderPath ;
//--- which file Train()'s Net.Save() calls (and this method's own Net.Load()) actually target:
//--- the shared FILE_COMMON production weights normally, or a LOCAL per-agent cache file when
//--- running inside the Strategy Tester/optimizer (see InitNeuralNetwork) so that repeated
//--- optimization passes with an unchanged topology can reuse an already-trained model instead of
//--- re-running every era from scratch, without ever touching the live production .nnw/.cfg.
string m_activeFileName ;
bool m_activeFileCommon ;
2026-07-29 14:49:54 -04:00
//--- Name of the terminal-wide global variable this instance holds as an exclusive claim on
//--- m_activeFileName, or "" when it holds none. See AcquireConfigLock().
string m_configLockName ;
2026-07-18 14:56:41 -04:00
//--- user-settable via Inputs.mqh's TrainingOptimizer (SGD or ADAM), read into this member at
//--- construction. Honored by all three signal types - PAI/CONV via BuildFreshTopology(), and
//--- LSTM via CSignalLSTM::AddCustomLayers - since CNeuronLSTMOCL (AI\Network.mqh) now has an
//--- accelerated SGD+momentum path (LSTM_UpdateWeightsMomentum) alongside its original Adam-only
//--- one, and CNet::CNet's GPU/DirectML init gate no longer restricts LSTM topologies to ADAM.
2026-07-14 22:36:27 -04:00
int m_optimizationAlgo ;
int m_historyBars ;
2026-08-11 21:53:37 -04:00
//--- Input-window derivation for a NEW model (existing models adopt theirs from the .cfg): median
//--- confirmed swing leg from raw highs/lows - strict local extrema over +/-WINDOW_SWING_WING bars,
//--- alternation enforced - snapped down to {12,16,20,24,32}. Raw price, not the ZigZag indicator,
//--- because this runs at init where custom indicators are still cold (see the cold-cache rule).
int DeriveHistoryBars ( void ) ;
2026-07-14 22:36:27 -04:00
int m_outputNeuronsCount ;
int m_minNeuronsCount ;
int m_initialNeuronsCount ;
int m_neuronsCount ;
double m_neuronsReduction ;
int m_hiddenLayersCount ;
2026-07-22 22:51:04 -04:00
//--- LSTM-only recurrent hidden-unit count - see LstmHiddenSize's declaration comment
//--- (Variables\Inputs.mqh). Harmless, unused constant contribution to m_fingerprint for MLP/CONV.
int m_lstmHiddenSize ;
//--- CONV-only convolutional output-filter count - see ConvFilterCount's declaration comment
//--- (Variables\Inputs.mqh). Harmless, unused constant contribution to m_fingerprint for MLP/LSTM.
int m_convFilterCount ;
2026-07-14 22:36:27 -04:00
int m_minTrainYear ;
bool m_isInitialized ;
fix(deinit): a full model write was running ahead of the cheap cleanup
"Abnormal termination" is back, and this time it is not the arrows. The
timing names the culprit exactly:
16:02:31.547 OnDeinit: shutting down
16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up
16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE
OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining()
finalises an in-flight run, and FinalizeTrainRun() restores the best
checkpoint and then persists it - a full ~1MB model write per signal. So
the expensive step ran ahead of the cheap bounded one, which is precisely
the inversion the shutdown ordering exists to prevent. The previous fix
put PersistWeightsOnShutdown last and missed that StopTraining smuggles a
second save in at the front.
Two changes:
Cleanup now runs FIRST, then StopTraining, then the weight save. The
visible teardown is cheap and bounded, so it always completes even when
everything after it is killed.
And the deploy-persist inside FinalizeTrainRun is suppressed during
shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint
is already the live net by that line, and PersistWeightsOnShutdown writes
exactly those weights moments later. The old path wrote the same model
twice per signal - eight full writes across four charts - for no benefit.
A user-pressed Stop still persists immediately, because nothing else
would.
Compiles 0 errors / 0 warnings. Build tag deinit-order-v2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- true once OnDeinit has begun - see MarkShutdown()/FinalizeTrainRun().
bool m_shutdownInProgress ;
2026-07-14 22:36:27 -04:00
int m_fractalPeriods ;
2026-07-23 08:21:41 -04:00
//--- "weights" of the 4 CONFIDENCE TIERS a live AI fire can land in (0-100 each), the AI equivalent
//--- of classic indicators' several geometric m_pattern_N members. ConfidenceTier() buckets the
2026-07-26 18:33:12 -04:00
//--- live confidence magnitude (CalibratedConfidenceMagnitude()) into 4 equal bands between the
//--- HEAD'S OWN structural floor and 1.0 - 1/3 for the 3-class softmax, 0.5 for the regression head -
//--- tier 0 = weakest possible directional call, tier 3 = near-certain. UpdateSignalsWeights()
2026-07-23 08:21:41 -04:00
//--- (Expert\ExpertSignalCustom.mqh) then calibrates each tier's weight independently from its OWN
//--- realized win rate, same as any classic pattern.
//---
2026-07-26 18:33:12 -04:00
//--- The defaults span the SAME 0-100 conviction scale the classic patterns use, and that is the
//--- whole mechanism by which one Min vote to open governs both engines. Confidence is expressed as
//--- vote strength rather than as a separate entry floor, so Min_Vote_Open reads directly as "which
//--- tier is good enough", with no second scale to reason about:
//--- 25 tier 0 conf 0.333-0.500 (3-class) / 0.500-0.625 (regression) - a bare plurality
//--- 50 tier 1 conf 0.500-0.667 / 0.625-0.750
//--- 75 tier 2 conf 0.667-0.833 / 0.750-0.875
//--- 100 tier 3 conf 0.833-1.000 / 0.875-1.000 - near-certain
//--- So Min_Vote_Open = 50 lets tier 1 and up trade when the AI votes alone, 75 lets tier 2 and up,
//--- and so on. A near-coin-flip 0.34 argmax is NOT blocked at the AI boundary - it votes 25 and is
//--- filtered by the vote threshold, exactly as a weight-10 classic confirmation would be. Note these
//--- are only the DEFAULTS: with UseDatabaseRanking on, each tier's weight moves to its own measured
//--- win rate, which is precisely why the tiers must be separate patterns rather than one continuous
//--- confidence-to-weight formula - a formula would leave the ranking system nothing to calibrate.
//---
2026-07-23 08:21:41 -04:00
//--- This used to be a single m_pattern_0 covering every fire regardless of confidence. That was a
//--- real bug, not just a missed opportunity: with only one pattern, UpdateSignalsWeights()'
//--- averaging step collapses (average of one value is that value), so the SAME win rate got written
//--- into both the pattern weight AND the module weight (filter.Weight()) - and Direction() multiplies
//--- them together. A genuine 70%-win-rate model was therefore scored as (0.70*70)=49, not 70 - a
//--- quadratic, not linear, derating that got harsher the further from 100% the model's real win rate
//--- was. Classic indicators never hit this because their module weight blends across SEVERAL
//--- differently-performing patterns, diluting any one pattern's own score instead of squaring it.
//--- Splitting into 4 genuinely-different tiers restores that same blending for AI signals.
//---
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- The constructor defaults are the 25/50/75/100 table above, NOT the 80/87/93/100 one this
//--- paragraph used to quote. That older set was written when tier 0 still sat behind a confidence
//--- floor and an alternation gate, so every tier arrived pre-filtered and belonged in the 80-100
//--- band; both gates are gone (see LongCondition's comments), tier 0 is now a bare plurality, and
//--- the table was rebased to span the full conviction scale so Min_Vote_Open can select a tier.
//--- The stale numbers outlived the change by long enough to be quoted back as fact in a design
//--- discussion - a second table in the same declaration block is exactly how that happens.
//--- Ensemble mode uses the same shared weighted-vote path as a solo model; no quorum gate.
2026-07-23 08:21:41 -04:00
int m_pattern_0 , m_pattern_1 , m_pattern_2 , m_pattern_3 ;
2026-08-15 04:44:10 -04:00
//--- TRAINING TARGET (Meta_Labeling_Design.md). 0 = per-bar direction via triple barrier (every
//--- pre-2026-08-15 model); 1 = meta-labeling: one training row per JOURNALED CANDIDATE (a classic
//--- pattern instance from the signal DB), binary label "did the trade this candidate proposes win
//--- its triple barrier", 2-output softmax head (set once in CSignalMETA's constructor);
//--- 2 = fractal direction: per-bar 3-class label = direction to the next confirmed fractal extreme
//--- (TrainTargetFractal(), driven by the TrainingTarget input). Never mutated after configuration -
//--- it feeds the fingerprint (|TGT:META1 / |TGT:FRA1 tokens) like any other identity-defining member.
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999
p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of
cost). One net for all 52 pattern-sides, AIType=AI_META.
- NetForward.mqh: the host-side softmax+CE gradient generalized total==3 ->
2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no
compute backend changes.
- SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on
disk (decoupled from the config fingerprint that burned four S1 runs); the
GMT->server offset is measured PER ROW against entryPrice vs bar open
(DST-immune, histogram logged); a window-span regime filter drops the
pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the
input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR).
- Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate
calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell
lets checkpoint selection, the edge floor, the plateau ladder and the
family-wise deploy gate run UNCHANGED: precision reads as win rate among
traded candidates, chance as the base win rate, recalls as sensitivity/
specificity. Era-end META line: coverage x (p - break-even) vs the null.
- Labels are the side-conditional triple-barrier win caches - never the DB's
stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base
rate). Live inference + online learning guarded off until S3.
- Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename
slot keep meta models fully separate from direction models.
Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with
AIType=AI_META; S3 wires the votes via the per-side hooks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
int m_trainTarget ;
2026-08-15 16:50:36 -04:00
//--- True when this signal runs as one of the AI_HYBRID ensemble's members - see EnsembleMember().
bool m_ensembleMember ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- Slot in g_warriorEnsemble (registration order, -1 = not an ensemble member). Doubles as the
//--- bit index in the combined-vote masks and the cursor index - see the registry's header comment.
int m_ensembleIndex ;
2026-08-15 19:00:40 -04:00
//--- CONDITIONAL leg excursions for the fractal target's geometry (user request 2026-08-15:
//--- "calculate MAE and MFE from a fractal to the next"). One sample per Buy/Sell-labeled IS bar,
//--- recorded during the label prebuild by FractalDirectionLabel: favourable and adverse travel in
//--- ATR units measured from the labeled bar's close over the LEG the label points at (every bar up
//--- to and including the next fractal extreme), not over a fixed horizon. DeriveBarrierGeometry
//--- reads its quantiles instead of the pooled every-bar excursions, so the stop/target are sized
//--- for the bars the model actually trades. Safe against circularity ONLY because the fractal
//--- label does not depend on SL/TP (the barrier label does - never feed it this path). Recording
//--- stops once geometry is derived (the pair is pinned; later bars must not silently re-shape it).
double m_fracLegFav [ ] ;
double m_fracLegAdv [ ] ;
int m_fracLegCount ;
void RecordFractalLegExcursion ( const double fav , const double adv )
{
if ( fav < = 0.0 & & adv < = 0.0 )
return ;
int cap = ArraySize ( m_fracLegFav ) ;
if ( m_fracLegCount > = cap )
{
cap + = cap / 2 + 256 ;
ArrayResize ( m_fracLegFav , cap ) ;
ArrayResize ( m_fracLegAdv , cap ) ;
}
m_fracLegFav [ m_fracLegCount ] = MathMax ( fav , 0.0 ) ;
m_fracLegAdv [ m_fracLegCount ] = MathMax ( adv , 0.0 ) ;
m_fracLegCount + + ;
}
2026-08-15 16:54:43 -04:00
//--- This member's slot in the combined ensemble panel; claimed lazily on first publish (-1 = none).
int m_ensemblePanelSlot ;
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999
p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of
cost). One net for all 52 pattern-sides, AIType=AI_META.
- NetForward.mqh: the host-side softmax+CE gradient generalized total==3 ->
2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no
compute backend changes.
- SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on
disk (decoupled from the config fingerprint that burned four S1 runs); the
GMT->server offset is measured PER ROW against entryPrice vs bar open
(DST-immune, histogram logged); a window-span regime filter drops the
pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the
input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR).
- Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate
calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell
lets checkpoint selection, the edge floor, the plateau ladder and the
family-wise deploy gate run UNCHANGED: precision reads as win rate among
traded candidates, chance as the base win rate, recalls as sensitivity/
specificity. Era-end META line: coverage x (p - break-even) vs the null.
- Labels are the side-conditional triple-barrier win caches - never the DB's
stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base
rate). Live inference + online learning guarded off until S3.
- Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename
slot keep meta models fully separate from direction models.
Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with
AIType=AI_META; S3 wires the votes via the per-side hooks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Meta candidate store for the CURRENT era's bar grid, populated by MetaPrepareEra() (overridden
//--- in CSignalMETA; empty and unused for direction models). MQL5 series indices shift on every new
//--- closed bar, so these are re-resolved from the corpus' fixed bar TIMES at each era start -
//--- same invalidation reasoning as the label caches.
int m_metaCandBar [ ] ; // series index of the candidate's fire bar
char m_metaCandSide [ ] ; // +1 long / -1 short
double m_metaCandNetVote [ ] ; // the firing filter's own vote margin (raw weight units)
short m_metaCandFamily [ ] ; // 0=MA 1=RSI 2=MACD 3=Ichimoku
short m_metaCandPattern [ ] ; // Pattern_N within the family
int m_metaCandCount ;
//--- per-bar chain: m_metaCandHead[barIdx] -> first candidate id at that bar (-1 none),
//--- m_metaCandNext[candId] -> next candidate at the same bar. Rebuilt with the store.
int m_metaCandHead [ ] ;
int m_metaCandNext [ ] ;
//--- Candidate id for each pass-2 queue slot, parallel to m_isTrainQueue (see Training.mqh's
//--- queueing block); -1 on every slot for direction models. Swapped in lockstep by the shuffle.
int m_isTrainQueueCand [ ] ;
2026-08-13 09:03:40 -04:00
//--- Per-family (0-3) and per-side (0=long 1=short) OOS decomposition of the meta head's era -
//--- candidates / base wins / operating-point trades / wins among trades. The aggregate META line
//--- can hide a deployable subset inside a blended 66%: measured 2026-08-13, 350 eras of real
//--- +1-2pp ranking skill never cleared a 67.5% break-even IN AGGREGATE, and whether any family
//--- or side clears it alone is exactly what this answers. Reset each era beside m_oosBuyFired.
int m_metaFamCand [ 4 ] , m_metaFamWins [ 4 ] , m_metaFamFired [ 4 ] , m_metaFamFiredWins [ 4 ] ;
int m_metaSideCand [ 2 ] , m_metaSideWins [ 2 ] , m_metaSideFired [ 2 ] , m_metaSideFiredWins [ 2 ] ;
2026-07-14 22:36:27 -04:00
//--- functions
virtual bool InitIndicators ( CIndicators * indicators ) ;
2026-07-23 08:21:41 -04:00
//--- sets ID/m_id/m_folderPath/m_fileName/m_pattern_count from the subclass constructor - defaults
//--- to 4 (the confidence tiers - see m_pattern_0's declaration comment), not 1
void SetIdentity ( string id , string shortId , int patternCount = 4 ) ;
2026-07-14 22:36:27 -04:00
//--- hook for neuron-type-specific layers (Conv+Pool, LSTM, ...); default is a plain perceptron (no-op)
virtual bool AddCustomLayers ( CArrayObj * topology ) { return true ; }
2026-07-29 00:38:05 -04:00
//--- Reusable front-end stages, composed by the AddCustomLayers() overrides. Each subclass names the
//--- stages it wants instead of re-declaring the CLayerDescription fields, so the topologies cannot
fix(ai): drop the conv pooling stage - it reduced across filters, not time
FeedForwardConv emits POSITION-MAJOR output, matrix_o[out + window_out * i],
so one bar's window_out filter responses are contiguous and consecutive bars
sit window_out apart. Both pooling implementations (FeedForwardProof and
CPU_FeedForwardProof) slide FLAT over that buffer - pos = i * step, reducing
`window` CONSECUTIVE elements. On a position-major layout those neighbours
are different FILTERS of the same bar, never one filter across time.
At the shipped 3/2 the pool computed max(bar0_f0, bar0_f1, bar0_f2), then
max(bar0_f2, bar0_f3, bar0_f4), with every 8th window straddling a bar
boundary. So it collapsed unrelated feature detectors into whichever fired
hardest, passed gradient to that winner only, and halved the feature map
while doing it - all below every learnable layer, where nothing above can
recover it. The removed inputs' own labels ("3 Bars") show time-axis pooling
was the intent throughout.
Measured cost: CONV sat pinned at ~40% balanced accuracy for 510 eras with
Sell recall 0%, while plain MLPs on the same data reached 57-61%. HYBRID,
which also carried this stage, came second-worst of the batch-norm group.
Not fixable in the topology: pooling one filter across time needs a stride
of window_out BETWEEN samples within a window, which a consecutive-window
kernel cannot express at any window/step. That needs a stride-aware kernel
in Network.cl + WarriorCPU.cpp + WarriorDML.cpp and a DLL rebuild, and is
only worth doing if a conv front-end earns its place without downsampling
first - with 20 sliding positions there is little to gain by halving them.
ConvPoolWindow/ConvPoolStep and their enums are removed with it, along with
the |CP: fingerprint term added earlier today.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:28:44 -04:00
//--- silently drift apart: HYBRID is *defined* as AddConvStage + AddLstmStage, which makes its
2026-07-29 00:38:05 -04:00
//--- "matches the standalone CONV front-end exactly, then adds LSTM" contract structural rather than
//--- a comment. (They had already drifted - HYBRID guarded the LSTM step with MathMax(1,...) and
//--- CSignalLSTM did not, so a historyBars of 1 gave the two a different step.)
fix(ai): drop the conv pooling stage - it reduced across filters, not time
FeedForwardConv emits POSITION-MAJOR output, matrix_o[out + window_out * i],
so one bar's window_out filter responses are contiguous and consecutive bars
sit window_out apart. Both pooling implementations (FeedForwardProof and
CPU_FeedForwardProof) slide FLAT over that buffer - pos = i * step, reducing
`window` CONSECUTIVE elements. On a position-major layout those neighbours
are different FILTERS of the same bar, never one filter across time.
At the shipped 3/2 the pool computed max(bar0_f0, bar0_f1, bar0_f2), then
max(bar0_f2, bar0_f3, bar0_f4), with every 8th window straddling a bar
boundary. So it collapsed unrelated feature detectors into whichever fired
hardest, passed gradient to that winner only, and halved the feature map
while doing it - all below every learnable layer, where nothing above can
recover it. The removed inputs' own labels ("3 Bars") show time-axis pooling
was the intent throughout.
Measured cost: CONV sat pinned at ~40% balanced accuracy for 510 eras with
Sell recall 0%, while plain MLPs on the same data reached 57-61%. HYBRID,
which also carried this stage, came second-worst of the batch-norm group.
Not fixable in the topology: pooling one filter across time needs a stride
of window_out BETWEEN samples within a window, which a consecutive-window
kernel cannot express at any window/step. That needs a stride-aware kernel
in Network.cl + WarriorCPU.cpp + WarriorDML.cpp and a DLL rebuild, and is
only worth doing if a conv front-end earns its place without downsampling
first - with 20 sliding positions there is little to gain by halving them.
ConvPoolWindow/ConvPoolStep and their enums are removed with it, along with
the |CP: fingerprint term added earlier today.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:28:44 -04:00
bool AddConvStage ( CArrayObj * topology ) ;
2026-07-29 00:38:05 -04:00
bool AddLstmStage ( CArrayObj * topology ) ;
2026-07-30 15:20:30 -04:00
//--- Which front-end stages this subclass's AddCustomLayers() actually appends. Declared once per
//--- subclass and consumed by everything that has to reason about the built shape (the LSTM capacity
//--- budget, the startup config line), so those can never disagree with what was constructed. A
//--- virtual rather than an AIType check, so a future composition cannot silently get the wrong answer.
virtual bool UsesConvStage ( void ) const { return false ; }
virtual bool UsesLstmStage ( void ) const { return false ; }
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999
p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of
cost). One net for all 52 pattern-sides, AIType=AI_META.
- NetForward.mqh: the host-side softmax+CE gradient generalized total==3 ->
2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no
compute backend changes.
- SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on
disk (decoupled from the config fingerprint that burned four S1 runs); the
GMT->server offset is measured PER ROW against entryPrice vs bar open
(DST-immune, histogram logged); a window-span regime filter drops the
pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the
input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR).
- Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate
calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell
lets checkpoint selection, the edge floor, the plateau ladder and the
family-wise deploy gate run UNCHANGED: precision reads as win rate among
traded candidates, chance as the base win rate, recalls as sensitivity/
specificity. Era-end META line: coverage x (p - break-even) vs the null.
- Labels are the side-conditional triple-barrier win caches - never the DB's
stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base
rate). Live inference + online learning guarded off until S3.
- Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename
slot keep meta models fully separate from direction models.
Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with
AIType=AI_META; S3 wires the votes via the per-side hooks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- META-TARGET SEAMS (all no-ops for direction models; overridden only by CSignalMETA).
bool IsMetaTarget ( void ) const { return m_trainTarget = = 1 ; }
2026-08-15 04:44:10 -04:00
//--- FRACTAL TARGET (TrainingTarget input, 2026-08-15 user request: predict swing turns again).
//--- The label is DIRECTION TO THE NEXT CONFIRMED FRACTAL EXTREME on every bar - the reference
//--- library's per-bar extremum-direction target, ~balanced by construction. Chosen over ZigZag
//--- pivots on the user's own point: a strict 5-bar fractal confirms only 2 bars after its extreme
//--- (no repaint embargo, so labels resolve almost to the present) and fires ~5x more often, which
//--- keeps the classes balanced without any of the 31:1 machinery the original "is this bar the
//--- pivot" target needed (b4a704d). Everything else - the barrier walk, measured SL/TP geometry,
//--- win caches, the era gate's realized-win scoring - runs unchanged, so the deploy decision still
//--- prices the trade actually placed.
bool IsFractalTarget ( void ) const { return m_trainTarget = = 2 ; }
//--- Labels.mqh: fractal-direction label for one bar (overrides the barrier verdict in
//--- AdvanceBarrierLabelState when IsFractalTarget()).
ENUM_SIGNAL FractalDirectionLabel ( int idx ) ;
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999
p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of
cost). One net for all 52 pattern-sides, AIType=AI_META.
- NetForward.mqh: the host-side softmax+CE gradient generalized total==3 ->
2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no
compute backend changes.
- SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on
disk (decoupled from the config fingerprint that burned four S1 runs); the
GMT->server offset is measured PER ROW against entryPrice vs bar open
(DST-immune, histogram logged); a window-span regime filter drops the
pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the
input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR).
- Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate
calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell
lets checkpoint selection, the edge floor, the plateau ladder and the
family-wise deploy gate run UNCHANGED: precision reads as win rate among
traded candidates, chance as the base win rate, recalls as sensitivity/
specificity. Era-end META line: coverage x (p - break-even) vs the null.
- Labels are the side-conditional triple-barrier win caches - never the DB's
stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base
rate). Live inference + online learning guarded off until S3.
- Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename
slot keep meta models fully separate from direction models.
Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with
AIType=AI_META; S3 wires the votes via the per-side hooks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Resolve the candidate corpus onto this era's bar grid (fills the m_metaCand* store). Called at
//--- every era start, right after the bar grid is sized; returning false aborts the training run.
virtual bool MetaPrepareEra ( const int bars ) { return true ; }
//--- Append the per-candidate setup descriptor to TempData, AFTER BuildFeatureWindow() has filled
//--- the shared bar window. The input layer is sized historyBars*features + MetaDescWidth(), so
//--- every feedForward on a meta net MUST run this between window build and forward.
virtual void AppendCandidateFeatures ( const int candId ) { }
//--- Width of that descriptor; 0 for direction models so NetInputWidth() stays byte-identical.
virtual int MetaDescWidth ( void ) const { return 0 ; }
//--- The one true input width every feedForward guard compares against.
int NetInputWidth ( void ) const { return ( int ) m_historyBars * m_neuronsCount + MetaDescWidth ( ) ; }
//--- P(win) from the 2-output head's raw activations in TempData (after Net.getResults) - the
//--- 2-class softmax collapses to a logistic over the logit difference. Same CLASS_LOGIT_SCALE the
//--- training gradient applies, so the probability is the one the loss was optimizing. -1 = no data.
double MetaWinProbability ( void )
{
if ( TempData . Total ( ) < 2 )
return -1.0 ;
double z = CLASS_LOGIT_SCALE * ( TempData . At ( 0 ) - TempData . At ( 1 ) ) ;
return 1.0 / ( 1.0 + MathExp ( - z ) ) ;
}
//--- Triple-barrier outcome of the candidate's own side at its fire bar - the meta LABEL. Reads the
//--- side-conditional win caches the label prebuild already computes for every bar; loss AND
//--- timeout are both 0, matching the design ("win=1 / loss-or-timeout=0").
bool MetaCandidateWon ( const int candId , const int barIdx )
{
if ( barIdx < 0 | | barIdx > = ArraySize ( m_labelCacheHasValue ) | | ! m_labelCacheHasValue [ barIdx ] )
return false ;
if ( m_metaCandSide [ candId ] > 0 )
return ( barIdx < ArraySize ( m_winLongCache ) ) ? m_winLongCache [ barIdx ] : false ;
return ( barIdx < ArraySize ( m_winShortCache ) ) ? m_winShortCache [ barIdx ] : false ;
}
//--- First candidate id at a bar (-1 none) / next in the same-bar chain.
int MetaCandFirst ( const int barIdx ) const
{ return ( barIdx > = 0 & & barIdx < ArraySize ( m_metaCandHead ) ) ? m_metaCandHead [ barIdx ] : -1 ; }
int MetaCandNext ( const int candId ) const
{ return ( candId > = 0 & & candId < ArraySize ( m_metaCandNext ) ) ? m_metaCandNext [ candId ] : -1 ; }
2026-07-30 15:20:30 -04:00
//--- AddConvStage runs BEFORE AddLstmStage wherever both are present (HYBRID), so the LSTM is fed the
//--- conv feature map rather than the raw flattened input.
bool HasConvBeforeLstm ( void ) const { return UsesConvStage ( ) & & UsesLstmStage ( ) ; }
feat(ai): real conv receptive field + the reference's channel pool
CONV's convolution used window = step = one bar, which is a per-bar
projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never
mixed information across time, so "convolutional" described the layer type
and nothing about what it computed. Same finding that sank HYBRID's LSTM.
Pooling was removed on 2026-07-29 for being misconfigured against the conv
output's memory layout. That removal was right; leaving the conv at a
one-bar window was not. The two belong together: the NeuroNet_DNG reference
(references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels
byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with
pool(window=4, step=4), and the pool only earns its place because a conv
with a real receptive field sits above it.
The input is bar-major (BufferTempData appends m_neuronsCount contiguous
features per bar), so a flat window of k*m_neuronsCount spans exactly k
bars - the receptive field needed NO kernel change. The conv output is
position-major, so window == step == window_out is a clean
max-over-channels, which is what the reference does and what the existing
pool kernels already implement correctly.
New chain at H1 defaults (420 = 20 bars x 21):
conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152
pool w=8 s=8 -> 19
conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars)
We deliberately stop before the reference's SECOND pool: a channel pool
emits one scalar per position, so a trailing pool would hand the dense stack
18 values and force it to fan out 18 -> 64. That is a bottleneck below every
learnable layer - the same class of mistake the 2026-07-29 removal was about.
Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor
tracked sliding POSITIONS, but a conv's real width is units_count *
window_out. Any pool stacked on a conv would therefore have sized against a
width window_out times too small and silently built the wrong shape. Both
branches now read the built layer's actual Neurons(), which is what the
batch-norm branch already did for the same reason.
Also closes the architecture-pinning trap: a .nnw persists the window each
conv was built with, so an existing CONV/HYBRID model would have loaded
cleanly and gone on training under the OLD architecture. The conv weight
tensor is (window+1)*window_out, so this cannot be repaired in place -
EnforceTopologyContract now detects it, reports both shapes, and retrains.
Conv chain shape is derived in one place (ConvReceptiveFieldBars /
ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions /
ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup
config line, so what is built and what is logged cannot drift.
Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
//--- Conv chain shape - see the definitions above AddConvStage. Every consumer reads these rather
//--- than re-deriving the arithmetic, so the built topology and the logged shape cannot disagree.
int ConvReceptiveFieldBars ( void ) const ;
int ConvFirstStagePositions ( void ) const ;
bool HasSecondConvStage ( void ) const ;
int ConvOutputPositions ( void ) const ;
int ConvOutputWidth ( void ) const ;
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in
ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp,
.stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn
chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA
after a reset but before an era completed re-created a .nnw from the
freshly-built, never-run net, so the next attach loaded an era-0 stub
instead of starting clean. For LSTM/HYBRID that stub is worse than nothing -
a layer that has never run a forward pass has m_iInputs<=0, so Save omits
every LSTM buffer (see 413ff7e). Skip the save when no era completed and no
model was loaded; that is exactly the post-reset and first-attach state.
Also sweep _shadowclone.tmp, which the reset did not cover.
Separately, ComputeLstmHiddenSize budgeted every topology against the
flattened input (historyBars x neuronsCount). True for LSTM, wrong for
HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature
map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults.
The quadratic is dominated by the inputs term, so overstating the fan-in
2.6x cost a full ladder step (16 units where the budget affords 32). New
virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this
rather than an AIType check. desc.window is advisory only - CNet never
passes it to the layer - but is now truthful for the same reason.
Derived values stay out of the weights-filename fingerprint and are adopted
from the .cfg, so existing models keep their saved width; only fresh ones
pick up the corrected budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
//--- Actual input width the LSTM block sees, which is NOT always the flattened input.
int LstmFanIn ( void ) const ;
2026-07-30 15:20:30 -04:00
//--- " | conv 21->8 x20 bars | lstm 160->32" for the startup config line; "" when neither applies.
string FrontEndConfigSummary ( void ) const ;
feat(ai): batch normalization between dense layers
The only bounded stage in the entire forward path was the sigmoid
classification head - every hidden stage is PRELU. That is a network with
no internal scale control, and the failure ordered exactly by depth: on
SP500 H1 the shallow perceptron held ~52% balanced accuracy while the
deepest topology sat on the 33.3% one-class floor, with the per-bar logit
spread decaying monotonically (0.45 -> 0.38 over ~200 eras) until the
evidence tilt fell under the class-prior tilt. That is the signature of
internal covariate shift, which chapter 6.1 of the reference book is
entirely about and which the NeuroNet_DNG engine addresses with a layer
this project never had.
Two mechanisms make this the right fix rather than more hyperparameter
nudging:
- it decouples WEIGHT_DECAY from the learned function (van Laarhoven
2017) - with a normalized layer downstream, decay can no longer grind
the discriminative signal away, it only rescales the effective
learning rate;
- it is the precondition for ever running an unbounded logit head here.
The 2026-07-27 attempt blew up (IS error 5.6e15) precisely because
nothing upstream constrained scale.
Implementation notes:
- CNeuronBatchNormOCL computes host-side rather than as a fourth copy of
a kernel across Network.cl + WarriorCPU.cpp + WarriorDML.cpp. The math
is elementwise O(n); this way it behaves identically on all four
compute tiers, needs no DLL rebuild, and cannot drift between
backends. Same precedent as the softmax+CCE gradient and the
per-sample loss weighting, both computed in MQL5 for that reason.
- Statistics are exponential moving, not a stored mini-batch: training
is pure online SGD, one update per sample, so there is no batch to
average over. BatchNormWindow is an EMA window length.
- gamma/beta are excluded from weight decay, deliberately - decaying
gamma toward zero is the exact pathology being fixed.
- The layer self-sizes from whatever sits below it, because a conv/pool
stage's output width is derived inside the CNet constructor and is not
knowable to the topology builder.
- Checkpoint capture/restore/blend carry gamma/beta and the running
statistics alongside the dense matrix, so the plateau ladder cannot
restore a mismatched pair.
- SeedOutputLayerBias accepted only an exact defNeuronBaseOCL as the
weight-carrying penultimate layer; with normalization enabled that is
the batch-norm layer, so the cold-start bias seed would have silently
stopped being applied.
- Refuses to build, loudly, if a topology asks for normalization with no
compute backend at all - rather than quietly training a different
architecture than the one requested.
EnableBatchNorm (default on) and BatchNormWindow (1000 samples) are
inputs so the effect can be A/B'd without a recompile. Both feed the
weights-filename fingerprint, appended conditionally so existing non-BN
configs keep their fingerprints and are not forced to retrain.
Verified: analytic gradients match finite differences to 1.5e-7 relative
over 200 random cases; a faithful port of the full forward/backward chain
collapses to the 33.3% floor by era 4 without this layer and holds
36-43% with it. Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:34:29 -04:00
//--- Appends a batch-normalization layer, or does nothing (returning success) when EnableBatchNorm is
//--- off. `units` is advisory only - CNet sizes the layer from whatever sits below it, because a conv
//--- or pool stage's output width is derived inside the CNet constructor and is not knowable here.
//--- See AI\NeuronBatchNorm.mqh for what the layer does and why it exists.
bool AddBatchNormStage ( CArrayObj * topology , int units ) ;
2026-07-14 22:36:27 -04:00
//--- hardcoded activation for the common tapering Dense hidden-layer stack built by
//--- BuildFreshTopology() (below AddCustomLayers, above the output layer). PRELU (leaky ReLU,
//--- 0.01 slope) is the default: it doesn't saturate/vanish the way TANH does across a deep
//--- taper, and is what the Conv layer itself already uses (see CSignalCONV::AddCustomLayers).
//--- CSignalLSTM overrides this to TANH instead - these Dense layers sit directly on top of the
//--- LSTM layer's own TANH-bounded ([-1,1]) output, so keeping them bounded too avoids feeding an
//--- unbounded activation straight off a bounded recurrent output, which is untested territory
//--- here and not what the user asked for ("Tanh for LSTM gates").
virtual ENUM_ACTIVATION HiddenLayerActivation ( void ) { return PRELU ; }
2026-07-29 12:00:40 -04:00
//--- Single source of truth for the output head's activation. BuildFreshTopology() stamps it into a
//--- NEW topology; EnforceTopologyContract() re-asserts it after every Load(), because a .nnw
//--- persists the activation and would otherwise pin a superseded architecture forever (see
//--- CNet::EnforceOutputActivation's declaration comment in AI\Network.mqh for the incident this
//--- comes from). Deliberately one expression called from both places - when these were two separate
//--- literals, changing the head in BuildFreshTopology() silently did nothing to any existing model.
//--- Regression (1 output): TANH - its native [-1,1] range maps directly onto the -1/0/1
//--- Sell/Neutral/Buy target convention. Classification (3 outputs): SIGMOID - see the long
//--- rationale at BuildFreshTopology()'s use of this method for why the head must stay BOUNDED.
ENUM_ACTIVATION OutputLayerActivation ( void ) const { return ( m_outputNeuronsCount = = 1 ) ? TANH : SIGMOID ; }
refactor(ai): derive the first dense layer's width instead of asking for it
InitialNeurons was an input whose only defensible value depends on two
things the user cannot see when picking from a dropdown: how wide the input
vector ended up after feature selection, and how much in-sample data the
study period actually yields. Left to a hand-picked constant it was badly
wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a
292,583-weight model, against ~36,500 training bars of which only ~2,236
are directional. That is 6.6 weights per training bar, and it EXPANDS a set
of highly correlated inputs rather than compressing them.
The symptom was already in the logs and had been read as a depth problem:
the shallowest topology consistently beat the deepest (perceptron 52.7%
balanced, hybrid 41.3%). Over-parameterization predicts that ordering just
as well as covariate shift does, and only one of the two had been addressed.
ComputeFirstLayerWidth() budgets roughly one first-layer weight per
in-sample bar. Measured across the configurations in use:
M15 10y -> 256 units, 129,071 weights, 0.73 per bar
H1 10y -> 64 units, 28,727 weights, 0.65 per bar
H4 10y -> 16 units, 7,559 weights, 0.68 per bar
Two design points that matter:
- It estimates in-sample bars from the STUDY PERIOD and timeframe, not
from Bars(). What is downloaded grows over a terminal's lifetime, and a
topology that widened as history filled in would re-key its own weights
file and discard a trained model.
- The result is snapped down to a coarse power-of-two ladder, so the
estimate would have to be wrong by ~2x to change the answer.
Every field it reads is already part of the weights-filename fingerprint,
so the derived value needs no fingerprint entry of its own. The public
setter is removed - it could only have been called after construction, and
would either be ignored or silently re-key the model mid-run.
Where the data cannot support even the floor (D1 over 10 years is under
2,000 bars) it now says so and names the fixes, rather than quietly
training a model with more weights than examples.
The DB config fingerprint drops the term too, which re-keys existing
pattern databases once - correct, since a model an order of magnitude
smaller should not inherit the old one's win-rate history.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
//--- Width of the first dense layer, DERIVED rather than configured. It used to be an input
//--- (FIRST_LAYER_NEURONS, default 500) whose only sensible value depends entirely on two things the
//--- user cannot see: how wide the input vector ended up after feature selection, and how much
//--- in-sample data the study period actually yields. Left to a hand-picked constant it was badly
//--- wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model,
//--- against ~36,500 training bars of which only ~2,236 are directional. That is ~8 parameters per
//--- sample, and it EXPANDS a set of highly correlated inputs instead of compressing them. The
//--- symptom is already in the logs: the shallowest topology consistently beat the deepest, which is
//--- what over-parameterization looks like from the outside.
//--- Computed from the STUDY PERIOD rather than from bars currently downloaded, so the answer is a
//--- deterministic function of the inputs and cannot drift as history fills in - and is then snapped
//--- to a coarse power-of-two ladder so even a large error in the estimate lands on the same rung.
//--- Every field it reads is already part of the weights-filename fingerprint, so the derived value
//--- needs no fingerprint entry of its own. MUST be called before the fingerprint is built and never
//--- again (see the note on fingerprint-feeding members at the top of this file).
int ComputeFirstLayerWidth ( void ) const ;
2026-07-30 09:22:11 -04:00
//--- Expected in-sample training rows for the configured study period, split and timeframe. Factored
//--- out of ComputeFirstLayerWidth so every derived capacity decision spends the SAME budget - three
//--- stages each guessing at the training-set size independently is how they drift apart.
double EstimatedInSampleBars ( void ) const ;
//--- Conv output-filter count and LSTM hidden width, DERIVED for the same reason the first-layer width
//--- is. Both were inputs whose defaults (16 filters, 32 units) were fixed constants picked without
//--- reference to the input width they sit on or the data available to fit them - so on a minimal
//--- feature set the conv stage EXPANDED the input, and the LSTM block quietly carried more weights
//--- than the entire dense taper below it. Both MUST be called before the fingerprint is built and
//--- never again: they assign fingerprint-feeding members (see the note at the top of this file).
int ComputeConvFilterCount ( void ) const ;
int ComputeLstmHiddenSize ( void ) const ;
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- Dense-taper DEPTH, derived 2026-07-30 from the two endpoints the taper connects. It was the depth
//--- suffix on each AI_CHOICE entry (MLP_3L/MLP_4L/..._2L); asking a user to pick a layer count while
//--- the code derives the widths those layers taper between is asking for half a decision. Reads
//--- m_initialNeuronsCount, so it MUST be called after ComputeFirstLayerWidth and before the
//--- fingerprint - see the note on fingerprint-feeding members at the top of this file.
int ComputeHiddenLayerCount ( void ) const ;
2026-07-29 12:00:40 -04:00
//--- Re-assert everything about a just-loaded net that lives in the FILE but is owned by the CODE.
//--- Call after every successful Net.Load(); no-ops (and stays silent) when the file already agrees.
void EnforceTopologyContract ( void ) ;
2026-07-14 22:36:27 -04:00
//--- common network bootstrap: indicators, topology build/load, training-file bookkeeping
bool InitNeuralNetwork ( CIndicators * indicators ) ;
2026-07-29 14:49:54 -04:00
//--- Exclusive per-config claim, so two charts can never train into one set of model files. Every
//--- retrain-affecting input is already hashed into m_activeFileName, so "same file" IS "same
//--- config" - which makes the filename the only correct lock identity. Live charts only: each
//--- tester/optimizer agent is a separate process with its own sandboxed _optcache copy, and they
//--- are *meant* to run the same config in parallel.
bool AcquireConfigLock ( void ) ;
void ReleaseConfigLock ( void ) ;
2026-07-14 22:36:27 -04:00
void DrawObject ( datetime time , double signal , double high , double low ) ;
void DeleteObject ( datetime time ) ;
2026-07-21 00:03:45 -04:00
//--- Time-ordered NMS sweep over m_arrowSignalCache: prunes each same-direction run down to its
//--- earliest bar (deleting redundant neighbors within m_signalClusterWindow). Run once per era end.
void PruneDirectionalClusters ( int bars ) ;
2026-08-10 14:26:12 -04:00
//--- Whether BOTH directions can currently be traded, which is the precondition for the alternation
//--- rule in the NMS paths: with only one side enabled there is no opposite signal to wait for, so
//--- requiring alternation would suppress everything after the first call. This build has no
//--- long-only/short-only input - the AI head emits both and nothing downstream restricts by side -
//--- so it is constant true. Kept as a named predicate rather than folded away so the precondition
//--- is stated where the rule reads it, and adding a direction restriction later has one place to
//--- change instead of three call sites that silently assume both sides.
bool BothDirectionsTradeable ( void ) const { return true ; }
2026-07-21 00:03:45 -04:00
//--- Live newest-bar NMS accept test (time-keyed, idempotent per bar time - see m_signalClusterWindow).
2026-07-21 12:30:29 -04:00
bool NmsLiveAccept ( datetime barTime , ENUM_SIGNAL dir , double conf )
2026-07-21 00:03:45 -04:00
{
if ( m_signalClusterWindow < = 0 )
return true ;
2026-07-21 12:30:29 -04:00
if ( dir ! = Buy & & dir ! = Sell )
return true ;
// Idempotent re-eval of the same bar (RefreshLatestSignal can run more than once per bar).
if ( dir = = Buy & & m_nmsLiveBuyTime = = barTime )
return m_nmsLiveBuyAccept ;
if ( dir = = Sell & & m_nmsLiveSellTime = = barTime )
return m_nmsLiveSellAccept ;
2026-07-21 00:03:45 -04:00
long minGap = ( long ) m_signalClusterWindow * PeriodSeconds ( ) ;
2026-07-21 12:30:29 -04:00
datetime lastSame = ( dir = = Buy ) ? m_nmsLiveBuyTime : m_nmsLiveSellTime ;
bool accept ;
// 1) Same-direction contiguous collapse: suppress if within the window of the previous SEEN
// same-direction bar (advance last-seen below either way, so a whole run collapses to one).
if ( lastSame ! = 0 & & ( long ) ( barTime - lastSame ) < = minGap )
accept = false ;
else
{
// 2) Cross-direction resolution vs the last KEPT opposite signal: keep the stronger side.
accept = true ;
if ( m_nmsLiveKeptTime ! = 0 & & m_nmsLiveKeptDir ! = dir & &
( long ) ( barTime - m_nmsLiveKeptTime ) < = minGap )
{
if ( conf > m_nmsLiveKeptConf )
DeleteObject ( m_nmsLiveKeptTime ) ; // this bar is stronger: remove the weaker opposite arrow
else
accept = false ; // the kept opposite is stronger: suppress this bar
}
2026-08-10 14:26:12 -04:00
// 3) ALTERNATION. Rule 1 only collapses a same-direction run inside the window; past it, a
// second Buy is emitted with no Sell in between, giving Buy/Buy/Buy/Sell. When BOTH directions
// are tradeable that sequence is the model re-entering a move it is already in, not finding a
// new one. Require the kept sequence to alternate: the first signal of a run passes (nothing
// to alternate with), and after that a direction only passes if the last KEPT signal was the
// opposite one.
// Gated on both directions being enabled - in a long-only or short-only configuration there
// is no opposite signal to wait for, so this would suppress everything after the first.
// NOTE this is deliberately a POST-PROCESSING rule, not a change to the label. The barrier
// target has no "must flip" invariant - consecutive Buy labels are routinely correct - and an
// earlier alternation gate was removed with the triple-barrier relabel for exactly that
// reason. What alternates is what gets ACTED ON: arrows, the pass-3 declustered tally the
// deploy gate grades, and the live trade.
if ( accept & & BothDirectionsTradeable ( ) & & m_nmsLiveKeptTime ! = 0 & & m_nmsLiveKeptDir = = dir )
accept = false ;
2026-07-21 12:30:29 -04:00
}
2026-07-21 00:03:45 -04:00
if ( dir = = Buy )
{
m_nmsLiveBuyTime = barTime ;
2026-07-21 12:30:29 -04:00
m_nmsLiveBuyAccept = accept ;
2026-07-21 00:03:45 -04:00
}
2026-07-21 12:30:29 -04:00
else
2026-07-21 00:03:45 -04:00
{
m_nmsLiveSellTime = barTime ;
2026-07-21 12:30:29 -04:00
m_nmsLiveSellAccept = accept ;
2026-07-21 00:03:45 -04:00
}
2026-07-21 12:30:29 -04:00
if ( accept )
{
m_nmsLiveKeptTime = barTime ;
m_nmsLiveKeptDir = dir ;
m_nmsLiveKeptConf = conf ;
}
return accept ;
2026-07-21 00:03:45 -04:00
}
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
int PurgeChart ( void ) ;
2026-07-14 22:36:27 -04:00
ENUM_SIGNAL DoubleToSignal ( double value ) ;
2026-07-18 12:10:37 -04:00
//--- Shared status-label formatting for all three of Train()'s era passes (pass 1 sequential scan/
//--- display, pass 2 shuffled backProp, pass 3 post-training OOS scoring) - see m_isTrainQueue's and
//--- m_isPass2Active's declaration comments for why the era loop is now three passes instead of one.
//--- Extracted so the panel keeps updating every bar throughout ALL three passes instead of freezing
//--- during pass 2/3 the way it did when this was pass 1-only inline code - a stalled-looking panel
2026-07-18 14:56:41 -04:00
//--- during a still-running era was reported as looking "stuck". Throttled internally (see
//--- m_lastStatusLabelUpdateTick) - SetStatusLabel() (System\StatusLabel.mqh) does real text-layout
//--- work (TextGetSize/WrapLineInto per line) AND an unconditional ChartRedraw(0) every call, which
//--- gets slower as more chart objects accumulate over a long backtest. Calling it unconditionally
//--- once per bar was already the existing (pass-1-only) behavior; doing that again independently in
//--- pass 2 AND pass 3 roughly TRIPLED total ChartRedraw() calls per era and was the actual cause of
//--- a training run observed taking 1.5+ hours without completing era 0 - not the shuffle itself.
2026-07-18 23:03:51 -04:00
//--- forceRefresh=true bypasses the throttle below - used exactly once per era, right after the
//--- era-end block (Train()) finalizes m_eraCount/dOosForecast, so the panel's "Era %d"/"OOS Acc"
//--- fields update in the SAME moment the console's "training in progress - era N" line does.
//--- Without it, the panel's last real redraw during a normal (throttled) bar-scan call happened
//--- DURING pass 3, before m_eraCount++ - so it kept showing era N-1's number (with what was, by
//--- then, already era N's near-final accuracy) for this era's entire duration, only catching up to
//--- the correct era number once the NEXT era's own bar-scan calls started - a full one-era-behind
//--- display lag relative to the console log, reported in practice.
void UpdateTrainingStatusLabel ( const string & progressLine , double neuron0 , double neuron1 , double neuron2 , double signalValue , bool forceRefresh = false ) ;
2026-07-18 14:56:41 -04:00
//--- Per-instance (NOT a function-local static - see CExpertSignalCustom::Direction()'s declaration
//--- comment for why that distinction matters for a method shared across PAI/CONV/LSTM instances)
//--- wall-clock throttle gate for UpdateTrainingStatusLabel()'s ChartRedraw().
uint m_lastStatusLabelUpdateTick ;
2026-07-18 23:03:51 -04:00
//--- Last values passed to UpdateTrainingStatusLabel() - cached (updated on EVERY call, throttled
//--- or not) so the forced era-end refresh above has something real to redraw with instead of a
//--- stale/zeroed placeholder, since no "current bar" exists once an era's own three passes are done.
double m_lastDisplayNeuron0 , m_lastDisplayNeuron1 , m_lastDisplayNeuron2 , m_lastDisplaySignal ;
2026-07-22 22:51:04 -04:00
//--- Latest OOS Buy/Sell recall (-1 = n/a, same convention as logBuyRecallPct etc.), cached the same
//--- way as m_lastDisplayNeuron0 above so UpdateTrainingStatusLabel() can show it on every call, not
//--- just the era-end one that actually just computed it. Surfaced on-chart (not just the Experts
//--- log) because a Buy/Sell-diluted-by-Neutral headline accuracy number is what a trader watching
//--- the panel sees by default, but Buy/Sell recall is what actually predicts trading performance -
//--- Neutral is "don't trade," so a model can look good on blended accuracy purely by calling Neutral
//--- often, while its actual Buy/Sell calls are unreliable.
int m_lastBuyRecallPct , m_lastSellRecallPct ;
2026-07-15 21:47:09 -04:00
//--- turns the classification output layer's 3 values (TempData[0..2], SIGMOID activation - see
//--- BuildFreshTopology() - each already in [0,1]) into a softmax probability distribution in
//--- place, and returns the signed dPrevSignal convention (+P(buy), -P(sell), 0.0 exactly for
//--- neutral) used by both Train()'s live-forecast branch and RefreshLatestSignal(). Softmax is
//--- monotonic per-element so it can't change which of the 3 wins - it only exists to turn the 3
//--- independently-trained values into a normalized confidence. Max-subtracted before exp() for
//--- numerical stability regardless (safe since softmax(x) == softmax(x - max(x))).
2026-07-14 22:36:27 -04:00
double ApplyClassificationSoftmax ( void ) ;
2026-07-23 19:36:34 -04:00
//--- Post-hoc logit adjustment / prior correction: reads the raw softmax probabilities
//--- ApplyClassificationSoftmax() just left in TempData[0..2] and returns the PRIOR-CORRECTED signed
//--- decision (same +P'(buy)/-P'(sell)/0-neutral convention). This is the exact rule live trading
//--- fires on and the rule the live-fired precision metric scores. See the definition for the math.
double AdjustedSignalFromSoftmax ( void ) ;
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- Margin between the winning class and its best rival, from the softmax already in TempData.
//--- Returns <0 when the winner is Neutral (not a directional call, so no operating point applies)
//--- or when the outputs are unreadable. This is the statistic the threshold is expressed in - NOT
//--- the winning probability on its own, which moves with how confident the net is overall rather
//--- than with how CLOSE the decision was, and would therefore drift as calibration changes.
double DirectionalMargin ( void ) ;
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Reset / accumulate / fit, in the order the calibration walk calls them. See
//--- DIR_CONF_THRESHOLD_BINS and DIR_CONF_CALIB_PCT_OF_IS.
feat: excursion-size head (Stage 1, measurement only)
Direction is closed - normalised asymmetry fails on three instruments
with a working positive control, and the classifier's own best-of-999
era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is
a different question and RANGE clears at ~4x its null.
Checked the denomination before building on that, since the source memo
warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is
predictable" is a claim about travel RELATIVE to current ATR, not a
restatement of "ATR is autocorrelated". It is exactly the part a fixed
multiple (stop 3.31*ATR, target 1.64*ATR) discards.
A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches
ladder rung k) upward and downward. Survival parameterisation rather than
regressing the multiple, because it needs nothing new from CNet: sigmoid
outputs and the per-neuron delta the `total != 3` branch already applies
(a quantile head would need a linear activation and a pinball gradient in
Network.mqh, Network.cl and the DirectML path, on a class four topologies
share). Targets are free - m_ladderUpAt already records first-touch age
per rung with 0 meaning never reached.
Separate net, not extra outputs on the classifier: more outputs would
change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push
the count off 3 - the exact condition backProp uses to select the joint
softmax gradient the 3-class head depends on. The classifier is
bit-for-bit unaffected and this is removable without trace.
STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the
constant per-rung base rate - the baseline a fixed ATR multiple already
assumes - with both predictors fitted IS and evaluated OOS, so neither
gets a look at the test set. Positive skill justifies Stage 2 (drive
SL/TP and sizing off ExcursionQuantile, which is defined and deliberately
uncalled). Zero or negative means ATR already carries everything and
Stage 2 must not be built.
Trains only on primary occurrences: the replay queue oversamples for
CLASS balance, and a direction-balanced sample is a biased SIZE sample.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION-SIZE HEAD - see Expert\AIBase\Excursion.mqh. Predicts how FAR price travels, never
//--- which way; Stage 1 measures whether it beats a constant ATR multiple and places no orders.
bool ExcursionBuildTopology ( CArrayObj & topology ) ;
bool ExcursionEnsureHead ( void ) ;
bool ExcursionTargets ( int idx ) ;
void ExcursionTrainStep ( int idx ) ;
void ExcursionScoreStep ( int idx ) ;
feat: race the excursion head against a trailing-quantile incumbent
Beating a frozen global constant is the weakest admissible bar for
replacing a global constant. The honest incumbent is a rolling rung
frequency: it adapts to the volatility regime - exactly what the head
claims to predict - and needs no model, no 760 inputs and no training.
Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one
ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon`
entries are held back UNRESOLVED: a bar's rung outcomes are only known
one horizon later, so using them would be lookahead and would flatter the
incumbent into an opponent the head could never fairly beat. Pass 3 walks
oldest-to-newest, so "pushed more than horizon bars ago" is exactly
"resolved by now". Each push is O(rungs), not O(window).
The head's decision-rung Brier is pro-rated to the trailing estimate's
coverage before the ratio, since the incumbent only scores bars where its
window is warm.
This line is worth reading on its own, independently of the head: if the
trailing quantile beats the global constant, that is a cheap risk-control
win available with no machine learning at all - and it is the same number
either way, so the run answers both questions in one pass.
The ring is deliberately NOT reset per era - it estimates the market, not
the era, and re-warming 500 bars every era would leave the incumbent
unusable over the first chunk of every scoring pass, handing the head a
free win on exactly those bars.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
void ExcursionTrailPush ( void ) ;
feat: excursion-size head (Stage 1, measurement only)
Direction is closed - normalised asymmetry fails on three instruments
with a working positive control, and the classifier's own best-of-999
era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is
a different question and RANGE clears at ~4x its null.
Checked the denomination before building on that, since the source memo
warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is
predictable" is a claim about travel RELATIVE to current ATR, not a
restatement of "ATR is autocorrelated". It is exactly the part a fixed
multiple (stop 3.31*ATR, target 1.64*ATR) discards.
A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches
ladder rung k) upward and downward. Survival parameterisation rather than
regressing the multiple, because it needs nothing new from CNet: sigmoid
outputs and the per-neuron delta the `total != 3` branch already applies
(a quantile head would need a linear activation and a pinball gradient in
Network.mqh, Network.cl and the DirectML path, on a class four topologies
share). Targets are free - m_ladderUpAt already records first-touch age
per rung with 0 meaning never reached.
Separate net, not extra outputs on the classifier: more outputs would
change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push
the count off 3 - the exact condition backProp uses to select the joint
softmax gradient the 3-class head depends on. The classifier is
bit-for-bit unaffected and this is removable without trace.
STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the
constant per-rung base rate - the baseline a fixed ATR multiple already
assumes - with both predictors fitted IS and evaluated OOS, so neither
gets a look at the test set. Positive skill justifies Stage 2 (drive
SL/TP and sizing off ExcursionQuantile, which is defined and deliberately
uncalled). Zero or negative means ATR already carries everything and
Stage 2 must not be built.
Trains only on primary occurrences: the replay queue oversamples for
CLASS balance, and a direction-balanced sample is a biased SIZE sample.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
void ExcursionResetEraScores ( void ) ;
double ExcursionQuantile ( bool upward , double tau ) ;
void ExcursionReport ( void ) ;
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only
the second one knew it. Logit-adjusted cross-entropy has no term for "how
often should I trade", so the head calls a direction on 87-91% of bars. The
selection metric is precision x coverage credit, saturating at the coverage
floor - above the floor extra calls earn NOTHING and only precision counts.
So selection wanted few good calls, the loss produced many mediocre ones, and
all selection could do was pick the least-bad era out of what it was handed.
Nothing pushed the model toward selectivity.
This gives the decision RULE the policy instead of distorting the loss (which
is estimating class probabilities correctly, and a probability estimate should
not be bent to encode a trading policy - Elkan 2001: estimate, then choose the
operating point separately). AdjustedSignalFromSoftmax now abstains unless the
winning direction's softmax margin over its best rival clears a fitted
threshold. Margin, not the winning probability: the latter moves with overall
calibration rather than with how close the decision actually was.
Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS
sample, so the margin histogram is harvested there for free (primary
occurrences only, so the oversampled replay queue cannot skew the operating
point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate
grades the thresholded model on bars the threshold never saw. Fitting on
pass 3's own predictions would be choosing the operating point on the data
being graded - the best-of-N error corrected in five other places here.
Objective: maximise IS directional precision subject to still clearing the
SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived
locally so the two cannot drift apart). Swept top-down in one pass; ties go
to the LOWER threshold, since equal precision for less coverage is strictly
worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than
on a guess.
The threshold is part of the MODEL, not the run: captured with
Net.CaptureWeights(), restored with the weights at both restore sites, and
appended to the .cfg under the same length-guard convention so a deployed
model reloads at the operating point its gate actually cleared. A pre-2026-08-09
.cfg reads 0.0, which is exactly the behaviour it was trained under.
Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop
can be attributed to the operating point rather than guessed at.
Both build variants compile 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
void ResetDirConfHistogram ( void ) ;
void AccumulateDirConfSample ( double margin , bool wasCorrect , bool isPrimaryBar ) ;
void FitDirConfThreshold ( void ) ;
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- CALIBRATION BAND BOUNDS, in pass-1 bar indices (0 = newest bar, so LARGER index = OLDER).
//--- The era's bars lay out, newest to oldest:
//---
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
//--- [0, oosCutoff) OOS - graded by pass 3, never trained on
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- [oosCutoff, calibLo) purge - one label horizon, discarded entirely
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
//--- [calibLo, calibHi) CALIBRATION - fits the threshold, never trained
//--- on, never graded by pass 3, never seen by the
//--- deploy gate. ALSO the pattern-DB backfill's
//--- source (see StartPatternDatabaseBackfill)
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of 983a6a3, could never fire because IS
never showed p < p0.
Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS,
purged from backprop by one label horizon on BOTH sides (the far-side
purge is not optional: without it the newest training bars carry labels
partly decided by price action inside the slice, putting the memorization
straight back into the curve). Score it in a new chunked pass 2.5, after
pass 2 has trained and before pass 3 grades - the only position where the
histogram is simultaneously not-trained-on, not-graded, and current with
the weights it will be applied to.
Costs 15% of the training data. Worth it beyond honesty: the deploy gate
needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned
near zero dilutes any edge concentrated in the confident bars across every
bar the model calls, driving dirPrecPct toward chance by construction. A
threshold that can be selective is the only mechanism by which a small,
concentrated edge could ever clear that gate.
Also: a sparse histogram now KEEPS the previous threshold instead of
resetting to 0.0. A failed measurement must not decay to the most exposed
setting in the range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- [calibHi, calibHi + horizon) purge - one label horizon, discarded entirely
//--- [calibHi + horizon, ...) IS - the backprop queue
//---
//--- A purge on BOTH sides, not just the OOS one: a triple-barrier label is decided by the horizon
//--- bars that FOLLOW its bar, so without the far-side purge the newest training bars would carry
//--- labels partly determined by price action inside the calibration slice - the same Lopez de Prado
//--- ch. 7 leak the OOS boundary already guards against, and it would put the memorization straight
//--- back into the curve this slice exists to keep clean.
int CalibPurgeBars ( void ) const { return ( int ) MathMax ( m_barrierHorizonBars , 1 ) ; }
int CalibLoIndex ( int oosCutoff ) const { return oosCutoff + CalibPurgeBars ( ) ; }
//--- Zero (an empty band) whenever the era is too short to carve one without eating the training set;
//--- callers must treat that as "no calibration this era" and leave the threshold where it is.
int CalibBandBars ( int totalIter , int oosCutoff ) const
{
int isSpan = totalIter - CalibLoIndex ( oosCutoff ) - CalibPurgeBars ( ) ;
if ( isSpan < = 0 )
return 0 ;
return ( int ) ( isSpan * ( DIR_CONF_CALIB_PCT_OF_IS / 100.0 ) ) ;
}
int CalibHiIndex ( int totalIter , int oosCutoff ) const
{ return CalibLoIndex ( oosCutoff ) + CalibBandBars ( totalIter , oosCutoff ) ; }
2026-07-23 19:36:34 -04:00
//--- EMA-updates the persisted true class base rates (m_priorBuy/Sell/Neutral) from a just-finished
//--- era's true class counts. No-op on an empty/degenerate tally.
void UpdateClassPriors ( long buyCnt , long sellCnt , long neutralCnt ) ;
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior
Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add
tau*log(prior_c) to each class logit inside the training gradient. Softmax
CE on adjusted logits is consistent for BALANCED error - the metric
checkpoint selection already ranks on - so the loss and the deploy decision
finally optimize the same thing.
The engine already computed a true softmax + categorical-CE gradient and
wrote it over the per-neuron sigmoid delta, so this is an offset added to
three logits in the two places that gradient is built (backProp scalar path
and backPropOCL). No backend, kernel or DLL change; the forward pass and
every inference path are untouched, which is the point - the network learns
to absorb the offset, so its raw argmax becomes the balanced-optimal
decision with nothing applied at inference.
Replaces rather than stacks. Minority replay is disabled while this is on,
and the post-hoc inference prior is forced off. Stacking is not a
theoretical worry: simulated on the measured 1118/1119/34298 distribution
in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced,
Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance -
and BOTH together score 45.4% with Neutral recall at 0%, worse than either
alone. Buda et al. 2018 predicts exactly that.
Motivation from the six-chart run: every topology took one direction to
~50% recall and abandoned the other, the direction chosen arbitrarily (the
batch-norm control went Buy 1% / Sell 42%, the inverse of the other five).
One era in 1,301 cleared the per-class recall floor.
Fingerprinted conditionally, so the converged 60.7% models on disk keep
their filenames and stay loadable as the fallback.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//--- Installs tau*log(prior_c) on Net from the freshly measured priors. Called once per era
//--- start, straight after UpdateClassPriors, so the offsets track the same distribution the
//--- era is scored against. No-op (and actively clears stale offsets) when the input is off.
void ApplyLogitAdjustment ( void ) ;
2026-07-23 19:36:34 -04:00
//--- Small binary sidecar (fileName + ".stats") persisting the calibration state that must survive a
//--- restart for live trading to behave like training: the true class priors and m_confidenceCalScale.
bool SaveModelStats ( string fileName , bool common ) ;
bool LoadModelStats ( string fileName , bool common ) ;
2026-07-24 11:52:19 -04:00
//--- Deploy-time (chart, backend present) self-check: runs the just-saved deployed model through both
//--- the backend and a temporary pure-MQL5 (CNet::SetCpuInference) clone on the same input window and
//--- returns true only if the outputs match within CPU_INFERENCE_MAX_DIFF. Gates whether an
//--- inference-only backtest may run DLL-free. Fails safe (returns false) on any error/mismatch or an
//--- architecture whose CPU path isn't ported yet - the caller then keeps the model on the DLL path.
bool ValidateCpuInference ( void ) ;
2026-07-25 01:07:21 -04:00
//--- Build the panel's "Buy/Sell accuracy: IS x% OOS y%" line (directional win-rate, Neutral excluded)
//--- from the cumulative counts (m_cumIsCorrect etc.); returns "...: measuring..." until at least one
//--- directional call has been validated. Shared by the training and live/complete simple panels.
2026-07-25 00:02:34 -04:00
string ComputeCompoundedAccuracyLine ( void ) ;
2026-07-24 11:52:19 -04:00
//--- Persist/restore the drawn directional arrows (the "WarSig_" objects) to a sidecar file so they
//--- survive an EA remove/re-add, recompile, or restart WITHOUT a retrain - the chart objects are
//--- destroyed on unload (destructor PurgeChart) and OnInit has no other way to bring them back.
//--- Stores each arrow's time/code/price and its hide state (OBJPROP_TIMEFRAMES), so the show/hide
//--- toggle is preserved too. Chart-only (a backtest has no persistent chart to restore to).
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
bool SaveChartSignals ( bool pruneChartObjects = true ) ;
2026-07-24 11:52:19 -04:00
void LoadChartSignals ( void ) ;
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
//--- The shutdown half of that pair: persist, THEN clear the chart, and report both counts. See the
//--- definition for why the order is fixed and why the clear is conditional on the write.
void PersistAndClearChartSignals ( void ) ;
//--- How many arrows the last successful SaveChartSignals() wrote - reporting only.
int m_lastArrowsSaved ;
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- One-shot latch for PurgeChart()'s "saved N but the chart holds none" warning. PurgeChart runs
//--- twice on a clean removal - once from the shutdown path and again from the destructor, which is
//--- deliberate (the destructor covers teardowns that never reach OnDeinit) - and the second call
//--- necessarily finds an already-emptied chart with m_lastArrowsSaved still set. Without this latch
//--- that harmless second pass would report the discrepancy every single time and train the reader
//--- to ignore the one case where it is real.
bool m_purgeMismatchWarned ;
fix: clear stale signal arrows when a fresh model starts at era 0
Arrow cleanup existed on two paths - the panel's reset-weights, and the
topology-mismatch discard - but both are gated on there being a saved .nnw to
delete. The third case had no cleanup at all: a fresh topology at era 0 with no
weights behind it, which is what a changed config produces. A new fingerprint
makes a new m_fileName, so the previous model's files are not "discarded", they
are simply not this model's files, and nothing ever cleared the chart.
That is not cosmetic. Arrows outlive the model that drew them twice over:
1. The chart objects live in the CHART, not the sidecar, so they survive a
remove/re-add, a recompile, a restart and a fresh deploy no matter what
happens to any file on disk.
2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for
SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the
dead model's calls and writes them out under the NEW model's filename -
laundering them into the new model's history where nothing can separate
them afterwards.
Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it
cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the
namespaced chart objects and logs why - and called it from all three paths.
The call sits at the BuildFreshTopology() call site, not inside it: the genetic
tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never
touch the chart. All three sites run after m_fileName has its config fingerprint
appended, so they target the right sidecar.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 10:13:01 -04:00
//--- Wipe this model's drawn arrows AND their .arrows sidecar, plus any deferred restore still in
//--- flight. Call from every path that discards or replaces the trained weights - see the definition
//--- for why leaving them behind resurrects a dead model's calls through SaveChartSignals.
void ClearPersistedChartSignals ( const string reason ) ;
2026-07-26 11:17:55 -04:00
//--- Deferred ("async") half of LoadChartSignals: LoadChartSignals only PARSES the sidecar into the
//--- m_arrowRestore* buffers (an ~80KB read - instant) and returns, so OnInit never blocks; this then
//--- creates the chart objects in ARROW_RESTORE_BUDGET_MS slices, driven by the same 500ms timer that
//--- already paces training. MQL5 has no threads - a chart runs one thread - so blocking OnInit is what
//--- made the terminal look frozen (no panel, no status label, no journal) while ~2900 arrows were
//--- rebuilt. Time-boxed slices give the terminal room to paint the UI between them instead.
void AdvanceChartSignalRestore ( void ) ;
//--- parsed-but-not-yet-drawn arrows, consumed by AdvanceChartSignalRestore (see above)
datetime m_arrowRestoreTime [ ] ;
int m_arrowRestoreCode [ ] ;
double m_arrowRestorePrice [ ] ;
long m_arrowRestoreTf [ ] ;
int m_arrowRestoreIndex ;
bool m_arrowRestorePending ;
uint m_arrowRestoreStartMs ;
2026-07-26 12:55:31 -04:00
//--- Deferred ("async") half of StartChartSignalRescan (public, defined inline further down): drains
//--- the per-bar inference loop in ARROW_RESTORE_BUDGET_MS slices off PollTraining's timer instead of
//--- blocking the button-click handler for however long a full lookback scan takes. Internal-only -
//--- called from PollTraining(), never from outside the class - so this stays protected while
//--- StartChartSignalRescan()/RescanPending() (which the panel button needs) are public.
2026-07-26 12:52:56 -04:00
void AdvanceChartSignalRescan ( void ) ;
int m_rescanIndex ;
int m_rescanHi ;
int m_rescanBarsNow ;
bool m_rescanPending ;
uint m_rescanStartMs ;
2026-07-26 13:58:05 -04:00
//--- Raw (PRE prior-correction) argmax tally, accumulated per-bar across AdvanceChartSignalRescan's
//--- slices - lets the completion log distinguish "the network itself calls Neutral almost everywhere"
//--- from "the network still discriminates, but AdjustedSignalFromSoftmax's logit-prior correction is
//--- suppressing it down to Neutral" - both produce an identical all-Neutral m_arrowSignalCache/empty
//--- chart otherwise.
int m_rescanRawBuy ;
int m_rescanRawSell ;
int m_rescanRawNeutral ;
2026-07-14 22:36:27 -04:00
bool ResizeBuffers ( int barIndex ) ;
bool RefreshData ( ) ;
research: export the feature matrix and a raw OHLCV grid for offline work
The bottleneck on this project has never been the modelling - it is that
every hypothesis costs a compile, a deploy, an attach and a log read, and
answers exactly one question. Days have gone into questions that are
seconds of arithmetic once the data is in hand.
Adds a RESEARCH-ONLY build, gated behind WARRIOR_EXPORT_FEATURES and
never compiled into a shipped binary, which writes two things to
Common\Files\Warrior_EA\Research\ and then does nothing at all:
<symbol>_<tf>_features.csv - one row per bar: index, time, OHLC, ATR,
and the m_neuronsCount feature values. Exactly what the network sees.
The raw bars ride along on purpose: with OHLC and ATR offline, every
barrier geometry, horizon and in-trade target is recomputable without
MetaTrader in the loop.
<symbol>_<tf>_rates.csv - raw OHLCV across a grid of 8 symbols x 5
timeframes. The 26 engineered features only exist for the attached
chart (indicator handles bind to PERIOD_CURRENT); raw rates do not, so
ONE attach yields the whole research grid. The bar time also makes
session/hour/day-of-week derivable - the only inputs in play that are
not a transform of the same OHLCV series.
Safety, because this binary gets attached to a chart on a LIVE ACCOUNT to
reach real history:
- OnTick returns immediately, so Expert.OnTick() - the entire trading
path - is unreachable regardless of the AlgoTrading toggle, the
signal state or the inputs. Structurally incapable of sending an
order, not merely unlikely to.
- No config lock. It never trains and never saves a model, so it has
nothing to protect against a concurrent chart - and taking the lock
would make it refuse to start exactly when the config it wants to
read is already open, which is when it is most useful.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 15:49:57 -04:00
# ifdef WARRIOR_EXPORT_FEATURES
//--- RESEARCH BUILD ONLY, never compiled into a shipped binary. Dumps exactly what the network sees -
//--- one row per bar: index, time, OHLC, ATR, then the m_neuronsCount feature values - to a CSV under
//--- Common\Files\Warrior_EA\Research\. Exporting the RAW BARS alongside the features is the point:
//--- with OHLC+ATR in hand every barrier geometry, horizon and in-trade target can be recomputed
//--- offline, so a research question costs seconds in Python instead of a compile/attach/read cycle.
void ExportFeatureMatrix ( void ) ;
//--- Raw OHLCV for a grid of symbols/timeframes - see the definition for why the grid is worth more
//--- than the engineered features on their own.
void ExportRawRates ( void ) ;
# endif
2026-07-14 22:36:27 -04:00
bool BufferTempData ( int idx ) ;
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
//--- Assembles the full m_historyBars-wide input window ending AT bar r into TempData, OLDEST BAR
//--- FIRST. Use this everywhere instead of hand-rolling the loop: the chronological order is load-
//--- bearing for the LSTM/HYBRID stacks and cannot be enforced by convention across eight call
//--- sites. See the definition comment in AIBase\Features.mqh for the measurement behind that.
bool BuildFeatureWindow ( int r ) ;
2026-07-14 22:36:27 -04:00
//--- shared by OnTickHandler() and the timer-driven PollTraining() - see definition
void ScheduleTrainingIfNeeded ( void ) ;
void Train ( datetime StartTrainBar = 0 ) ;
fix: a restart no longer loses the measured geometry or the training window
Terminal restart, 22:25: all four resumed models sat on empty windows with
enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt:
1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's
dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the
resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as
"Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to
the training-window rule before computing its window; the pre-scan did not.
The rule is now factored into TrainWindowStart() and both use it. The scan
also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as
OnInit), and deployed models keep their watermark - for them it gates
inference recency, not a training window.
2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran
against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon
latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A
leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the
label cache wiped if it moved (labels from two horizons answer different
questions), and the geometry deriver refuses to run from it - a pair derived
over a warm-up window would get PINNED.
3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model
creation and at weights-reset - both BEFORE era 0 derives - so the measured
pair lived only in memory: every restart read back zeros, adopted nothing,
fell back to the enum barriers, and the era-0-only gate meant a resumed model
could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6.
Now: the settled pair is pinned to the .cfg the moment derivation completes
(one-shot, atomic write), and the derive gate accepts any model with no
pinned pair, not just era 0 - mid-run stability is carried by
m_geometryDerived itself, which never allows a second derivation.
Both build variants compile 0 errors, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//--- the training window's start time - shared by Train()'s era start and the label-cache pre-scan
datetime TrainWindowStart ( datetime startTrainBar ) ;
2026-07-14 22:36:27 -04:00
//--- outer loop around Train(): when AutoTuneIndicators is on, tries randomized AD indicator
//--- input variations across m_indicatorTuneTrials calls to Train(), keeping the best-OOS one
void TuneIndicatorsAndTrain ( datetime StartTrainBar = 0 ) ;
fix: live inference queried the 1-tick forming bar - a window training never built
RefreshLatestSignal ran at the first tick after a bar opens and built its
window at r=0: series index 0 at that instant is a candle with one tick of
data - (close-open)/atr ~ 0, high ~ low, degenerate volume, indicators on a
1-tick bar. Training never produces such a window (every labeled bar is fully
closed, entry at that bar's CLOSE), so the deployed model's final timestep -
the one the LSTM/HYBRID output is keyed to - was out-of-distribution on every
live decision, and pass 3's deploy-gate OOS scores measured a different query
than live executed. The parity index is r=1: the newest CLOSED bar, whose
close IS the current price - the exact instant the label's hypothetical entry
happens. Single backtests shared the old skew (same r=0), which is why the
tester agreed with live while both disagreed with training.
Bookkeeping split that the index change forces: m_lastBarTime/dtStudied stay
anchored to the FORMING bar's open (they gate against SERIES_LASTBAR_DATE;
anchoring at bar 1 would re-fire the refresh every tick), while bt - the
arrow, its High/Low placement, and NMS declustering - anchors to the decision
bar, now matching the rescan path's convention.
Also: a failed refresh no longer trades the previous bar's signal for the
whole bar. RefreshLatestSignal returns success, zeroes dPrevSignal on failure
(no opinion beats a stale one), and RefreshConvergedSignal advances dtStudied
only on success so the next tick retries - the tester path (m_lastBarTime)
already worked this way; this is the live path catching up.
FORCES RE-VALIDATION of deployed models: the effective live query distribution
changes. Bundled with the backprop transpose fix's retrain.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:10:23 -04:00
//--- recomputes dPrevSignal/chart arrow for the newest CLOSED bar (bar 1 - see the definition's
//--- 2026-08-11 parity comment); used after restoring a checkpointed model at the end of Train()
//--- so the live signal matches the deployed weights. Returns false when the window failed to
//--- build (dPrevSignal is zeroed, caller should not advance its new-bar watermark).
bool RefreshLatestSignal ( ) ;
2026-07-15 21:47:37 -04:00
//--- inference-only "new bar" handler used once m_trainingComplete is true - see
//--- ScheduleTrainingIfNeeded()'s declaration comment for why this must NOT call Net.backProp()
void RefreshConvergedSignal ( void ) ;
2026-07-24 11:52:19 -04:00
//--- Online continual-learning step (live chart only) - see its implementation comment and the
//--- ONLINE_LEARN_* tunables. Backprops the deployed Net on bars whose ZigZag label has just become
//--- CONFIRMED (m_swingConfirmationBars matured), then blends the shadow under a rolling-accuracy
//--- guardrail. No-op in the tester/optimizer (m_inferenceOnly) and while training is active.
void OnlineLearnStep ( void ) ;
2026-07-29 00:03:54 -04:00
//--- Alpha-balanced focal sample weight (Lin et al. 2017 eq. 5) for ONE streamed bar - see the
//--- ONLINE_LEARN_* block's CLASS IMBALANCE comment for the derivation. Shared deliberately by
//--- OnlineLearnStep() (the live path) and AdvanceOosSimulationChunk() (the simulation of that
//--- path): the simulation's reported accuracy is only a valid forecast of live continual-learning
//--- behaviour if it optimises the IDENTICAL objective, so the formula must exist in exactly one
//--- place. p* are this bar's pre-update softmax probabilities, already normalised in place by
//--- ApplyClassificationSoftmax(). Returns 1.0 for the regression head (no class structure).
double OnlineSampleWeight ( ENUM_SIGNAL trueSignal , double pBuy , double pSell , double pNeutral ) ;
2026-07-15 21:47:37 -04:00
//--- lazily bootstraps m_shadowNet if it's still NULL: tries loading a persisted shadow file
//--- first (continuity across EA restarts), falling back to cloning Net's current weights (via
//--- the same Save()/Load() pattern StartOosContinualSimulation() uses for m_simOosNet) if no
//--- compatible shadow file exists yet. No-op if m_shadowNet is already valid. See m_shadowNet's
//--- declaration comment for the full EMA shadow-weight deployment rationale.
void EnsureShadowNet ( void ) ;
//--- persists m_shadowNet alongside every Net.Save() call, using the same run metadata (error/
//--- undefine/forecast/era/trainingComplete/indicator params) the caller already computed for
//--- Net.Save() itself - see m_shadowNet's declaration comment. No-op if the shadow isn't
//--- bootstrapped yet.
void SaveShadowNet ( const double & indicatorParams [ ] ) ;
2026-07-14 22:36:27 -04:00
//--- method of initialization of the indicators
bool InitOpen ( CIndicators * indicators ) ;
bool InitClose ( CIndicators * indicators ) ;
bool InitHigh ( CIndicators * indicators ) ;
bool InitLow ( CIndicators * indicators ) ;
bool InitVolumes ( CIndicators * indicators ) ;
bool InitTime ( CIndicators * indicators ) ;
//--- addToCollection=false is used by ReInitADIndicators() to rebuild an already-collected
2026-07-22 22:51:04 -04:00
//--- handle's params (here: a re-tuned period) without re-adding the (same) pointer into
//--- indicators a second time
bool InitMA ( CIndicators * indicators , bool addToCollection = true ) ;
bool InitRSI ( CIndicators * indicators , bool addToCollection = true ) ;
2026-07-26 18:33:12 -04:00
bool InitMACDFeature ( CIndicators * indicators , bool addToCollection = true ) ;
bool InitIchimoku ( CIndicators * indicators , bool addToCollection = true ) ;
2026-07-14 22:36:27 -04:00
bool InitADCumulativeDelta ( CIndicators * indicators , bool addToCollection = true ) ;
bool InitADShorteningOfThrust ( CIndicators * indicators , bool addToCollection = true ) ;
bool InitADWyckoffEventStream ( CIndicators * indicators , bool addToCollection = true ) ;
bool InitADWyckoffFailedStructure ( CIndicators * indicators , bool addToCollection = true ) ;
bool InitADWyckoffSignificantBarInversion ( CIndicators * indicators , bool addToCollection = true ) ;
2026-07-16 00:56:33 -04:00
bool InitADZigZag ( CIndicators * indicators , bool addToCollection = true ) ;
2026-07-14 22:36:27 -04:00
//--- common=false targets a LOCAL (non-shared) file - used by the tester/optimizer per-agent
//--- weight cache so cross-pass reuse never touches the production FILE_COMMON config/weights.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
bool SaveTopologyConfiguration ( string fileName , int initialNeuronsCount , int hiddenLayersCount , double neuronsReduction , int minNeuronsCount , int optimizationAlgo , int historyBars , int outputNeuronsCount , int neuronsCount , int studyPeriod , int minTrainYear , bool isInitialized , int stopTrainWR , int fractalPeriods , int convFilterCount , int lstmHiddenSize , bool common = true ) ;
//--- The four DERIVED shape fields are by REFERENCE and are ADOPTED from the .cfg, not compared
//--- against it. See the block in the definition for why a derived value must never be able to
//--- mismatch: it is measured from data that legitimately changes, and a mismatch here discards
//--- a trained model. studyPeriod left the parameter list entirely - the input is gone; its
//--- on-disk slot is still read positionally and ignored, like the retired MinWR slot.
2026-08-11 21:53:37 -04:00
bool LoadAndCompareTopologyConfiguration ( string fileName , int & initialNeuronsCount , int & hiddenLayersCount , double neuronsReduction , int minNeuronsCount , int optimizationAlgo , int & historyBars , int outputNeuronsCount , int neuronsCount , int minTrainYear , bool isInitialized , int stopTrainWR , int fractalPeriods , int & convFilterCount , int & lstmHiddenSize , bool common = true ) ;
2026-07-26 10:15:56 -04:00
//--- Retry helpers for the tester/opt seed-copy race: a live chart's own atomic Save() (write .savetmp,
//--- then FileMove() over the real file) can hold the source or destination file for a moment, and a
//--- concurrent FileCopy/FileOpen from a Strategy Tester agent reading the SAME production file can hit
//--- a transient Windows sharing violation in that narrow window. Both retry a handful of times with a
//--- short pause rather than silently treating a transient lock as "no model"/"corrupt file" - see their
//--- call sites in InitNeuralNetwork.
bool CopyFileWithRetry ( string srcFileName , string dstFileName ) ;
2026-07-26 10:46:19 -04:00
bool CopySharedFile ( string srcFileName , string dstFileName , bool quiet ) ;
2026-07-26 10:15:56 -04:00
bool LoadNetWithRetry ( double & indicatorParams [ ] ) ;
2026-07-14 22:36:27 -04:00
//--- input data
bool m_useVolumes ;
bool m_useTime ;
bool m_useATR ;
2026-07-22 22:51:04 -04:00
//--- Uses its own period (m_indicatorTuner.maPeriod), fed as ATR-normalized OHLC distance-from-MA
//--- (4 values, same convention as the base close-open/high-open/low-open features) plus the MA's
//--- own bar-over-bar change (1 value, ATR-normalized like every other price-domain feature here -
//--- not volume's previous-bar-ratio scheme, since a moving average lives in price units and
//--- already has ATR as its natural scale reference). See BufferTempDataCompute()'s m_useMA block
//--- for the exact 5 values. maPeriod starts equal to the Classic Signals PeriodMA input (see
//--- CADIndicatorTuner's constructor) but may diverge from it once AutoTuneIndicators searches a
//--- trial - the Classic Signals MA vote itself is untouched by that search, since it needs no
//--- training/warm-up and there is nothing for a tuning trial to validate it against.
2026-07-22 17:17:23 -04:00
bool m_useMA ;
//--- RSI is already a 0-100 oscillator, so the only transform needed is /100 to match every other
//--- feature's roughly [-1,1]/[0,1] scale - no ATR or distance normalization applies. See
2026-07-22 22:51:04 -04:00
//--- BufferTempDataCompute()'s m_useRSI block for the exact value. Same maPeriod/rsiPeriod
//--- divergence-from-the-Classic-Signals-input note as m_useMA above applies to rsiPeriod.
2026-07-22 17:17:23 -04:00
bool m_useRSI ;
2026-07-26 18:33:12 -04:00
//--- MACD as 3 ATR-normalized values (main line, signal line, histogram) - see
//--- BufferTempDataCompute()'s m_useMACD block. Deliberately kept to 3 despite MACD being cheap: the
//--- point of adding it next to m_useMA is the SECOND timescale (m_useMA supplies exactly one moving
//--- average) and the histogram, which is the only acceleration term anywhere in the feature vector -
//--- the level information itself is already covered by the MA distances. ATR-normalized rather than
//--- left raw because the MACD lines live in price units, exactly like the MA feature.
bool m_useMACD ;
//--- Ichimoku as 8 values - see BufferTempDataCompute()'s m_useIchimoku block for each. This is the
//--- widest single classic-indicator feature here and it earns that width by carrying multi-timescale
//--- support/resistance GEOMETRY (three lookbacks plus a forward-projected cloud) that nothing else in
//--- the vector encodes: the swing-context block's confirmed pivots are >=m_swingConfirmationBars bars
//--- stale by construction, and its Donchian/SMA values are single-scale.
//--- LOOKAHEAD - MT5's iIchimoku stores raw per-bar values and shifts only the DRAWING, so the cloud
//--- sitting under bar idx is SenkouSpan*(idx + ichiKijun), and the Chikou value plotted at bar idx
//--- would be Close(idx - ichiKijun) - a FUTURE bar. The feature block applies the +Kijun offset and
//--- never calls ChinkouSpan(); Signals\SignalIchimoku.mqh's class comment documents the buffer
//--- convention in full, and the same reasoning governs both.
bool m_useIchimoku ;
2026-07-20 19:17:23 -04:00
//--- Normalized ZigZag swing-context features: 5 confirmed-pivot values (direction/magnitude/age of
//--- the last CONFIRMED swing) plus 4 recent-price-action values (Donchian range position at 20/50
//--- bars, 20-bar return, 20-bar SMA extension) that give fresh, non-repainting trend/position
//--- context the >=100-bar-stale pivot anchor can't - see BufferTempDataCompute()'s m_useSwingContext
//--- block for the exact 9 values. Reads the same
2026-07-19 11:04:38 -04:00
//--- m_ADZigZag the training labels already come from (see that member's declaration comment) rather
//--- than a separate indicator instance - NOT gated behind AutoTuneIndicators/ReInitADIndicators like
//--- the AD* indicators below, since m_ADZigZag itself is deliberately never tuned (same reason as
//--- the label side: tuning the ground truth alongside the model scored against it would let a trial
//--- cherry-pick an easier target). Critical correctness constraint, not just a style choice: ZigZag's
//--- most recent 1-3 legs repaint (see m_swingConfirmationBars' declaration comment), so every read of
//--- m_ADZigZag for THIS feature - exactly like the label side - must only trust a pivot that is
//--- already at least m_swingConfirmationBars bars old relative to the bar the feature is being
//--- computed for. Skipping that embargo would leak future information a live bar couldn't actually
//--- have had yet - silent lookahead bias inflating backtest/training performance without being real.
bool m_useSwingContext ;
feat: add configurable news event proximity/impact as an NN input feature
Price, time, volume, and volatility were already trained-model input
features; the real economic calendar (already used for the live
NewsFilter veto) is now an optional one too, reusing
System/NewsRelevance.mqh's symbol-relevance logic from the prior fix.
New EnableNews/NewsFeatureWindowMinutes inputs gate two features per
bar: minutes-since and minutes-until the nearest symbol-relevant
calendar event, impact-weighted. Deliberately limited to proximity +
impact, not actual-vs-forecast deviation - release schedules are
public knowledge ahead of time (not lookahead bias to use for a
historical training bar), but a release's actual outcome is not.
Wired identically to the existing EnableVolume/EnableTime/EnableATR
toggles: InitIndicators() accounts for the +2 neuron count,
BufferTempDataCompute() appends the two feature values, PAI/CONV/LSTM
all wired in Warrior_EA.mq5. Compiled clean (MetaEditor, 0 errors/0
warnings).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 17:26:04 -04:00
//--- see System\NewsRelevance.mqh's declaration comment for what this feature actually encodes
//--- (event proximity + impact, not actual-vs-forecast deviation) and why the forward-looking half
//--- of it isn't lookahead bias.
bool m_useNews ;
int m_newsFeatureWindowMinutes ;
fix(signals): revive a dead MA model, and demote Sanyaku from state to event
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
//--- Cross-asset panel: the only feature block here whose inputs are NOT a transform of this
//--- symbol's own OHLCV series. See System\CrossAsset.mqh for the reasoning; in short, every other
//--- feature the network sees is a function of one price series, and that whole family measured at
//--- the noise floor, so the panel exists to give it information a single series cannot contain.
//--- Built ONCE per training run (BuildCrossAssetPanel) rather than per bar - a per-bar cross-symbol
//--- lookup would be pairs x bars iBarShift calls.
bool m_useCrossAsset ;
CCrossAssetPanel m_crossAsset ;
bool BuildCrossAssetPanel ( int bars ) ;
2026-08-11 21:29:14 -04:00
//--- Train->serve parity for the panel (2026-08-11): the pair set is a MEASURED property of the
//--- terminal, so like the derived barrier pair it is pinned in the .cfg, not the filename hash
//--- (see BuildConfigFingerprint's XA note). Empty = no set pinned yet; first successful Build
//--- stamps it and re-saves the .cfg one-shot, exactly the m_geometryCfgSaved pattern.
string m_crossAssetPairsPinned ;
bool m_crossAssetCfgSaved ;
2026-08-16 13:39:00 -04:00
//--- Alternative-data panel (2026-08-16): the second feature block whose inputs are not a
//--- transform of this symbol's own series, and the first whose inputs are not derivable from
//--- the terminal at all - COT positioning, the VIX complex, macro series, collected and
//--- publication-stamped by research/altdata, served as plain CSVs. See System\AltData.mqh for
//--- the lookahead/degradation/pinning contracts. No use-input: file present = on (the feature
//--- SET is a measured property, like the derived barrier pair), .cfg name-list pin keeps a
//--- trained model's inputs meaning what they meant.
bool m_useAltData ;
2026-08-16 15:12:54 -04:00
//--- EnableAltData input, distinct from m_useAltData: the input says the OPERATOR wants the block,
//--- m_useAltData says it is actually contributing features (input on AND file present AND >=1
//--- column). Gating consumption only - collection (AltDataFetch) runs regardless, so the files
//--- stay current for the day the operator turns it back on.
bool m_altDataEnabled ;
2026-08-16 20:04:13 -04:00
//--- One-shot guard for the "data landed after the model was pinned" warning - the upkeep tick
//--- runs every 30 minutes and this must not become a recurring line nobody reads.
bool m_altDataLateWarned ;
2026-08-16 13:39:00 -04:00
CAltDataPanel m_altData ;
string m_altDataNamesPinned ;
string ReadAltDataPinFromCfg ( void ) ;
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks
Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature,
default on). Spread is the one microstructure channel that is both FX-available and
genuinely historical in the Strategy Tester - "during testing, the spread is not modeled
but is taken from historical data" - so unlike swap, signed tick flow or depth of market it
is something a backtest can honestly validate.
What it encodes, stated precisely because the raw measurement overstates it.
research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5
of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges
the spread inside its own barriers, so a wide-spread bar is mechanically likelier to
resolve as a loss and the feature would partly be predicting its own cost model. Relabelling
at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology
and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime
reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when
realised volatility is below its own ATR estimate, which genuinely predicts whether
ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side.
Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated
in the spread series. Both cached on length alone:
if(m_crossAsset.Bars() >= bars) return true;
MQL5 series indices are relative to NOW, so one new closed candle shifts every index by
one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer
the newest, and every cross-asset value is read one bar out of step with the price features
sitting beside it in the same vector - silently, with no error and no shape change. This is
the same class of defect as the dtStudied watermark behind the zero-direction backtests.
Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the
label/feature bar caches already use.
And a performance fix that fell out of it: with correct invalidation the panel rebuilds on
every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one
full multi-symbol resample per simulated bar at training depth. Inference only reads bars
0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The
cache check is >=, so a deeper panel left from training still satisfies it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
//--- Spread as a feature. The only microstructure channel that is BOTH available on FX and
//--- genuinely historical in the Strategy Tester ("during testing, the spread is not modeled but
//--- is taken from historical data") - unlike swap (no history at all), signed tick flow
//--- (TICK_FLAG_BUY/SELL are empty on FX) or depth of market (absent on retail FX, never replayed).
//--- Measured as the strongest single feature in research/test_spread.py, though see the feature
//--- block for what it actually encodes and why that is less than it first appears.
//--- Series is copied ONCE per bar grid, not per bar: CopySpread is a range call, not a lookup.
bool m_useSpreadFeature ;
int m_spreadSeries [ ] ;
int m_spreadSeriesBars ;
//--- Newest bar the copy was anchored to. MQL5 series indices are relative to NOW, so a single
//--- new closed candle shifts every index by one: a cache keyed only on length would keep serving
//--- index 0 as a bar that is no longer the newest, silently misaligning the spread series against
//--- the price buffers it must line up with. Same invalidation key the label/feature bar caches
//--- use (see EnsureBarCachesCapacity) and the same failure the zero-direction hunt traced.
datetime m_spreadSeriesAnchor ;
datetime m_crossAssetAnchor ;
bool EnsureSpreadSeries ( int bars ) ;
2026-07-14 22:36:27 -04:00
bool m_useADCumulativeDelta ;
bool m_useADShorteningOfThrust ;
bool m_useADWyckoffEventStream ;
bool m_useADWyckoffFailedStructure ;
bool m_useADWyckoffSignificantBarInversion ;
public :
CExpertSignalAIBase ( void ) ;
~ CExpertSignalAIBase ( void ) ;
2026-08-16 13:59:03 -04:00
//--- Reload the alt-data panel after CAltDataFetch rebuilt the feature CSV (OnTimer path,
//--- live only). Content-only refresh: the .cfg name-list pin fixes the width and column
//--- meanings, so new rows can arrive but the input contract cannot drift. Safe against the
//--- per-bar feature cache because new alt rows only ever matter to a NEW D1 bar, which
//--- resets that cache anyway.
void AltDataReload ( void )
{
2026-08-16 20:04:13 -04:00
//--- Gated on the OPERATOR's switch, NOT on m_useAltData. m_useAltData latches false at init
//--- whenever the CSV was absent, so gating the reload on it made the EA structurally unable to
//--- consume data IT HAD JUST DOWNLOADED: on the first run after the alt-data folder is wiped -
//--- the normal pre-test routine here - the models are built ~30s BEFORE the fetch completes, the
//--- reload became a permanent no-op, and the entire run trained on price alone while a complete
//--- feature file sat on disk. Measured 2026-08-16 on SP500 H4: models pinned at fingerprint
//--- 6de8ba37 (0 alt features) at 19:36:40, SP500_D1.csv rebuilt with 13 features at 19:37:13,
//--- and every era after that trained without them - silently, because nothing looked again.
if ( ! m_altDataEnabled )
return ;
int before = m_altData . FeatureCount ( ) ;
m_altData . Load ( m_symbol . Name ( ) , ( ENUM_TIMEFRAMES ) m_period ) ;
int after = m_altData . FeatureCount ( ) ;
//--- Loading here is INERT while m_useAltData is false (both consumption sites gate on it), so
//--- this cannot widen the feature vector out from under a model whose width is already pinned.
//--- That pinning is exactly why arriving data cannot be adopted mid-run, and why the only
//--- honest response is to say so once, loudly, instead of leaving the run looking healthy.
if ( before = = 0 & & after > 0 & & ! m_altDataLateWarned )
{
m_altDataLateWarned = true ;
Print ( ID + " : ALT DATA ARRIVED AFTER THIS MODEL WAS BUILT - " + IntegerToString ( after ) +
" features are on disk now, but this model's input width was pinned WITHOUT them, so it "
" is training on price alone and will keep doing so for the rest of this run. "
" RE-ATTACH THE EA (or reload the chart) to build models that actually train on the "
" alt-data block. This is what happens when the alt-data folder is empty at attach time "
" and the EA downloads it moments later. " ) ;
}
2026-08-16 13:59:03 -04:00
}
2026-07-14 22:36:27 -04:00
//--- "voting" that price will grow/fall, common to every AI signal (single market model)
virtual int LongCondition ( void ) ;
virtual int ShortCondition ( void ) ;
// |dPrevSignal| is already a 0..1 confidence for classification output (softmax
// probability of the winning class) and typically bounded for regression output
2026-07-17 23:21:12 -04:00
// (tanh-activated network); OpenParams() clamps regardless. Scaled by m_confidenceCalScale
// (classification head only - 1.0/no-op for regression) so callers get an empirically
// calibrated magnitude instead of the raw, uncalibrated softmax value - see
// m_confidenceCalScale's declaration comment.
double CalibratedConfidenceMagnitude ( void ) const
{
double mag = MathAbs ( dPrevSignal ) ;
2026-07-27 15:52:39 -04:00
if ( ! MathIsValidNumber ( mag ) )
return 0.0 ;
2026-07-17 23:21:12 -04:00
if ( m_outputNeuronsCount = = 3 )
mag = MathMin ( 1.0 , mag * m_confidenceCalScale ) ;
2026-07-27 15:52:39 -04:00
if ( ! MathIsValidNumber ( mag ) )
return 0.0 ;
2026-07-17 23:21:12 -04:00
return mag ;
}
virtual double AIConfidence ( void ) override { return CalibratedConfidenceMagnitude ( ) ; }
2026-07-14 22:36:27 -04:00
// Signed for direction-aware use (AI-driven early exit): sign matches dPrevSignal's
// convention (+ buy, - sell, 0 neutral/no signal yet). dPrevSignal == -2 is the
// "not yet studied" sentinel, not a real sell signal - treat it as no confidence.
2026-07-17 23:21:12 -04:00
virtual double SignedAIConfidence ( void ) override
{
if ( dPrevSignal = = -2 )
return 0.0 ;
double sign = ( dPrevSignal > 0.0 ) ? 1.0 : ( dPrevSignal < 0.0 ) ? -1.0 : 0.0 ;
2026-07-27 15:52:39 -04:00
if ( sign = = 0.0 )
return 0.0 ;
2026-07-17 23:21:12 -04:00
return sign * CalibratedConfidenceMagnitude ( ) ;
}
2026-07-14 22:36:27 -04:00
//--- event handlers, common to every AI signal
virtual void OnTickHandler ( void ) ;
//--- drives the same training-scheduling check as OnTickHandler(), but callable from a timer so
//--- it isn't dependent on ticks (which don't arrive while the market is closed)
void PollTraining ( void ) ;
virtual void OnChartEventHandler ( const int id ,
const long & lparam ,
const double & dparam ,
const string & sparam ) ;
2026-07-23 08:21:41 -04:00
//--- methods of adjusting "weights" of the 4 confidence-tier market models - see m_pattern_0's
//--- declaration comment
2026-07-14 22:36:27 -04:00
void Pattern_0 ( int value ) { m_pattern_0 = value ; }
2026-07-23 08:21:41 -04:00
void Pattern_1 ( int value ) { m_pattern_1 = value ; }
void Pattern_2 ( int value ) { m_pattern_2 = value ; }
void Pattern_3 ( int value ) { m_pattern_3 = value ; }
2026-07-14 22:36:27 -04:00
virtual void ApplyPatternWeight ( int patternNumber , int weight ) ;
feat(rank): AI models rank their own confidence tiers from held-out outcomes
Closes the caveat 4858507 shipped with: the vote is a confidence percentage,
but only to the extent the pattern weights are measured. AI tier weights sat
at their designed defaults (25/50/75/100) because AI rows only ever arrive
from LIVE journaling, of which a training run produces almost none.
AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed
every scanned bar by ConfidenceTier(), which reads dPrevSignal - and
dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an
entire era's fires were bucketed by one stale, unrelated bar's confidence and
landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0)
T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the
calibration clamp. The clamp was real and was fixed then; this is a second,
independent cause of the identical output that survived that fix untouched -
which is why the log kept reading the same afterwards. Two causes, one symptom.
Now ConfidenceTierFor(adjSig): the bar this iteration actually scored.
WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of
"fill the database during training". The user's own observation is the reason:
a classic Pattern_2 is a fixed geometric condition, so its win rate is
legitimately accumulated over years, but an AI Pattern_2 means "confidence
landed in tier 2" and tier 2 under era 100's weights is a different statement
from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation
is exactly what is wrong here - it would average together models that no
longer exist, while colliding with the per-table row cap and mixing
measured-on-holdout outcomes into the live ledger's own tables. What the DB
actually supplies is a measured win rate per pattern, and pass 3 already
computes that on held-out bars, thousands at a time. So the model ranks itself
once per era, REPLACING rather than accumulating, which makes the weights
describe the current weights by construction.
ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades
BEFORE shrinking, which here would fire on every tier every era and hand all
four the pooled rate - the tiers could never separate and the mechanism would
be inert. Shrinkage is the answer to a small sample; a floor in front of it
means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N
pseudo-observations centred on the model's pooled holdout rate, counted in
EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw
fires can be worth ~12 independent ones. Rounded to the integer, not to the
decade NormalizeWinRate() uses, which would collapse the shrunk tiers back
into one number.
NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard:
weights are computed at the END of era N, so the vote scored during era N was
cast with era N-1's weights. The deploy gate never grades a vote whose weights
were fitted on the bars it is scoring. Residual leakage remains - the same OOS
bars each era under a different model - and is stated in the code rather than
papered over.
Both DB clobber paths are closed: ApplyPatternWeight() declines once
self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by
SelfRanked() - guarding only the tiers would have let the hourly ranking pass
undo half the self-ranking.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- Re-derives the four tier weights (and the module weight) from THIS era's held-out outcomes.
//--- Called once per era at the end of pass 3, when m_oosTierFired/Hits are complete.
void RankTiersFromOos ( void ) ;
//--- Direct tier setter, deliberately NOT routed through ApplyPatternWeight(): that override
//--- declines writes once self-ranking is live, which is exactly what must not happen to the
//--- self-ranker's own writes. Two doors, because they serve opposite purposes.
void ApplyTierWeight ( const int tier , const int weight )
{
switch ( tier )
{
case 0 : Pattern_0 ( weight ) ; break ;
case 1 : Pattern_1 ( weight ) ; break ;
case 2 : Pattern_2 ( weight ) ; break ;
default : Pattern_3 ( weight ) ; break ;
}
}
//--- True once this model has measured its own tier win rates on held-out bars. While true the
//--- signal DB's ranking is declined for this filter - see ApplyPatternWeight's comment and
//--- CExpertSignalCustom::SelfRanked().
virtual bool SelfRanked ( void ) const override { return m_tiersSelfRanked ; }
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials
Era-680 report, all three observations one equation: "peak 29, no arrows at
threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows
everywhere". Under the voters-only divisor, any bar with at least one
directional voter read the weighted mean of the firing tiers' weights - and
once the tiers self-ranked to each model's pooled win rate (~28-31), that
mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four
unanimous: ~29. Min_Vote_Open was a step function around that constant -
above it nothing ever fired, below it everything did - and the label's 12
was a 3v1 split netting through the same divisor. Not three display bugs:
one arithmetic that could not express agreement.
The divisor is now the CAPABLE weight - every filter that could vote,
whether it did or not:
* live (Direction): VoteCapableWeight() - classic pattern ladders always,
veto filters never, AI members once past the same readiness test
LongCondition gates on. A model still training must not dilute an
ensemble it cannot join: four trainees + one deployed model is a solo
chart wearing an ensemble label, and the solo vote reads full strength.
* gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every
member that EVALUATED the bar, Neutral included.
* overlay sweep + prospective readout: weight counts whenever the member
has data; a snapshotted Neutral dilutes.
One arithmetic, four sites, same numbers everywhere.
What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 -
the CEILING, which is the pooled win rate and is what the peak displays;
3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly
three-quarters of the ensemble's trust agrees, net". It MUST sit below the
ceiling to ever fire - the census/peak states the ceiling.
This is the ensemble the user specified in the original design discussion
("if the perceptron also votes, both together reach the threshold; if
another NN votes the other side, the threshold is not reached") - union
semantics was the pre-ensemble behaviour, kept until measurement showed its
vote magnitude was a constant.
Plus overlay DECLUSTERING, the other half of "arrows on every bar": the
same three NMS rules as the per-member arrows (same-direction runs collapse
to their first bar, cross-direction flicker keeps the stronger side), online
over the sweep's strictly oldest->newest walk. Suppression is a verdict and
deletes a standing arrow; the den==0 no-data skip still never does.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
//--- An AI member's say in the consensus denominator: its module weight once it is ALLOWED to
//--- vote (the same readiness test LongCondition gates on), zero before that. A model still
//--- training must not dilute the ensemble it cannot join - four training members and one
//--- deployed one is a solo chart wearing an ensemble label, and the solo vote must read at
//--- full strength.
virtual double VoteCapableWeight ( void ) override
{
if ( ! m_trainingComplete & & ! ( m_inferenceOnly & & m_modelLoadedFromDisk ) )
return 0.0 ;
return ModuleWeight ( ) ;
}
feat(chart): reconstruct the filtered view behind the handover point
Completes the filtered view from 282b535, which only reached forward of
attach. On a multi-hour training run that is the entire time you are looking
at the chart, so the answer to "how would the whole bot have traded" was
blank exactly when it was wanted.
The sweep lives on the AGGREGATE signal, which is the only object holding
every filter. AI members contribute their CACHED per-bar decision from the
era scan - no inference re-runs, the cache already spans the chart - and the
classic ladders are replayed with EvalShift(i), the same mechanism
CSignalMETA's candidate sweep uses and exact because every classic pattern
condition anchors on StartIndex(). Combination is the live one: weighted mean
over voting filters, abstentions out of both sums, against Min_Vote_Open.
THE REPLAY CORRUPTS LIVE JOURNALING IF LEFT UNGUARDED, and this is the part
that is not obvious. Live journaling reads m_active_pattern_long/short from
the PREVIOUS Direction() call. Replaying hundreds of past bars between two
live bars leaves those slots holding whichever bar the sweep stopped on, so
the next live bar journals that pattern under the current timestamp - a
corrupted row in the very table pattern win rates are computed from, which is
now also where vote weights come from. Save/RestoreVoteState() brackets every
replayed call. CSignalMETA gets away without it only because its sweep runs
once, at the first era, before any of that state matters.
TWO SOURCES OF TRUTH, KEPT APART. A reconstruction cannot know the broker
rejected an order - it has no stops level, ATR warm-up or swing-history sync
as they were at that moment - so it is an upper bound: honest about the vote,
optimistic about placement. It therefore stops dead at the handover bar,
which is latched ONCE so later rebuilds cannot creep it forward and start
overwriting real decisions with guesses, and its arrows say "reconstructed
(vote only - order validation not replayed)" in the tooltip. Someone
comparing two arrows either side of that line has to be able to tell which is
a record and which is a replay, and the chart is the only place they look.
Re-armed on any era boundary (summed era counters), because that is when the
answer changes - RankTiersFromOos has just re-derived every tier's vote weight
- and only between sweeps, so a restart cannot leave the previous pass's tail
undrawn. Chunked at 150 bars per timer slice: each bar replays Direction() on
every classic filter, which is real indicator work on the chart thread, and an
unchunked sweep here is the 2026-07-26 arrow-restore freeze waiting to happen.
SIGNAL_RESCAN_LOOKBACK_BARS moves to ExpertSignalCustom.mqh alongside
SIG_ARROW_PREFIX - same include-order reason, and the two rebuilds should
reach the same distance or the raw and filtered views are not comparable.
Known gap: on a classic-only chart the reconstruction is built once and not
refreshed when the hourly DB ranking moves the classic weights.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:05:13 -04:00
//--- Era counter, so the EA can notice an era boundary and rebuild the historical overlay - the
//--- weights (and every tier weight RankTiersFromOos just re-derived) changed, so the
//--- reconstruction is stale the instant an era ends.
long EraCount ( void ) const { return m_eraCount ; }
//--- This model's cached decision for bar `idx`, already converted to the signed vote it would
//--- have cast. No inference runs here - the era scan already wrote every scored bar into
//--- m_arrowSignalCache, which is why the reconstruction can cover the whole chart cheaply.
//--- -2.0 is the "never scored" sentinel and returns false: a bar outside the scan is NOT an
//--- abstention, and counting it as one would let a model that never saw a bar dilute the vote
//--- there. (An actual Neutral returns true with a 0.0 vote, which the caller drops from both
//--- sums exactly as the live path does.)
feat(chart): show the PROSPECTIVE vote while the models are still training
The readout sat at "VOTE 0.0%, 0 voters" constantly. Correct, and useless.
LongCondition()/ShortCondition() return 0 behind the readiness gate for the
entire training run - a model that is not deployed does not vote - so the LIVE
vote is structurally zero for hours, which is exactly the period the readout
is being watched. Worse, it was the same display whether the models were
silent, undeployed, or the filter list was empty: three different situations,
one number.
When no filter casts a real vote, the readout now shows the PROSPECTIVE one -
what these models are saying right now, through the identical tier/weight
arithmetic, minus the readiness gate. That is the same quantity the historical
overlay reconstructs on cached bars, deliberately, so the live line and the
reconstructed arrows are the same measure and can be read against each other.
It can never be mistaken for a decision: labelled "-> training, not tradable
yet", drawn dimmer than "no trade", and `fires` is forced false regardless of
magnitude, because saying "-> TRADE" about a number that cannot place an order
is the precise overstatement this readout exists to prevent. m_direction is
untouched - display only, no trading path reads it.
Confirms the sweep fix from 155f56e is live and working:
"Filtered view: swept 4999 bar(s), 767 had a voter, drew 0 arrow(s).
Strongest vote 36.0% against a 40.0% threshold."
4,999 bars against the previous 0. The remaining emptiness is the models, not
the plumbing - see the reply for why lowering the threshold further is the
wrong response to it.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:29:27 -04:00
//--- What this model would vote on the CURRENT bar if it were deployed. Same arithmetic as the
//--- live path (LiveVoteContribution) and the same arithmetic the historical overlay uses on
//--- cached bars - deliberately, so the on-chart readout and the reconstructed arrows are the
//--- same quantity and can be read against each other. The readiness gate in LongCondition() is
//--- what this bypasses, and ONLY for display: dPrevSignal is the decision, deployed or not.
//--- The -2 "not yet studied" sentinel falls out safely - DoubleToSignal() returns Undefine for
//--- it, so LiveVoteContribution() yields 0.0 rather than a spurious Sell.
virtual bool ProspectiveVote ( double & signedVote , double & weight ) override
{
signedVote = 0.0 ;
weight = 0.0 ;
fix(chart): the prospective vote was four models' opinion of ONE frozen bar
"Still glued to buy." Verified in the pass structure rather than guessed:
dPrevSignal is written ONLY by pass 1 (Training.mqh 1439/1442 - the sole
assignment sites), and pass 1 SKIPS the feedForward for any bar a later pass
will forward anyway (laterPassForwards) - which is the whole OOS window and
the calibration band. Pass 3 forwards the newest bars every era but never
writes dPrevSignal. Net effect: after every era, dPrevSignal holds the
model's opinion of the newest PURGE-BAND EDGE BAR pass 1 happened to forward
- one fixed mid-history bar, re-evaluated era after era. The readout was
therefore showing four models' verdict on the same frozen bar, and that bar
reads Buy. Glued to Buy, with flashes of Sell only while pass 1 was actively
walking (the one window where dPrevSignal moves).
ProspectiveVote() now reads the newest ARROW-CACHE entry first (walking back
from the decision bar, bounded at 16), falling back to dPrevSignal only when
the cache holds nothing. Pass 3 writes the adjusted decision for the newest
(OOS) bars each era, so the cache's first non-sentinel entry is the model's
most recent verdict on near-current data - and it is the same value the
filtered overlay draws from, so the label and the reconstruction stay one
quantity. A cached Neutral stops the walk: that is a real decision (vote 0,
abstain -> shows in the "flat" count), not a missing one. Early in an era
the cache is wiped to sentinel and everything falls through to dPrevSignal
exactly as before, until pass 3 refills the newest rows.
Expect the label to change per era now (as each pass 3 re-scores the newest
bars under that era's weights), with the vote/flat split moving as members
genuinely flip between direction and Neutral on recent data.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 21:47:51 -04:00
//--- NEWEST CACHED DECISION FIRST, dPrevSignal only as a fallback - and the order is the fix
//--- for a readout that sat glued to one direction for hours (reported 2026-08-18).
//---
//--- dPrevSignal is written ONLY by pass 1, and pass 1 skips the feedForward for any bar a
//--- later pass will forward anyway (see laterPassForwards) - which is the whole OOS window
//--- and the calibration band. Pass 3 forwards the newest bars every era but never writes
//--- dPrevSignal. Net effect: after every era, dPrevSignal holds the model's opinion of the
//--- newest PURGE-BAND EDGE BAR pass 1 happened to forward - one fixed mid-history bar,
//--- re-evaluated era after era. Four models staring at the same frozen bar that reads Buy
//--- is a label glued to Buy, regardless of what any of them think about the market now.
//---
//--- The arrow cache is the honest source: pass 3 writes the ADJUSTED decision for the newest
//--- (OOS) bars each era, so the first non-sentinel entry walking back from the decision bar
//--- is this model's most recent verdict on near-current data. A cached NEUTRAL stops the walk -
//--- that is a real decision (vote 0, abstain), not a missing one. The walk is bounded: past
//--- ~16 bars the answer is no fresher than dPrevSignal and the fallback is fine. Early in an
//--- era the cache is wiped to the -2 sentinel and everything falls through to dPrevSignal,
//--- exactly as before, until pass 3 refills the newest rows.
for ( int idx = 1 ; idx < = 16 ; idx + + )
{
if ( CachedVoteAt ( idx , signedVote ) )
{
weight = ModuleWeight ( ) ;
return true ;
}
}
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era
Full-pipeline analysis after "threshold 30, attained often, nothing drawn,
still glued to buy". The log falsified the premise before any code did:
21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0%
21:42:07 swept 4999, 0 voters, drew 0
21:51:30 swept 4999, 0 voters, drew 0
21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0%
The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root
cause, three symptoms: every display path read m_arrowSignalCache, which is
wiped to sentinel at each era start and only complete again when pass 3
finishes. With eras at ~30s and a sweep at ~17s:
* ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its
else-branch deleted the arrow on every voteless bar - erasing the previous
sweep's entire output. The chart cycled populated -> blank -> populated;
the user kept catching the blank phase.
* READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every
era and fell through to dPrevSignal - the frozen purge-band edge bar that
reads Buy. 659638e fixed which bar was frozen, not the freezing.
* VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a
different fraction of half-rebuilt caches.
THE FIX, structural rather than another patch:
1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one
moment the cache is complete - and now copies it (raw signals, newest
LOOKBACK+16 bars) into member-owned snapshot state, unconditionally,
BEFORE its early return: an all-Neutral era is a snapshot worth showing,
not an absence of one. Raw signals rather than votes, so a tier re-rank
between eras reprices them at read time via LiveVoteContribution for free.
2. The sweep (SnapshotVoteAt) and the prospective readout both read
snapshots; the readout's fallback chain is live-cache -> snapshot ->
dPrevSignal, and the snapshot leg is the one that fires most of the time.
3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual
sub-threshold vote takes an arrow down. This alone ends the wipe half of
the flicker even where snapshots are missing (before the first era).
4. Arming moved from an era-counter diff (which fires at era BOUNDARIES,
i.e. precisely when caches are about to be wiped) to
g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's
snapshot just got fresher", the only event a redraw can act on. 60s rate
limit collapses the four members' burst into one sweep. Classic-only
charts arm once at start.
5. Census now reports the direction split - "922 had a voter (610 buy / 312
sell)" - so "the vote leans buy" is checkable from the log instead of
inferred from arrow colours.
Also visible in the log and worth knowing: the threshold flip-flopped
30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51
ran at 40), so part of the observed blankness was configuration, not code.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
//--- Era-end snapshot next - the fallback that actually fires for ~90% of every era, because
//--- the live cache above is wiped at era start and only refills when pass 3 completes. Before
//--- this existed the chain fell straight through to dPrevSignal (the frozen purge-band edge
//--- bar), which is what kept the readout pinned to one direction for hours.
if ( m_prospectiveSigSnap ! = -2.0 & & MathIsValidNumber ( m_prospectiveSigSnap ) )
{
signedVote = LiveVoteContribution ( m_prospectiveSigSnap ) ;
weight = ModuleWeight ( ) ;
return true ;
}
fix(chart): the prospective vote was four models' opinion of ONE frozen bar
"Still glued to buy." Verified in the pass structure rather than guessed:
dPrevSignal is written ONLY by pass 1 (Training.mqh 1439/1442 - the sole
assignment sites), and pass 1 SKIPS the feedForward for any bar a later pass
will forward anyway (laterPassForwards) - which is the whole OOS window and
the calibration band. Pass 3 forwards the newest bars every era but never
writes dPrevSignal. Net effect: after every era, dPrevSignal holds the
model's opinion of the newest PURGE-BAND EDGE BAR pass 1 happened to forward
- one fixed mid-history bar, re-evaluated era after era. The readout was
therefore showing four models' verdict on the same frozen bar, and that bar
reads Buy. Glued to Buy, with flashes of Sell only while pass 1 was actively
walking (the one window where dPrevSignal moves).
ProspectiveVote() now reads the newest ARROW-CACHE entry first (walking back
from the decision bar, bounded at 16), falling back to dPrevSignal only when
the cache holds nothing. Pass 3 writes the adjusted decision for the newest
(OOS) bars each era, so the cache's first non-sentinel entry is the model's
most recent verdict on near-current data - and it is the same value the
filtered overlay draws from, so the label and the reconstruction stay one
quantity. A cached Neutral stops the walk: that is a real decision (vote 0,
abstain -> shows in the "flat" count), not a missing one. Early in an era
the cache is wiped to sentinel and everything falls through to dPrevSignal
exactly as before, until pass 3 refills the newest rows.
Expect the label to change per era now (as each pass 3 re-scores the newest
bars under that era's weights), with the vote/flat split moving as members
genuinely flip between direction and Neutral on recent data.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 21:47:51 -04:00
signedVote = 0.0 ;
feat(chart): show the PROSPECTIVE vote while the models are still training
The readout sat at "VOTE 0.0%, 0 voters" constantly. Correct, and useless.
LongCondition()/ShortCondition() return 0 behind the readiness gate for the
entire training run - a model that is not deployed does not vote - so the LIVE
vote is structurally zero for hours, which is exactly the period the readout
is being watched. Worse, it was the same display whether the models were
silent, undeployed, or the filter list was empty: three different situations,
one number.
When no filter casts a real vote, the readout now shows the PROSPECTIVE one -
what these models are saying right now, through the identical tier/weight
arithmetic, minus the readiness gate. That is the same quantity the historical
overlay reconstructs on cached bars, deliberately, so the live line and the
reconstructed arrows are the same measure and can be read against each other.
It can never be mistaken for a decision: labelled "-> training, not tradable
yet", drawn dimmer than "no trade", and `fires` is forced false regardless of
magnitude, because saying "-> TRADE" about a number that cannot place an order
is the precise overstatement this readout exists to prevent. m_direction is
untouched - display only, no trading path reads it.
Confirms the sweep fix from 155f56e is live and working:
"Filtered view: swept 4999 bar(s), 767 had a voter, drew 0 arrow(s).
Strongest vote 36.0% against a 40.0% threshold."
4,999 bars against the previous 0. The remaining emptiness is the models, not
the plumbing - see the reply for why lowering the threshold further is the
wrong response to it.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:29:27 -04:00
if ( ! MathIsValidNumber ( dPrevSignal ) )
return false ;
signedVote = LiveVoteContribution ( dPrevSignal ) ;
weight = ModuleWeight ( ) ;
return true ;
}
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era
Full-pipeline analysis after "threshold 30, attained often, nothing drawn,
still glued to buy". The log falsified the premise before any code did:
21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0%
21:42:07 swept 4999, 0 voters, drew 0
21:51:30 swept 4999, 0 voters, drew 0
21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0%
The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root
cause, three symptoms: every display path read m_arrowSignalCache, which is
wiped to sentinel at each era start and only complete again when pass 3
finishes. With eras at ~30s and a sweep at ~17s:
* ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its
else-branch deleted the arrow on every voteless bar - erasing the previous
sweep's entire output. The chart cycled populated -> blank -> populated;
the user kept catching the blank phase.
* READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every
era and fell through to dPrevSignal - the frozen purge-band edge bar that
reads Buy. 659638e fixed which bar was frozen, not the freezing.
* VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a
different fraction of half-rebuilt caches.
THE FIX, structural rather than another patch:
1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one
moment the cache is complete - and now copies it (raw signals, newest
LOOKBACK+16 bars) into member-owned snapshot state, unconditionally,
BEFORE its early return: an all-Neutral era is a snapshot worth showing,
not an absence of one. Raw signals rather than votes, so a tier re-rank
between eras reprices them at read time via LiveVoteContribution for free.
2. The sweep (SnapshotVoteAt) and the prospective readout both read
snapshots; the readout's fallback chain is live-cache -> snapshot ->
dPrevSignal, and the snapshot leg is the one that fires most of the time.
3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual
sub-threshold vote takes an arrow down. This alone ends the wipe half of
the flicker even where snapshots are missing (before the first era).
4. Arming moved from an era-counter diff (which fires at era BOUNDARIES,
i.e. precisely when caches are about to be wiped) to
g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's
snapshot just got fresher", the only event a redraw can act on. 60s rate
limit collapses the four members' burst into one sweep. Classic-only
charts arm once at start.
5. Census now reports the direction split - "922 had a voter (610 buy / 312
sell)" - so "the vote leans buy" is checkable from the log instead of
inferred from arrow colours.
Also visible in the log and worth knowing: the threshold flip-flopped
30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51
ran at 40), so part of the observed blankness was configuration, not code.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
//--- The sweep's data source - see CExpertSignalCustom::SnapshotVoteAt for why this is a
//--- snapshot and not the live cache.
virtual bool SnapshotVoteAt ( const int idx , double & signedVote ) override
{
signedVote = 0.0 ;
if ( idx < 0 | | idx > = m_overlaySnapBars )
return false ;
double sig = m_overlaySigSnap [ idx ] ;
if ( sig = = -2.0 | | ! MathIsValidNumber ( sig ) )
return false ;
signedVote = LiveVoteContribution ( sig ) ;
return true ;
}
feat(chart): reconstruct the filtered view behind the handover point
Completes the filtered view from 282b535, which only reached forward of
attach. On a multi-hour training run that is the entire time you are looking
at the chart, so the answer to "how would the whole bot have traded" was
blank exactly when it was wanted.
The sweep lives on the AGGREGATE signal, which is the only object holding
every filter. AI members contribute their CACHED per-bar decision from the
era scan - no inference re-runs, the cache already spans the chart - and the
classic ladders are replayed with EvalShift(i), the same mechanism
CSignalMETA's candidate sweep uses and exact because every classic pattern
condition anchors on StartIndex(). Combination is the live one: weighted mean
over voting filters, abstentions out of both sums, against Min_Vote_Open.
THE REPLAY CORRUPTS LIVE JOURNALING IF LEFT UNGUARDED, and this is the part
that is not obvious. Live journaling reads m_active_pattern_long/short from
the PREVIOUS Direction() call. Replaying hundreds of past bars between two
live bars leaves those slots holding whichever bar the sweep stopped on, so
the next live bar journals that pattern under the current timestamp - a
corrupted row in the very table pattern win rates are computed from, which is
now also where vote weights come from. Save/RestoreVoteState() brackets every
replayed call. CSignalMETA gets away without it only because its sweep runs
once, at the first era, before any of that state matters.
TWO SOURCES OF TRUTH, KEPT APART. A reconstruction cannot know the broker
rejected an order - it has no stops level, ATR warm-up or swing-history sync
as they were at that moment - so it is an upper bound: honest about the vote,
optimistic about placement. It therefore stops dead at the handover bar,
which is latched ONCE so later rebuilds cannot creep it forward and start
overwriting real decisions with guesses, and its arrows say "reconstructed
(vote only - order validation not replayed)" in the tooltip. Someone
comparing two arrows either side of that line has to be able to tell which is
a record and which is a replay, and the chart is the only place they look.
Re-armed on any era boundary (summed era counters), because that is when the
answer changes - RankTiersFromOos has just re-derived every tier's vote weight
- and only between sweeps, so a restart cannot leave the previous pass's tail
undrawn. Chunked at 150 bars per timer slice: each bar replays Direction() on
every classic filter, which is real indicator work on the chart thread, and an
unchunked sweep here is the 2026-07-26 arrow-restore freeze waiting to happen.
SIGNAL_RESCAN_LOOKBACK_BARS moves to ExpertSignalCustom.mqh alongside
SIG_ARROW_PREFIX - same include-order reason, and the two rebuilds should
reach the same distance or the raw and filtered views are not comparable.
Known gap: on a classic-only chart the reconstruction is built once and not
refreshed when the hourly DB ranking moves the classic weights.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:05:13 -04:00
virtual bool CachedVoteAt ( const int idx , double & signedVote ) override
{
signedVote = 0.0 ;
if ( idx < 0 | | idx > = ArraySize ( m_arrowSignalCache ) )
return false ;
double sig = m_arrowSignalCache [ idx ] ;
if ( sig = = -2.0 | | ! MathIsValidNumber ( sig ) )
return false ;
signedVote = LiveVoteContribution ( sig ) ;
return true ;
}
2026-07-23 08:21:41 -04:00
//--- buckets the live confidence magnitude into one of the 4 tiers above - see m_pattern_0's
//--- declaration comment. Public so PollTraining()/status-display code could surface which tier is
//--- currently active if ever useful, though LongCondition/ShortCondition are the only callers today.
int ConfidenceTier ( void ) ;
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- The same bucketing asked of an ARBITRARY decision value rather than of dPrevSignal. Split out
//--- so the OOS scan can ask "what tier would this scanned bar have voted at" - it holds the bar's
//--- decision in a local, and dPrevSignal is the LIVE bar's, which is a different bar entirely.
int ConfidenceTierFor ( const double signal ) ;
2026-07-23 08:21:41 -04:00
int PatternWeightForTier ( int tier ) ;
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- THE VOTE THIS MEMBER WOULD CAST, in the units CExpertSignalCustom::Direction() actually sums:
//--- m_weight (0..1, DB-ranked) x the tier's pattern weight (0..100, DB-ranked), signed + for Buy
//--- and - for Sell, and exactly 0.0 when the decision is Neutral (an abstention, which live drops
//--- from BOTH the sum and the divisor).
//---
//--- This exists because the ensemble deploy gate was scoring a different quantity than the one that
//--- trades. It contributed the raw signed confidence x100 (a 33..100 number straight off the head)
//--- while live contributes a win-rate-derived tier weight; the two have no fixed relation, and
//--- g_ensembleVoteThreshold's own comment claims they fire on the same criterion. Same shape as the
//--- 2026-08-09 geometry incident: certified on one game, paid on another. One function, called from
//--- both the live vote path and the gate, is what keeps them from drifting apart again.
double LiveVoteContribution ( const double signal ) ;
2026-07-14 22:36:27 -04:00
//--- methods of setting adjustable parameters
refactor(ai): derive the first dense layer's width instead of asking for it
InitialNeurons was an input whose only defensible value depends on two
things the user cannot see when picking from a dropdown: how wide the input
vector ended up after feature selection, and how much in-sample data the
study period actually yields. Left to a hand-picked constant it was badly
wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a
292,583-weight model, against ~36,500 training bars of which only ~2,236
are directional. That is 6.6 weights per training bar, and it EXPANDS a set
of highly correlated inputs rather than compressing them.
The symptom was already in the logs and had been read as a depth problem:
the shallowest topology consistently beat the deepest (perceptron 52.7%
balanced, hybrid 41.3%). Over-parameterization predicts that ordering just
as well as covariate shift does, and only one of the two had been addressed.
ComputeFirstLayerWidth() budgets roughly one first-layer weight per
in-sample bar. Measured across the configurations in use:
M15 10y -> 256 units, 129,071 weights, 0.73 per bar
H1 10y -> 64 units, 28,727 weights, 0.65 per bar
H4 10y -> 16 units, 7,559 weights, 0.68 per bar
Two design points that matter:
- It estimates in-sample bars from the STUDY PERIOD and timeframe, not
from Bars(). What is downloaded grows over a terminal's lifetime, and a
topology that widened as history filled in would re-key its own weights
file and discard a trained model.
- The result is snapped down to a coarse power-of-two ladder, so the
estimate would have to be wrong by ~2x to change the answer.
Every field it reads is already part of the weights-filename fingerprint,
so the derived value needs no fingerprint entry of its own. The public
setter is removed - it could only have been called after construction, and
would either be ignored or silently re-key the model mid-run.
Where the data cannot support even the floor (D1 over 10 years is under
2,000 bars) it now says so and names the fixes, rather than quietly
training a model with more weights than examples.
The DB config fingerprint drops the term too, which re-keys existing
pattern databases once - correct, since a model an order of magnitude
smaller should not inherit the old one's win-rate history.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
//--- No public setter for m_initialNeuronsCount. It feeds the weights-filename fingerprint, and it is
//--- now derived exactly once, inside InitNeuralNetwork(), before that fingerprint is built - see
//--- ComputeFirstLayerWidth(). An external setter could only ever be called after construction and
//--- would either be ignored (if before init) or silently re-key the model mid-run (if after).
2026-07-14 22:36:27 -04:00
void OutputNeuronsCount ( int value ) { m_outputNeuronsCount = value ; }
refactor(ai): derive the dense taper's shape, not just its first layer
Deriving the first layer's width left NeuronsReduction and MinNeuronsCount
behind as inputs calibrated for something that no longer exists. Against a
hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine
funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to
64 -> 20 -> 20: the reduction factor stops mattering after one step, and
"minimum neurons per layer" silently becomes the width of every layer but
the first. Two knobs whose labels no longer describe what they do.
The taper now runs geometrically from the derived first-layer width down to
a final hidden layer sized off the output count, spread evenly over however
many layers the chosen AIType implies:
MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights
MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450
CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763
and it stays a funnel at the floor, where the old rule could not:
D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3
Both inputs are removed. With the width derived there is no freedom left in
the taper, so keeping either would only let the user contradict the
derivation. The layer COUNT stays selectable, because it is bundled into
AIType alongside the conv/LSTM front-end - depth is an architecture choice,
not a data-derived quantity, and pairing them means the two cannot
contradict each other.
m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing
reads them to build a topology any more, but they hold positional slots in
the .cfg sidecar and the weights fingerprint, and changing either value
would re-key every model on disk for no behavioural reason.
The DB config fingerprint drops both terms.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 14:03:42 -04:00
//--- No setter: the taper's endpoints are derived, not configured. See BuildFreshTopology()'s taper
//--- block and the note in Variables\Inputs.mqh. Kept as members only because the .cfg topology
//--- sidecar's field layout is positional and rewriting it would invalidate every model on disk.
2026-07-14 22:36:27 -04:00
void HiddenLayersCount ( int value ) { m_hiddenLayersCount = value ; }
2026-07-22 22:51:04 -04:00
void LstmHiddenSize ( int value ) { m_lstmHiddenSize = value ; }
void ConvFilterCount ( int value ) { m_convFilterCount = value ; }
2026-07-25 15:55:56 -04:00
//--- StopTrainWR(int) removed with the MinWR input - there is no absolute accuracy target any more;
//--- see LEGACY_CONVERGE_WR_SLOT and the plateau ladder.
2026-07-14 22:36:27 -04:00
void MinDirectionalRecall ( int value ) { m_minDirectionalRecallPct = value ; }
2026-07-26 18:33:12 -04:00
//--- MinSignalConfidence(double) removed with the AI entry floor - confidence now reaches the trade
//--- decision as vote weight (ConfidenceTier), gated by the one Min vote to open threshold that the
//--- classic votes already answer to. See m_pattern_0's declaration comment.
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- The SINGLE class-imbalance control - tau in Menon et al.'s logit adjustment. 0 disables the
//--- correction entirely. OversampleParity/EnableMinorityReplay/ConstrainReplay/LogitPriorStrength/
//--- UseLogitAdjustedLoss/FocalLossGamma/UseStaticPrior were removed 2026-07-31; see the
//--- class-imbalance block in Variables\Inputs.mqh for the audit that found five of them inert.
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior
Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add
tau*log(prior_c) to each class logit inside the training gradient. Softmax
CE on adjusted logits is consistent for BALANCED error - the metric
checkpoint selection already ranks on - so the loss and the deploy decision
finally optimize the same thing.
The engine already computed a true softmax + categorical-CE gradient and
wrote it over the per-neuron sigmoid delta, so this is an offset added to
three logits in the two places that gradient is built (backProp scalar path
and backPropOCL). No backend, kernel or DLL change; the forward pass and
every inference path are untouched, which is the point - the network learns
to absorb the offset, so its raw argmax becomes the balanced-optimal
decision with nothing applied at inference.
Replaces rather than stacks. Minority replay is disabled while this is on,
and the post-hoc inference prior is forced off. Stacking is not a
theoretical worry: simulated on the measured 1118/1119/34298 distribution
in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced,
Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance -
and BOTH together score 45.4% with Neutral recall at 0%, worse than either
alone. Buda et al. 2018 predicts exactly that.
Motivation from the six-chart run: every topology took one direction to
~50% recall and abandoned the other, the direction chosen arbitrarily (the
batch-norm control went Buy 1% / Sell 42%, the inverse of the other five).
One era in 1,301 cleared the per-class recall floor.
Fingerprinted conditionally, so the converged 60.7% models on disk keep
their filenames and stay loadable as the fallback.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
void LogitAdjustTau ( double value ) { m_logitAdjustTau = MathMax ( 0.0 , value ) ; }
2026-07-28 17:42:12 -04:00
void FreezePriorCalibration ( bool value ) { m_freezePriorCalibration = value ; }
2026-07-21 00:03:45 -04:00
void SignalClusterWindow ( int value ) { m_signalClusterWindow = value ; }
2026-07-14 22:36:27 -04:00
void SwingConfirmationBars ( int value ) { m_swingConfirmationBars = value ; }
2026-08-15 04:44:10 -04:00
//--- Called from ConfigureAISignal when the TrainingTarget input selects the fractal label. Guarded
//--- so CSignalMETA (whose constructor already claimed target 1) can never be flipped: the meta
//--- head's 2-output topology and candidate pipeline are incompatible with a per-bar 3-class label.
void TrainTargetFractal ( void ) { if ( m_trainTarget = = 0 ) m_trainTarget = 2 ; }
2026-08-15 16:50:36 -04:00
//--- ENSEMBLE MEMBERSHIP (AIType == AI_HYBRID: all four direction NNs on one chart). Feeds the
//--- |ENS1 fingerprint token, so an ensemble member's weight files can never collide with a solo
//--- model of identical settings on another chart of the same symbol - without it, the
//--- duplicate-chart guard would (correctly) fight the two charts over one .nnw.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- voteThreshold mirrors the Min_Vote_Open input into g_ensembleVoteThreshold (pass -1 to leave
//--- it untouched) - see that global's comment for why it is pushed in rather than read.
void EnsembleMember ( bool value , double voteThreshold = -1.0 )
{
m_ensembleMember = value ;
if ( voteThreshold > = 0.0 )
g_ensembleVoteThreshold = voteThreshold ;
if ( value & & m_ensembleIndex < 0 )
{
int n = ArraySize ( g_warriorEnsemble ) ;
ArrayResize ( g_warriorEnsemble , n + 1 ) ;
g_warriorEnsemble [ n ] = GetPointer ( this ) ;
m_ensembleIndex = n ;
}
}
//--- Minimum era among the ensemble members still genuinely training. Deployed/complete, stopped and
//--- paused members are exempt (a paused model must not deadlock the rest; when resumed it is behind,
//--- becomes the min itself, and the others hold until it catches up - the barrier re-syncs on its
//--- own). Falls back to this member's own era when nothing qualifies, which makes the barrier a
//--- no-op rather than a lock.
long EnsembleMinTrainingEra ( void )
{
long minEra = LONG_MAX ;
for ( int i = 0 ; i < ArraySize ( g_warriorEnsemble ) ; i + + )
{
CExpertSignalAIBase * mm = g_warriorEnsemble [ i ] ;
if ( CheckPointer ( mm ) = = POINTER_INVALID )
continue ;
if ( mm . m_trainingComplete | | mm . m_trainingStopRequested | | mm . m_trainingPaused | | ! mm . m_isInitialized )
continue ;
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- THE LIVENESS EXEMPTION (see ENSEMBLE_BARRIER_STUCK_MS). The three flags above are all
//--- VOLUNTARY - a member that chose to stop participating. A member whose era simply will not
//--- advance sets none of them, and the barrier had no way to tell "slow" from "never", so one
//--- broken member froze its whole chart with no time limit and no journal line. Excluded
//--- members keep training (they are still trying to recover) and rejoin the moment they
//--- complete an era - at which point, being behind, they legitimately become the minimum again
//--- and the others wait for them to catch up, exactly as a resumed laggard does.
if ( mm . m_barrierExcluded )
continue ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
if ( mm . m_eraCount < minEra )
minEra = mm . m_eraCount ;
}
return ( minEra = = LONG_MAX ) ? m_eraCount : minEra ;
}
//--- How many members are actively consuming training chunks right now: still training AND at the
//--- barrier's minimum era (a member held ABOVE the min declines its calls, so it costs nothing).
//--- TRAIN_TIME_BUDGET_MS divides the ensemble budget by this, which is the "faster member donates
//--- its power to the laggards" behaviour: 4 trainers -> 30ms each, one laggard left -> it gets the
//--- full 120ms, and the chart thread's UI headroom stays constant either way.
int EnsembleActiveTrainers ( void )
{
if ( ! m_ensembleMember )
return 1 ;
long minEra = EnsembleMinTrainingEra ( ) ;
int active = 0 ;
for ( int i = 0 ; i < ArraySize ( g_warriorEnsemble ) ; i + + )
{
CExpertSignalAIBase * mm = g_warriorEnsemble [ i ] ;
if ( CheckPointer ( mm ) = = POINTER_INVALID )
continue ;
if ( mm . m_trainingComplete | | mm . m_trainingStopRequested | | mm . m_trainingPaused | | ! mm . m_isInitialized )
continue ;
if ( mm . m_eraCount < = minEra )
active + + ;
}
return MathMax ( active , 1 ) ;
}
//--- True when this member has finished more eras than the slowest still-training member and must
//--- wait at the era barrier - checked at Train()'s entry (see the barrier note there).
bool EnsembleEraBarrierHolds ( void )
{
if ( ! m_ensembleMember | | m_trainingComplete )
return false ;
return m_eraCount > EnsembleMinTrainingEra ( ) ;
}
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so
every depth instrument from 1dda479/7e63a8b/45c9e21 was live.
- It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth"
in 27 MB of journal. The instrument built to find the depth shortfall returned
"not this".
- On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars -
same chart, same 832-value window, same indicators, byte-identical fingerprint -
while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the
symbol, the history, the bar count or the indicator depth. It is per-member.
- ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that.
The depth reading in project_silent_block_failures is therefore retired by its own
instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one.
1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING.
ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one
branch over three unrelated states, silent in all of them:
enabled == 0 -> nothing tunable is on. No cap. Healthy.
enabled > 0, servable == -1 -> a handle answered INVALID.
enabled > 0, servable == 0 -> created, never calculated.
BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable
from "no tunable indicators enabled" - and both returned `want` without printing a
character. That is exactly the state a per-member, every-index, depth-independent
failure produces, and it is the single reason a build carrying full depth
instrumentation logged nothing through the whole outage.
TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the
dead-handle case is reported (latched, with per-handle depths). The RETURN is
deliberately unchanged - what to do about a dead handle is not yet known, and
changing control flow on an unproven cause is how the last four fixes here went
wrong. SettledBars() routes its three pass-through states via ServableBars() so the
report is reachable from the training sweep, which is the only caller that hits it.
2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK.
"lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an
indicator warm-up or a history-edge read"). Which guard fired was INFERRED by
counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic
was right; every conclusion drawn from it was wrong, because a value count names a
POSITION and a position cannot tell cold from capped from invalid from off-the-end.
Every guard that can reject a bar now records itself - m_featureFailBlock - and the
report carries it, the series index, IndicatorDepthReport()'s per-handle depths,
and for each indicator whether the NEWEST bar reads. That last field is the whole
diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere
(cold or dead handle), newest-reads means a genuine history edge. Instrumented:
open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold().
3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION.
It armed only when m_featureFailTransient was set. Keeping that flag correct across
every guard is a list that has to stay right forever - the same shape of fix the
feature cache abandoned for the same reason - and the gate is pointless anyway: a
sweep where ZERO of 50,163 bars produced a window will produce zero again if it
restarts a millisecond later, transient or not. Doing that at full speed is what
starved six indicator threads on a six-core box. The backoff is now unconditional
on a total failure. The flag keeps its real job, deciding whether a MISS may be
cached, which is a per-bar question and not a scheduling one.
4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE.
EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its
comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member
that simply CANNOT finish an era is none of them, so it pinned the minimum at its
own era with no time limit - and the hold branch's only action was
`m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on
USDJPY the two members that could not train reported, and the two healthy members
frozen behind them wrote nothing anywhere. The outage was visible only through the
members that were not suffering it.
- BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from
m_lastEraCompleteTick precisely because the barrier resets that one. Only a
member AT the minimum can be a blocker; a member ahead is idle by design and is
never counted as stuck.
- After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from
the barrier minimum. It keeps training and rejoins the instant it completes an
era - at which point, being behind, it legitimately becomes the minimum again,
which is the documented resumed-laggard behaviour.
- Both transitions say so loudly, and the release states plainly that the
combined-vote score cannot be computed while the ensemble is desynchronised.
- A held member now writes a rate-limited journal line naming WHICH members it is
waiting on, so the blocker is read off one line.
5. THE PANEL FLICKER.
OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member
returns from Train() before ever setting it - so both writers thought they were the
only one updating the label and fought every tick. That is the reported "Getting
ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit
Perceptron but not Convolutional purely because Convolutional had a run active from
a completed era and Perceptron, resumed from disk, never did. Train()'s message is
the specific one, so it wins.
NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and
the per-handle depths. Read it. Do not reason around it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- Era-advance watchdog for the barrier, kept SEPARATE from m_lastEraCompleteTick on purpose. That
//--- one is reset by the barrier-hold branch itself (deliberately - a held member is idle, not
//--- stalled), so anything derived from it would read every held member as healthy and every holder
//--- as healthy too. This clock is stamped only by a real change of m_eraCount, so it measures the
//--- one thing the barrier needs: is this member still making progress?
//--- Called at Train() entry, before the barrier test, for `this` member only - all members of a
//--- chart share the one MQL5 thread, so each stamps itself and reads the others.
void BarrierEraHeartbeat ( void )
{
if ( ! m_ensembleMember )
return ;
uint nowTick = GetTickCount ( ) ;
if ( m_barrierEraTick = = 0 | | m_barrierEraSeen ! = m_eraCount )
{
//--- Progress (or the first observation). Rejoining is unconditional and immediate: a member
//--- that just completed an era is by definition not stuck, whatever it was doing before.
if ( m_barrierExcluded )
Print ( ID + " : REJOINING THE ERA BARRIER at era " + IntegerToString ( ( int ) m_eraCount ) +
" - it completed an era, so it is training again. It is behind the rest of the "
" ensemble, which means it now sets the minimum and the others wait for it to catch "
" up. The combined-vote score resumes once every member reports the same era. " ) ;
m_barrierExcluded = false ;
m_barrierEraSeen = m_eraCount ;
m_barrierEraTick = nowTick ;
return ;
}
//--- Same era as last look. Only a member that is AT the minimum can be the one blocking: a member
//--- ahead of it is not advancing because the barrier is holding it, which is correct behaviour and
//--- must never be mistaken for being stuck.
if ( m_barrierExcluded | | m_eraCount > EnsembleMinTrainingEra ( ) )
return ;
if ( nowTick - m_barrierEraTick < ENSEMBLE_BARRIER_STUCK_MS )
return ;
m_barrierExcluded = true ;
PrintFormat ( " %s: RELEASING THE ERA BARRIER - this member has not completed an era in %.0f minutes "
" (still at era %d) and every other member on this chart has been waiting on it for "
" that entire time. It is excluded from the barrier minimum so the rest can advance; "
" it keeps training and rejoins the moment it finishes an era. READ THE TRAIN STALL "
" LINE ABOVE for why it is not finishing - the barrier only reports that it is stuck, "
" never why. NOTE: while the ensemble is desynchronised the combined-vote OOS score "
" cannot be computed (it scores only bars EVERY member contributed at the same era), "
" so no ensemble verdict will be published until this member catches up. " ,
ID , ( nowTick - m_barrierEraTick ) / 60000.0 , ( int ) m_eraCount ) ;
}
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- Pass-3 hook: record the VOTE this member would have cast on this OOS bar into the combined-vote
//--- buffer. signedVote is in live vote units - m_weight x tier pattern weight, signed by direction
//--- (see LiveVoteContribution()) - NOT the raw confidence this used to carry. It is 0.0 when the
//--- member abstains, and an abstention is then excluded from the divisor as well as the sum, which
//--- is what CExpertSignalCustom::Direction() does live (see g_ensVoteVoterMask).
void EnsembleOosContribute ( const int barIdx , const double signedVote ,
feat(vote): thresholds become confidence percentages, on ONE scale everywhere
User request: "the entry/exit thresholds are manual numbers, I would like
them to be confidence percentages, so the current 20 would be only 20%
confidence in a profitable trade."
WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's
contribution are win rates: the pattern weight is that pattern's measured win
rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's
average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT
therefore produced a mean of PRODUCTS of two win rates - a genuinely
60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was
never on a probability scale, so its magnitude meant nothing on its own.
Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which
is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60;
MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight
stops being a discount on the probability and becomes how much a filter's
opinion COUNTS - which is what a module weight should always have been.
Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed.
ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three
other places compared against a 0..1 softmax confidence and would each have
become a fresh currency mismatch the moment the input changed meaning:
* the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now
reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the
classic side, which is the only reason that route exists - against the
same m_threshold_close the averaged vote uses. m_ai_exit_threshold is
retired rather than left dangling.
* m_oosDecisionSeries now carries the vote, not the confidence, so the exit
SIMULATION stops modelling a close rule the EA does not run.
* ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input
through that would have silently switched vote exits off in the
simulation while live went on running them - found before it shipped;
the bound now tracks the scale.
LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing,
SL/TP scaling and the intelligent trailing want a model confidence, not a win
rate.
CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a
real probability to the extent the pattern weights are. A pattern with fewer
than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a
designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the
signal DB fills, "60" means "the designed conviction of the patterns that
fired". Closing that gap is the next commit.
Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales
this removes.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
const double voteWeight ,
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
const bool winLong , const bool winShort , const bool dirLabel )
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
{
if ( ! m_ensembleMember | | m_ensembleIndex < 0 | | m_ensembleIndex > = 8 )
return ;
datetime t = m_Time . GetData ( barIdx ) ;
if ( t < = 0 )
return ;
if ( g_ensVoteEra ! = m_eraCount )
{
//--- first contribution of a new era resets the buffer (the era barrier keeps members aligned,
//--- so a mismatched stamp means "previous era's rows", never "a sibling's different era")
g_ensVoteEra = m_eraCount ;
g_ensVoteRows = 0 ;
g_ensVoteDoneMask = 0 ;
ArrayInitialize ( g_ensVoteCursor , 0 ) ;
}
int bit = ( 1 < < m_ensembleIndex ) ;
//--- monotonic cursor first (members scan bars oldest-to-newest, so the match is O(1) amortized),
//--- full wrap-around only when per-member window failures desynchronize the sequences
int row = -1 ;
int start = g_ensVoteCursor [ m_ensembleIndex ] ;
if ( start > g_ensVoteRows )
start = 0 ;
for ( int i = start ; i < g_ensVoteRows ; i + + )
if ( g_ensVoteTime [ i ] = = t ) { row = i ; break ; }
if ( row < 0 )
for ( int i = 0 ; i < start ; i + + )
if ( g_ensVoteTime [ i ] = = t ) { row = i ; break ; }
if ( row < 0 )
{
if ( g_ensVoteRows > = ArraySize ( g_ensVoteTime ) )
{
int cap = g_ensVoteRows + g_ensVoteRows / 2 + 512 ;
ArrayResize ( g_ensVoteTime , cap ) ;
ArrayResize ( g_ensVoteSum , cap ) ;
ArrayResize ( g_ensVoteMask , cap ) ;
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
ArrayResize ( g_ensVoteVoterMask , cap ) ;
feat(vote): thresholds become confidence percentages, on ONE scale everywhere
User request: "the entry/exit thresholds are manual numbers, I would like
them to be confidence percentages, so the current 20 would be only 20%
confidence in a profitable trade."
WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's
contribution are win rates: the pattern weight is that pattern's measured win
rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's
average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT
therefore produced a mean of PRODUCTS of two win rates - a genuinely
60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was
never on a probability scale, so its magnitude meant nothing on its own.
Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which
is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60;
MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight
stops being a discount on the probability and becomes how much a filter's
opinion COUNTS - which is what a module weight should always have been.
Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed.
ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three
other places compared against a 0..1 softmax confidence and would each have
become a fresh currency mismatch the moment the input changed meaning:
* the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now
reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the
classic side, which is the only reason that route exists - against the
same m_threshold_close the averaged vote uses. m_ai_exit_threshold is
retired rather than left dangling.
* m_oosDecisionSeries now carries the vote, not the confidence, so the exit
SIMULATION stops modelling a close rule the EA does not run.
* ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input
through that would have silently switched vote exits off in the
simulation while live went on running them - found before it shipped;
the bound now tracks the scale.
LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing,
SL/TP scaling and the intelligent trailing want a model confidence, not a win
rate.
CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a
real probability to the extent the pattern weights are. A pattern with fewer
than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a
designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the
signal DB fills, "60" means "the designed conviction of the patterns that
fired". Closing that gap is the next commit.
Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales
this removes.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
ArrayResize ( g_ensVoteWeightSum , cap ) ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArrayResize ( g_ensVoteWinLong , cap ) ;
ArrayResize ( g_ensVoteWinShort , cap ) ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
ArrayResize ( g_ensVoteDirLabel , cap ) ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
}
row = g_ensVoteRows + + ;
g_ensVoteTime [ row ] = t ;
g_ensVoteSum [ row ] = 0.0 ;
g_ensVoteMask [ row ] = 0 ;
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
g_ensVoteVoterMask [ row ] = 0 ;
feat(vote): thresholds become confidence percentages, on ONE scale everywhere
User request: "the entry/exit thresholds are manual numbers, I would like
them to be confidence percentages, so the current 20 would be only 20%
confidence in a profitable trade."
WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's
contribution are win rates: the pattern weight is that pattern's measured win
rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's
average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT
therefore produced a mean of PRODUCTS of two win rates - a genuinely
60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was
never on a probability scale, so its magnitude meant nothing on its own.
Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which
is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60;
MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight
stops being a discount on the probability and becomes how much a filter's
opinion COUNTS - which is what a module weight should always have been.
Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed.
ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three
other places compared against a 0..1 softmax confidence and would each have
become a fresh currency mismatch the moment the input changed meaning:
* the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now
reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the
classic side, which is the only reason that route exists - against the
same m_threshold_close the averaged vote uses. m_ai_exit_threshold is
retired rather than left dangling.
* m_oosDecisionSeries now carries the vote, not the confidence, so the exit
SIMULATION stops modelling a close rule the EA does not run.
* ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input
through that would have silently switched vote exits off in the
simulation while live went on running them - found before it shipped;
the bound now tracks the scale.
LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing,
SL/TP scaling and the intelligent trailing want a model confidence, not a win
rate.
CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a
real probability to the extent the pattern weights are. A pattern with fewer
than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a
designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the
signal DB fills, "60" means "the designed conviction of the patterns that
fired". Closing that gap is the next commit.
Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales
this removes.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
g_ensVoteWeightSum [ row ] = 0.0 ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- outcomes and label come from the shared label cache, so they are identical across
//--- members - whichever member reaches the bar first writes them
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
g_ensVoteWinLong [ row ] = winLong ;
g_ensVoteWinShort [ row ] = winShort ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
g_ensVoteDirLabel [ row ] = dirLabel ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
}
if ( ( g_ensVoteMask [ row ] & bit ) ! = 0 )
return ; // already contributed to this bar this era (defensive - a re-run must not double-count)
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
g_ensVoteSum [ row ] + = signedVote ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
g_ensVoteMask [ row ] | = bit ;
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- VOTER, not merely present. -0.0 compares equal to 0.0, so an abstention that arrived with a
//--- negative zero is still correctly excluded here.
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials
Era-680 report, all three observations one equation: "peak 29, no arrows at
threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows
everywhere". Under the voters-only divisor, any bar with at least one
directional voter read the weighted mean of the firing tiers' weights - and
once the tiers self-ranked to each model's pooled win rate (~28-31), that
mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four
unanimous: ~29. Min_Vote_Open was a step function around that constant -
above it nothing ever fired, below it everything did - and the label's 12
was a 3v1 split netting through the same divisor. Not three display bugs:
one arithmetic that could not express agreement.
The divisor is now the CAPABLE weight - every filter that could vote,
whether it did or not:
* live (Direction): VoteCapableWeight() - classic pattern ladders always,
veto filters never, AI members once past the same readiness test
LongCondition gates on. A model still training must not dilute an
ensemble it cannot join: four trainees + one deployed model is a solo
chart wearing an ensemble label, and the solo vote reads full strength.
* gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every
member that EVALUATED the bar, Neutral included.
* overlay sweep + prospective readout: weight counts whenever the member
has data; a snapshotted Neutral dilutes.
One arithmetic, four sites, same numbers everywhere.
What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 -
the CEILING, which is the pooled win rate and is what the peak displays;
3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly
three-quarters of the ensemble's trust agrees, net". It MUST sit below the
ceiling to ever fire - the census/peak states the ceiling.
This is the ensemble the user specified in the original design discussion
("if the perceptron also votes, both together reach the threshold; if
another NN votes the other side, the threshold is not reached") - union
semantics was the pre-ensemble behaviour, kept until measurement showed its
vote magnitude was a constant.
Plus overlay DECLUSTERING, the other half of "arrows on every bar": the
same three NMS rules as the per-member arrows (same-direction runs collapse
to their first bar, cross-direction flicker keeps the stronger side), online
over the sweep's strictly oldest->newest walk. Suppression is a verdict and
deletes a standing arrow; the den==0 no-data skip still never does.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
//--- CONSENSUS DENOMINATOR: the member's weight counts because it CONTRIBUTED (it evaluated
//--- the bar and had a say), not because it voted a direction - a Neutral dilutes the vote
//--- here exactly as it does live. The voter mask stays direction-only: "did anyone actually
//--- vote" still gates whether the bar can fire at all.
g_ensVoteWeightSum [ row ] + = voteWeight ;
fix(gate): the ensemble gate certified a vote the EA never casts
g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on
the same criterion the live trade does". It did not. Two independent
mismatches, both silent:
CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100
number straight off the softmax head. Live contributes m_weight x the tier's
pattern weight, and BOTH of those are rewritten from the signal DB by
UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share
an axis and nothing relates them, so the same bar was one number to the gate
and a different one to the order path. Same shape as the 2026-08-09 geometry
incident: certified on one game, paid on another.
DENOMINATOR. The gate divided by the member count, so an abstaining member
pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero
contribution in BOTH the sum and the count (`if(direction == 0) continue;`
before `number++`) - live is a mean over VOTERS. The gate was therefore
scoring a strictly more agreement-heavy set of bars than the EA trades. The
contribution hook's own comment asserted the opposite ("abstentions dilute
the average exactly as they do in the live vote"), while the AI_CHOICE enum
20 lines away correctly documented union semantics.
LiveVoteContribution() is now the single definition of "what this member
votes", called from the gate; the live path reaches the same arithmetic
through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually
voted, separately from who evaluated the bar, because those are the divisor
and the shared-population test respectively.
ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar
wrapper - the OOS scan holds the scanned bar's decision in a local, and
dPrevSignal is a different bar.
NOT changed, deliberately: the gate still does not model live NMS
declustering, and the per-member solo gate still scores every directional
call rather than threshold-clearing ones. Both are selection-metric changes
and this codebase has twice been bitten by switching one blind.
Also corrects a stale paragraph in m_pattern_0's declaration block quoting
80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has
been since the confidence floor and alternation gate were removed; the block
carried both tables at once, and the dead one was quoted back as fact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
if ( signedVote ! = 0.0 )
g_ensVoteVoterMask [ row ] | = bit ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
g_ensVoteCursor [ m_ensembleIndex ] = row + 1 ;
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- This member's just-finished era, held until the ensemble verdict can act on it. Members reach
//--- their era end at different moments within a round, so the last one to arrive needs every
//--- member's figures - and, if the era wins, commits them as that member's best-checkpoint stats
//--- (see EnsembleCommitJointCheckpoint). Same quantities the solo gate keeps in m_best*.
double m_eraStatPrecPct ;
double m_eraStatChancePct ;
int m_eraStatCalls ;
bool m_eraStatTradeable ;
bool m_eraStatTwoSided ;
double m_eraStatScore ;
double m_eraStatBlended ;
double m_eraStatThreshold ;
//--- Which era this member's in-memory snapshot belongs to (-1 = none). The deploy gate requires
//--- EVERY member to be stamped with the SAME era as the winning vote: without it, a member whose
//--- capture failed would still be carrying a snapshot from an earlier era and the quartet that
//--- deployed would be one no combined measurement ever covered - the precise failure the joint
//--- checkpoint exists to rule out.
long m_checkpointEra ;
void EnsembleStashEraStats ( const double precPct , const double chancePct , const int calls ,
const bool tradeable , const bool twoSided , const double score ,
const double blended )
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
{
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_eraStatPrecPct = precPct ;
m_eraStatChancePct = chancePct ;
m_eraStatCalls = calls ;
m_eraStatTradeable = tradeable ;
m_eraStatTwoSided = twoSided ;
m_eraStatScore = score ;
m_eraStatBlended = blended ;
//--- the operating point belongs with the weights it was fitted for - see m_bestDirConfThreshold
m_eraStatThreshold = m_dirConfThreshold ;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- Called once per era from the era-end block, after this member's statistics are final. Marks
//--- this member done; the LAST still-training member to arrive runs the whole ensemble verdict
//--- (score the vote, rank the era, capture the joint checkpoint, advance the shared ladder, and
//--- decide deployment) for every member at once. `etaLocal` is the caller's own live learning rate,
//--- passed by reference because a shared warm restart has to reach the member that is mid-Train().
//--- Defined in Training.mqh - it needs NormalUpperTail() and the PLATEAU_* machinery.
void EnsembleOosPassComplete ( const long votedEra , double & etaLocal ) ;
//--- The verdict itself, and its pieces. needMask names the members whose reads the vote is built
//--- from (still-training members only - a paused or deployed member is not voting in training).
void EnsembleEraVerdict ( const int needMask , const long votedEra , double & etaLocal ) ;
void EnsembleCommitJointCheckpoint ( const long votedEra ) ;
//--- Does the best combined-vote era survive having been CHOSEN out of g_ensCandidateEras eras?
//--- Identical construction to BestCheckpointSurvivesSelection, applied to the vote.
bool EnsembleSurvivesSelection ( double & zObs , double & pFamily , int & nTried ) ;
2026-08-15 16:54:43 -04:00
//--- Single choke point for this signal's on-chart status text. Solo charts draw the full panel
//--- exactly as before; an ensemble member contributes only its headline (first line) to the one
//--- combined panel - see the ENSEMBLE PANEL block in System\StatusLabel.mqh. Every AI-side
//--- SetStatusLabel call site routes through here so no mode can regress into four stacked panels.
void PublishStatus ( const string text , const bool force = false )
{
if ( ! m_ensembleMember )
{
SetStatusLabel ( text ) ;
return ;
}
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first
THE ANSWER, off the instrumentation added in be39674, first run:
ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable
indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ...
Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982
MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not
short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem;
the previous session's five theories were all answering the wrong question.
And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on
that same chart at 15:24:18 and went on to train normally (feature health, 51
features, excursion head) reading the same indicator. Only ConvLSTM's handle -
the last member constructed - was dead. WHY is still not established. All four
members request ADMovingAverage with identical params, so MT5 hands them the SAME
refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over
exactly that shared handle; that is the obvious suspect and it is NOT yet proven,
so this commit does not act on it.
1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths.
"MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982
(h12)" says it is the SAME handle and someone released it; "(h-1)" says it was
never created. That is the difference between a refcount bug and a creation
failure and it is one field. This is the measurement the shared-handle suspicion
needs before anyone acts on it.
2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT.
A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles()
re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD
indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does
NOT release first: -1 means the terminal no longer knows the handle, so there is
nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could
decrement whatever now owns that number. 30s cooldown, because every ServableBars()
consumer reaches it including live inference on every tick. The feature cache is
dropped with it, and the log names before/after depths.
Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars
against a buffer that answers EMPTY_VALUE at every index - then discarding the era
and doing it again - is not a recovery.
3. THE SWEEP NOW HOLDS ON A DEAD HANDLE.
ServableBars() keeps answering `want` (its contract; live inference and online
learning have their own refusal paths and a 0 there reads as "no history at all").
SettledBars() - the training sweep's entry, the one caller that can afford to wait -
returns 0 instead, so Train() holds and reports rather than burning a full-history
pass it is guaranteed to throw away. A recreated handle is cold, so it primes
through the existing settle path on the next call. If the repair fails the member
holds indefinitely and says so every minute, and be39674's barrier liveness escape
releases the rest of the ensemble after 12 minutes - which is the correct
degradation and is exactly what the log shows happening.
4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST.
Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of
Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the
next free row on each member's FIRST PublishStatus() call, so the order was a race -
the members busy sweeping published before the ones sitting idle at the era barrier,
and be39674 sharpened it by (correctly) making a held member stop writing the terse
line. Rows are now keyed to m_ensembleIndex, the registration/construction order,
which is fixed for the life of the chart. Claimed on every publish rather than once,
so it is idempotent and refreshes the tag for a member whose ID was not final when it
first published (the config-tag suffix is appended during InitIndicators, after
EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded
from the model count, so a member that has not published yet leaves no gap and shifts
nobody.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
//--- Called EVERY publish, not just the first. The row is keyed to m_ensembleIndex so the call is
//--- idempotent and cheap, and re-asserting it refreshes the tag for a member whose ID was not yet
//--- final when it first published (the config-tag suffix is appended during InitIndicators, after
//--- EnsembleMember() registers). The old "claim once, first publisher wins the next free row"
//--- form is what ordered the panel by who was busiest instead of by member index.
m_ensemblePanelSlot = ClaimEnsemblePanelSlot ( DisplayName ( ) , m_ensembleIndex ) ;
2026-08-15 16:54:43 -04:00
int nl = StringFind ( text , " \n " ) ;
PublishEnsembleStatus ( m_ensemblePanelSlot , ( nl > 0 ) ? StringSubstr ( text , 0 , nl ) : text , force ) ;
}
2026-07-24 11:52:19 -04:00
void EnableOnlineLearning ( bool value ) { m_enableOnlineLearning = value ; }
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks
Two things, one of which was a compile error.
1. ExitPolicy() was declared in the protected block but is pushed in from
Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters.
2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured
from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot
begin until whatever is in flight returns - so a scan still running after
_StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge
never gets its turn.
New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress.
Deliberately NOT m_trainingStopRequested: that latches, and a latched flag
would permanently disable scans that must run again on the next Start.
Guarded, longest first:
- TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings
on the way out (best[] is mutated in place; the tuner otherwise keeps the
last trial's parameters, which nothing chose).
- ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so
m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar
read the last candidate's multiples as the configured geometry).
- ReportFeatureLabelInformation / ReportExcursionInformation / lag profile -
nulls ABANDON rather than truncate: fewer draws is not a smaller null, it
is a wrong one, and p shifts toward significance. m_dirEvidence staying
false is the safe direction.
- SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line
is dropped instead of latching a partial expectancy as the run's only report.
- ReportGeometryExpectancyScan - per ladder rung.
- HttpGet - one choke point for up to a dozen blocking WebRequests per
first-pass Update(). An in-flight request cannot be cancelled; refusing to
start another is the whole remedy.
- PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain -
entry points, so a queued event cannot open an era during teardown.
TuneIndicatorsAndTrain's guard is the first statement, ahead of the
m_tuneFilterDone / g_ensembleChartTuneDone latches.
- OnTick / OnTimer / OnChartEvent.
Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3
yield on a 120 ms budget); the warm-up scans did not, and they are the longest
uninterruptible stretches the EA has.
StopTraining() is unchanged: the operator's Stop still finalises synchronously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
//--- Exit policy, pushed in from Warrior_EA.mq5 so the gate grades the same rule the live path runs.
feat(vote): thresholds become confidence percentages, on ONE scale everywhere
User request: "the entry/exit thresholds are manual numbers, I would like
them to be confidence percentages, so the current 20 would be only 20%
confidence in a profitable trade."
WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's
contribution are win rates: the pattern weight is that pattern's measured win
rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's
average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT
therefore produced a mean of PRODUCTS of two win rates - a genuinely
60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was
never on a probability scale, so its magnitude meant nothing on its own.
Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which
is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60;
MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight
stops being a discount on the probability and becomes how much a filter's
opinion COUNTS - which is what a module weight should always have been.
Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed.
ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three
other places compared against a 0..1 softmax confidence and would each have
become a fresh currency mismatch the moment the input changed meaning:
* the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now
reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the
classic side, which is the only reason that route exists - against the
same m_threshold_close the averaged vote uses. m_ai_exit_threshold is
retired rather than left dangling.
* m_oosDecisionSeries now carries the vote, not the confidence, so the exit
SIMULATION stops modelling a close rule the EA does not run.
* ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input
through that would have silently switched vote exits off in the
simulation while live went on running them - found before it shipped;
the bound now tracks the scale.
LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing,
SL/TP scaling and the intelligent trailing want a model confidence, not a win
rate.
CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a
real probability to the extent the pattern weights are. A pattern with fewer
than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a
designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the
signal DB fills, "60" means "the designed conviction of the patterns that
fired". Closing that gap is the next commit.
Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales
this removes.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
//--- voteThreshold is Min_Vote_Close UNSCALED, on the same 0-100 confidence scale as the live
//--- close threshold (>100 disables it by arithmetic, exactly as live does). The bound moved
//--- with the 2026-08-18 currency change and MUST track it: while this still clamped at >1.0 a
//--- caller passing the unscaled input would have had EVERY value above 1 silently collapse to
//--- 0.0, switching vote exits off in the simulation while live went on running them.
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks
Two things, one of which was a compile error.
1. ExitPolicy() was declared in the protected block but is pushed in from
Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters.
2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured
from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot
begin until whatever is in flight returns - so a scan still running after
_StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge
never gets its turn.
New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress.
Deliberately NOT m_trainingStopRequested: that latches, and a latched flag
would permanently disable scans that must run again on the next Start.
Guarded, longest first:
- TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings
on the way out (best[] is mutated in place; the tuner otherwise keeps the
last trial's parameters, which nothing chose).
- ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so
m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar
read the last candidate's multiples as the configured geometry).
- ReportFeatureLabelInformation / ReportExcursionInformation / lag profile -
nulls ABANDON rather than truncate: fewer draws is not a smaller null, it
is a wrong one, and p shifts toward significance. m_dirEvidence staying
false is the safe direction.
- SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line
is dropped instead of latching a partial expectancy as the run's only report.
- ReportGeometryExpectancyScan - per ladder rung.
- HttpGet - one choke point for up to a dozen blocking WebRequests per
first-pass Update(). An in-flight request cannot be cancelled; refusing to
start another is the whole remedy.
- PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain -
entry points, so a queued event cannot open an era during teardown.
TuneIndicatorsAndTrain's guard is the first statement, ahead of the
m_tuneFilterDone / g_ensembleChartTuneDone latches.
- OnTick / OnTimer / OnChartEvent.
Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3
yield on a 120 ms budget); the warm-up scans did not, and they are the longest
uninterruptible stretches the EA has.
StopTraining() is unchanged: the operator's Stop still finalises synchronously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
void ExitPolicy ( double voteThreshold , bool holdToBarrier )
{
feat(vote): thresholds become confidence percentages, on ONE scale everywhere
User request: "the entry/exit thresholds are manual numbers, I would like
them to be confidence percentages, so the current 20 would be only 20%
confidence in a profitable trade."
WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's
contribution are win rates: the pattern weight is that pattern's measured win
rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's
average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT
therefore produced a mean of PRODUCTS of two win rates - a genuinely
60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was
never on a probability scale, so its magnitude meant nothing on its own.
Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which
is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60;
MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight
stops being a discount on the probability and becomes how much a filter's
opinion COUNTS - which is what a module weight should always have been.
Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed.
ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three
other places compared against a 0..1 softmax confidence and would each have
become a fresh currency mismatch the moment the input changed meaning:
* the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now
reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the
classic side, which is the only reason that route exists - against the
same m_threshold_close the averaged vote uses. m_ai_exit_threshold is
retired rather than left dangling.
* m_oosDecisionSeries now carries the vote, not the confidence, so the exit
SIMULATION stops modelling a close rule the EA does not run.
* ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input
through that would have silently switched vote exits off in the
simulation while live went on running them - found before it shipped;
the bound now tracks the scale.
LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing,
SL/TP scaling and the intelligent trailing want a model confidence, not a win
rate.
CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a
real probability to the extent the pattern weights are. A pattern with fewer
than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a
designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the
signal DB fills, "60" means "the designed conviction of the patterns that
fired". Closing that gap is the next commit.
Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales
this removes.
NOT COMPILED - user compiles in MetaEditor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
m_exitVoteThreshold = ( voteThreshold > 100.0 ) ? 0.0 : voteThreshold ;
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks
Two things, one of which was a compile error.
1. ExitPolicy() was declared in the protected block but is pushed in from
Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters.
2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured
from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot
begin until whatever is in flight returns - so a scan still running after
_StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge
never gets its turn.
New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress.
Deliberately NOT m_trainingStopRequested: that latches, and a latched flag
would permanently disable scans that must run again on the next Start.
Guarded, longest first:
- TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings
on the way out (best[] is mutated in place; the tuner otherwise keeps the
last trial's parameters, which nothing chose).
- ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so
m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar
read the last candidate's multiples as the configured geometry).
- ReportFeatureLabelInformation / ReportExcursionInformation / lag profile -
nulls ABANDON rather than truncate: fewer draws is not a smaller null, it
is a wrong one, and p shifts toward significance. m_dirEvidence staying
false is the safe direction.
- SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line
is dropped instead of latching a partial expectancy as the run's only report.
- ReportGeometryExpectancyScan - per ladder rung.
- HttpGet - one choke point for up to a dozen blocking WebRequests per
first-pass Update(). An in-flight request cannot be cancelled; refusing to
start another is the whole remedy.
- PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain -
entry points, so a queued event cannot open an era during teardown.
TuneIndicatorsAndTrain's guard is the first statement, ahead of the
m_tuneFilterDone / g_ensembleChartTuneDone latches.
- OnTick / OnTimer / OnChartEvent.
Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3
yield on a 120 ms budget); the warm-up scans did not, and they are the longest
uninterruptible stretches the EA has.
StopTraining() is unchanged: the operator's Stop still finalises synchronously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
m_exitHoldToBarrier = holdToBarrier ;
}
2026-07-14 22:36:27 -04:00
void MaxErasPerRun ( int value ) { m_maxErasPerRun = value ; }
void OOSSplit ( int value ) { m_oosSplitPct = value ; }
void HistoryBars ( int value ) { m_historyBars = value ; }
void MinTrainYear ( int value ) { m_minTrainYear = value ; }
void UseVolumes ( bool value ) { m_useVolumes = value ; }
void UseTime ( bool value ) { m_useTime = value ; }
void UseATR ( bool value ) { m_useATR = value ; }
2026-07-22 17:17:23 -04:00
void UseMA ( bool value ) { m_useMA = value ; }
void UseRSI ( bool value ) { m_useRSI = value ; }
2026-07-26 18:33:12 -04:00
void UseMACD ( bool value ) { m_useMACD = value ; }
void UseIchimoku ( bool value ) { m_useIchimoku = value ; }
2026-07-19 11:04:38 -04:00
void UseSwingContext ( bool value ) { m_useSwingContext = value ; }
feat: add configurable news event proximity/impact as an NN input feature
Price, time, volume, and volatility were already trained-model input
features; the real economic calendar (already used for the live
NewsFilter veto) is now an optional one too, reusing
System/NewsRelevance.mqh's symbol-relevance logic from the prior fix.
New EnableNews/NewsFeatureWindowMinutes inputs gate two features per
bar: minutes-since and minutes-until the nearest symbol-relevant
calendar event, impact-weighted. Deliberately limited to proximity +
impact, not actual-vs-forecast deviation - release schedules are
public knowledge ahead of time (not lookahead bias to use for a
historical training bar), but a release's actual outcome is not.
Wired identically to the existing EnableVolume/EnableTime/EnableATR
toggles: InitIndicators() accounts for the +2 neuron count,
BufferTempDataCompute() appends the two feature values, PAI/CONV/LSTM
all wired in Warrior_EA.mq5. Compiled clean (MetaEditor, 0 errors/0
warnings).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 17:26:04 -04:00
void UseNews ( bool value ) { m_useNews = value ; }
void NewsFeatureWindowMinutes ( int value ) { m_newsFeatureWindowMinutes = value ; }
fix(signals): revive a dead MA model, and demote Sanyaku from state to event
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
void UseCrossAsset ( bool value ) { m_useCrossAsset = value ; }
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks
Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature,
default on). Spread is the one microstructure channel that is both FX-available and
genuinely historical in the Strategy Tester - "during testing, the spread is not modeled
but is taken from historical data" - so unlike swap, signed tick flow or depth of market it
is something a backtest can honestly validate.
What it encodes, stated precisely because the raw measurement overstates it.
research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5
of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges
the spread inside its own barriers, so a wide-spread bar is mechanically likelier to
resolve as a loss and the feature would partly be predicting its own cost model. Relabelling
at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology
and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime
reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when
realised volatility is below its own ATR estimate, which genuinely predicts whether
ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side.
Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated
in the spread series. Both cached on length alone:
if(m_crossAsset.Bars() >= bars) return true;
MQL5 series indices are relative to NOW, so one new closed candle shifts every index by
one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer
the newest, and every cross-asset value is read one bar out of step with the price features
sitting beside it in the same vector - silently, with no error and no shape change. This is
the same class of defect as the dtStudied watermark behind the zero-direction backtests.
Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the
label/feature bar caches already use.
And a performance fix that fell out of it: with correct invalidation the panel rebuilds on
every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one
full multi-symbol resample per simulated bar at training depth. Inference only reads bars
0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The
cache check is >=, so a deeper panel left from training still satisfies it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
void UseSpreadFeature ( bool value ) { m_useSpreadFeature = value ; }
2026-07-14 22:36:27 -04:00
void UseADCumulativeDelta ( bool value ) { m_useADCumulativeDelta = value ; }
void UseADShorteningOfThrust ( bool value ) { m_useADShorteningOfThrust = value ; }
void UseADWyckoffEventStream ( bool value ) { m_useADWyckoffEventStream = value ; }
void UseADWyckoffFailedStructure ( bool value ) { m_useADWyckoffFailedStructure = value ; }
void UseADWyckoffSignificantBarInversion ( bool value ) { m_useADWyckoffSignificantBarInversion = value ; }
void AutoTuneIndicators ( bool value ) { m_autoTuneIndicators = value ; }
2026-08-16 15:12:54 -04:00
void UseAltData ( bool value ) { m_altDataEnabled = value ; }
2026-07-14 22:36:27 -04:00
//--- control-panel API (Warrior_EA.mq5): current-config-only training/weights control.
//--- "current config" == this signal instance's own m_fileName (symbol+period+id+topology),
//--- never touches another signal type's or another symbol/timeframe's saved files.
void PauseTraining ( void ) { m_trainingPaused = true ; PrintVerbose ( ID + " : training paused by user (era " + IntegerToString ( m_eraCount ) + " ) " ) ; }
void ResumeTraining ( void ) { m_trainingPaused = false ; PrintVerbose ( ID + " : training resumed by user (era " + IntegerToString ( m_eraCount ) + " ) " ) ; }
bool IsTrainingPaused ( void ) const { return m_trainingPaused ; }
bool IsTrainingStopped ( void ) const { return m_trainingStopRequested ; }
bool TrainingComplete ( void ) const { return m_trainingComplete ; }
fix(deinit): a full model write was running ahead of the cheap cleanup
"Abnormal termination" is back, and this time it is not the arrows. The
timing names the culprit exactly:
16:02:31.547 OnDeinit: shutting down
16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up
16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE
OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining()
finalises an in-flight run, and FinalizeTrainRun() restores the best
checkpoint and then persists it - a full ~1MB model write per signal. So
the expensive step ran ahead of the cheap bounded one, which is precisely
the inversion the shutdown ordering exists to prevent. The previous fix
put PersistWeightsOnShutdown last and missed that StopTraining smuggles a
second save in at the front.
Two changes:
Cleanup now runs FIRST, then StopTraining, then the weight save. The
visible teardown is cheap and bounded, so it always completes even when
everything after it is killed.
And the deploy-persist inside FinalizeTrainRun is suppressed during
shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint
is already the live net by that line, and PersistWeightsOnShutdown writes
exactly those weights moments later. The old path wrote the same model
twice per signal - eight full writes across four charts - for no benefit.
A user-pressed Stop still persists immediately, because nothing else
would.
Compiles 0 errors / 0 warnings. Build tag deinit-order-v2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- Set by OnDeinit before it calls StopTraining(), so FinalizeTrainRun() can tell a user-pressed Stop
//--- (persist the deployed model now - nothing else will) from a shutdown (PersistWeightsOnShutdown is
//--- moments away and writes the same bytes). See the guard in FinalizeTrainRun.
void MarkShutdown ( void ) { m_shutdownInProgress = true ; }
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks
Two things, one of which was a compile error.
1. ExitPolicy() was declared in the protected block but is pushed in from
Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters.
2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured
from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot
begin until whatever is in flight returns - so a scan still running after
_StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge
never gets its turn.
New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress.
Deliberately NOT m_trainingStopRequested: that latches, and a latched flag
would permanently disable scans that must run again on the next Start.
Guarded, longest first:
- TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings
on the way out (best[] is mutated in place; the tuner otherwise keeps the
last trial's parameters, which nothing chose).
- ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so
m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar
read the last candidate's multiples as the configured geometry).
- ReportFeatureLabelInformation / ReportExcursionInformation / lag profile -
nulls ABANDON rather than truncate: fewer draws is not a smaller null, it
is a wrong one, and p shifts toward significance. m_dirEvidence staying
false is the safe direction.
- SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line
is dropped instead of latching a partial expectancy as the run's only report.
- ReportGeometryExpectancyScan - per ladder rung.
- HttpGet - one choke point for up to a dozen blocking WebRequests per
first-pass Update(). An in-flight request cannot be cancelled; refusing to
start another is the whole remedy.
- PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain -
entry points, so a queued event cannot open an era during teardown.
TuneIndicatorsAndTrain's guard is the first statement, ahead of the
m_tuneFilterDone / g_ensembleChartTuneDone latches.
- OnTick / OnTimer / OnChartEvent.
Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3
yield on a 120 ms budget); the warm-up scans did not, and they are the longest
uninterruptible stretches the EA has.
StopTraining() is unchanged: the operator's Stop still finalises synchronously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
//--- THE ONE QUESTION every long loop in this class must ask: has this program been asked to stop?
//--- MetaTrader's ~4,500 ms teardown budget is measured from the STOP REQUEST, not from OnDeinit's
//--- first line, and OnDeinit cannot even begin until whatever is in flight returns. So a scan that
//--- runs for three seconds after _StopFlag is raised does not merely delay shutdown - it SPENDS the
//--- cleanup budget, and the chart keeps the status label, the panel and every arrow because the
//--- purge never got its turn. Training's bar loops have honoured this since they were written; the
//--- warm-up scans, the permutation nulls and the label prebuild did not, and they are the longest
//--- uninterruptible stretches the EA has.
//--- Deliberately NOT including m_trainingStopRequested: that is the operator's Stop-training button,
//--- it latches, and a latched flag would permanently disable scans that must still run on the next
//--- Start. These two are terminal - once either is true the program is going away.
bool ShutdownRequested ( void ) const { return ( IsStopped ( ) | | m_shutdownInProgress ) ; }
fix(reset): say what the reset actually did, per member and per file
The user reports "Delete & Reset Weights only wipes the first NN". I could not
find a code path that skips ensemble members, and I am not going to assert one:
the handler loops g_aiSignals[0..g_aiSignalCount), all four topologies register
unconditionally in OnInit, and SetIdentity gives each its own State\<id>\ folder
so the six deleted paths are genuinely distinct per member. What IS true is that
the whole success path was SILENT - six FileDelete calls per member printing only
on failure, and one chart-wide Alert - so a four-member reset and a one-member
reset produce byte-identical output. The symptom could be neither confirmed nor
refuted from a log. That is the defect I can fix today.
- COMPILED <timestamp> (__DATETIME__) beside the build tag. The hand-edited tag
had sat at scan-nofwd-v5 across a week of commits, so it could not answer the
question it exists for. The compile stamp cannot be forgotten. Tag bumped to
reset-census-v6.
- RegistryLine() (public): ID, active file path, common/local, era, deployed vs
training, ensemble index. The reset handler prints a numbered census of the
whole registry BEFORE the confirm dialog. If that says 1 on an AI_HYBRID chart
the fault is registration, not the reset - and RegisterAISignal already has a
loud MAX_AI_SIGNALS message for exactly that.
- The confirmation dialog now names the count, so a wrong registry is visible
before anything is deleted rather than after.
- ResetWeights prints one line per member: N deleted / N already absent / N
FAILED, plus a per-suffix breakdown. "absent" on a member that should have had
a .nnw is a completely different fault from "deleted"; they were identical.
- ResetWeights' return value was discarded. A member whose BuildFreshTopology
fails has had its files deleted and has no network - and the Alert still said
"weights reset". Counted now, with an INCOMPLETE alert when they disagree.
- Same for dbm.ResetDatabase(), whose bool was also dropped. The DB is one shared
file for every signal on the chart, so there is nothing per-member to loop -
the log now says that explicitly, since it is the question being asked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:57:26 -04:00
//--- ONE LINE OF IDENTITY, for the census Warrior_EA.mq5 prints before it acts on g_aiSignals[].
//--- Public because the EA has to be able to NAME what it is iterating: every panel action loops over
//--- that registry and, until now, none of them said how many members they reached or which files they
//--- touched. "The reset only wiped the first model" was therefore impossible to confirm OR refute
//--- from a log - the loop and a one-member loop produce identical output. This makes them different.
string RegistryLine ( void ) const
{
return StringFormat ( " %s | %s (%s) | era %d | %s%s " , ID , m_activeFileName ,
( m_activeFileCommon ? " common " : " local " ) , ( int ) m_eraCount ,
( m_trainingComplete ? " deployed " : " training " ) ,
( m_ensembleMember
? StringFormat ( " | ensemble member %d " , m_ensembleIndex ) : " | solo " ) ) ;
}
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach
Chart objects live in the MT5 chart PROFILE, not in this EA's files.
They survive a terminal restart, a recompile, and deleting every
.nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION
removes them - and MetaTrader force-terminates OnDeinit at roughly
4,500 ms, so a run killed mid-cleanup orphans them permanently with no
owner left to clean up after. That is the "deleted every file,
recompiled, restarted, old arrows and a stale panel still there"
report: nothing was wrong with the files and deleting them could not
have helped.
Both halves are fixed.
STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight
run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state
re-seed) and then write two full nets per chart. On four charts that is
the bulk of the budget, spent to preserve a PARTIAL era that was never
scored, never checkpointed and never deployable. FlushTrainRun()
discards it instead - drop the resumable bookkeeping, leave the net
neutral (unfreeze BN, flush the batch, batch size 1), skip the save -
and training resumes from the last completed era, which the era-end
save and the periodic autosave have already put on disk. What is
discarded is bounded by one era.
A CONVERGED model keeps the old finalise-and-save path: its weights can
carry online-learning updates made since the last era boundary, and for
a deployed model no further era boundary is coming to persist them.
MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model
loaded, sidecar missing - so the common paths returned leaving whatever
the previous instance stranded. LoadChartSignals now sweeps the arrow
namespace unconditionally before restoring, so the post-init chart
holds exactly what the sidecar holds whichever branch runs, and the
panel gets the same treatment before Create() (CAppDialog namespaces
its controls, so a killed Destroy strands the lot and the next attach
draws a second panel on the corpse).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
//--- SHUTDOWN FLUSH: abandon an in-flight run instead of finishing it, and resume from the last
//--- COMPLETED, already-persisted era. StopTraining() does the opposite - it finalises synchronously
//--- (restore the best checkpoint from its file, re-seed live state) and OnDeinit then writes two
//--- full nets per chart. On four charts that is the bulk of the deinit budget, spent to preserve a
//--- PARTIAL era that was never scored, never checkpointed and never deployable - and the price of
//--- overrunning is that the chart cleanup gets killed and strands its arrows and panel in the chart
//--- profile, where nothing will ever clean them up (see LoadChartSignals' orphan sweep).
//--- Every completed era is already on disk: the era-end save and the periodic autosave both persist
//--- independently, so what this discards is bounded by one era of work.
//--- Returns true if there really was an in-flight run to discard, so the caller can tell a flush
//--- from a no-op and skip the weight write only when the flush actually applied.
bool FlushTrainRun ( void )
{
bool inFlight = ( m_trainRunActive | | m_eraResumePending | | m_labelPrebuildActive | | m_simOosRunActive ) ;
m_trainingStopRequested = true ;
m_trainingPaused = false ;
//--- Drop the resumable bookkeeping WITHOUT calling FinalizeTrainRun: no checkpoint restore, no
//--- persist, no dtStudied advance. The next start re-derives all of it from the saved model.
m_trainRunActive = false ;
m_eraResumePending = false ;
m_haveOosCheckpoint = false ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1 ; // the joint-checkpoint era stamp goes with the snapshot it describes
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach
Chart objects live in the MT5 chart PROFILE, not in this EA's files.
They survive a terminal restart, a recompile, and deleting every
.nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION
removes them - and MetaTrader force-terminates OnDeinit at roughly
4,500 ms, so a run killed mid-cleanup orphans them permanently with no
owner left to clean up after. That is the "deleted every file,
recompiled, restarted, old arrows and a stale panel still there"
report: nothing was wrong with the files and deleting them could not
have helped.
Both halves are fixed.
STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight
run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state
re-seed) and then write two full nets per chart. On four charts that is
the bulk of the budget, spent to preserve a PARTIAL era that was never
scored, never checkpointed and never deployable. FlushTrainRun()
discards it instead - drop the resumable bookkeeping, leave the net
neutral (unfreeze BN, flush the batch, batch size 1), skip the save -
and training resumes from the last completed era, which the era-end
save and the periodic autosave have already put on disk. What is
discarded is bounded by one era.
A CONVERGED model keeps the old finalise-and-save path: its weights can
carry online-learning updates made since the last era boundary, and for
a deployed model no further era boundary is coming to persist them.
MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model
loaded, sidecar missing - so the common paths returned leaving whatever
the previous instance stranded. LoadChartSignals now sweeps the arrow
namespace unconditionally before restoring, so the post-init chart
holds exactly what the sidecar holds whichever branch runs, and the
panel gets the same treatment before Create() (CAppDialog namespaces
its controls, so a killed Destroy strands the lot and the next attach
draws a second panel on the corpse).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
m_labelPrebuildActive = false ;
if ( m_simOosRunActive )
{
delete m_simOosNet ;
m_simOosNet = NULL ;
m_simOosRunActive = false ;
}
//--- Leave the net in the same neutral state FinalizeTrainRun leaves it in - a frozen batch-norm or
//--- a half-filled mini-batch must not be what a later inference path finds. Cheap, unlike the save.
if ( CheckPointer ( Net ) ! = POINTER_INVALID )
{
Net . SetBatchNormFrozen ( false ) ;
Net . FlushBatch ( ) ;
Net . SetBatchSize ( 1 ) ;
}
return inFlight ;
}
2026-07-14 22:36:27 -04:00
void StopTraining ( void )
{
m_trainingStopRequested = true ;
m_trainingPaused = false ;
//--- ScheduleTrainingIfNeeded() refuses to schedule another "New Bar" event while
//--- m_trainingStopRequested is set, so a run interrupted mid-chunk would otherwise never get
//--- called again to finalize (restore the best checkpoint, persist state) - do it synchronously
//--- here instead. Safe to block briefly: this is just a checkpoint file restore, not the
//--- multi-minute bar loop.
if ( m_trainRunActive )
FinalizeTrainRun ( ) ;
Print ( ID + " : training stopped by user (era " + IntegerToString ( m_eraCount ) + " , weights as of last completed era retained) " ) ;
diag: inference-path census, to explain zero-trade backtests
A backtest of the CONVERGED CONV model produced "Final directional result:
0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in
the log could separate the three candidate causes, and each needs a
different fix:
1. RefreshLatestSignal never called (new-bar gate never fires)
2. called, but bailing at one of its two early returns
3. running fine, and the model genuinely answers Neutral every bar
Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown
via StopTraining (which the tester reaches through OnDeinit). Three
increments per bar against a full feedForward - not worth gating.
Ruled out while writing this, so the next session does not re-derive it:
- the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts
at Neutral, so a first Buy would still fire and show up as one non-zero
direction. We saw zero. It IS still a live hazard for a one-sided model -
CONV currently calls Buy:17% Sell:0%, and after the first Buy every later
Buy is suppressed until a Sell that never comes - but it cannot explain
an all-zero run.
- shallow buffers do not hard-fail the feature builder: the swing-context
Donchian loop breaks gracefully when it runs off loaded history. It does
mean converged-path inference computes Donchian/return/SMA features over
a TRUNCATED window versus training, which is a real train/inference skew
worth its own fix, but it degrades features rather than zeroing them.
Both builds 0/0. Diagnostic only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
PrintInferenceTally ( ) ;
}
//--- Inference-path census, printed at shutdown. WHY: a 2026-07-31 backtest of a CONVERGED CONV model
//--- produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore ZERO
//--- trades, and nothing in the log could distinguish the three candidate causes - RefreshLatestSignal
//--- never running, running but bailing at one of its two early returns, or running fine and the model
//--- genuinely answering Neutral every time. Each implies a completely different fix. Counting is the
//--- cheapest way to tell them apart and it costs nothing per bar.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Records one directional decision's passage through the readiness gate in LongCondition()/
//--- ShortCondition(). Called with `directional` = "this condition's own class is what the model
//--- actually answered", so a Neutral bar counts as neither blocked nor passed - the question being
//--- measured is what happened to the calls that HAD something to say.
void NoteVoteGate ( bool directional )
{
if ( ! directional )
return ;
bool open = m_trainingComplete | | ( m_inferenceOnly & & m_modelLoadedFromDisk ) ;
if ( m_voteGateCompleteAtFirst < 0 )
{
m_voteGateCompleteAtFirst = ( int ) m_trainingComplete ;
m_voteGateLoadedAtFirst = ( int ) m_modelLoadedFromDisk ;
}
if ( open )
m_voteGatePassed + + ;
else
m_voteGateBlocked + + ;
}
diag: inference-path census, to explain zero-trade backtests
A backtest of the CONVERGED CONV model produced "Final directional result:
0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in
the log could separate the three candidate causes, and each needs a
different fix:
1. RefreshLatestSignal never called (new-bar gate never fires)
2. called, but bailing at one of its two early returns
3. running fine, and the model genuinely answers Neutral every bar
Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown
via StopTraining (which the tester reaches through OnDeinit). Three
increments per bar against a full feedForward - not worth gating.
Ruled out while writing this, so the next session does not re-derive it:
- the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts
at Neutral, so a first Buy would still fire and show up as one non-zero
direction. We saw zero. It IS still a live hazard for a one-sided model -
CONV currently calls Buy:17% Sell:0%, and after the first Buy every later
Buy is suppressed until a Sell that never comes - but it cannot explain
an all-zero run.
- shallow buffers do not hard-fail the feature builder: the swing-context
Donchian loop breaks gracefully when it runs off loaded history. It does
mean converged-path inference computes Donchian/return/SMA features over
a TRUNCATED window versus training, which is a real train/inference skew
worth its own fix, but it degrades features rather than zeroing them.
Both builds 0/0. Diagnostic only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
void PrintInferenceTally ( void )
{
long attempts = m_refreshOk + m_refreshFailFeatures + m_refreshFailShort ;
if ( attempts < = 0 )
{
Print ( ID + " : inference census - RefreshLatestSignal was NEVER CALLED (0 attempts). The new-bar gate never fired. " ) ;
return ;
}
Print ( ID + " : inference census - " , attempts , " refresh attempts: " , m_refreshOk , " completed, " ,
m_refreshFailFeatures , " bailed in BufferTempData, " , m_refreshFailShort , " bailed on a short feature window " ,
" | decisions Buy: " , m_refreshBuy , " Sell: " , m_refreshSell , " Neutral: " , m_refreshNeutral ) ;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Second half of the census, and the half that separates "the model said nothing" from "the model
//--- spoke and was not allowed to vote" - see m_voteGateBlocked for why that distinction is the whole
//--- point. A run with directional decisions above and voteGate passed:0 below is the readiness-gate
//--- failure, NOT a Neutral model, and the fix is on the model-load path.
if ( m_voteGateCompleteAtFirst < 0 )
Print ( ID + " : inference census - vote gate was NEVER REACHED (no directional decision ever hit "
" LongCondition/ShortCondition). Either every decision was Neutral, or this filter was never polled. " ) ;
else
Print ( ID + " : inference census - vote gate passed: " , m_voteGatePassed , " blocked: " , m_voteGateBlocked ,
" | at first vote trainingComplete= " , ( m_voteGateCompleteAtFirst ! = 0 ? " true " : " false " ) ,
" modelLoadedFromDisk= " , ( m_voteGateLoadedAtFirst ! = 0 ? " true " : " false " ) ,
" inferenceOnly= " , ( m_inferenceOnly ? " true " : " false " ) ,
( m_voteGateBlocked > 0 & & m_voteGatePassed = = 0
? " <-- EVERY directional call was discarded here. This is the zero-direction cause. "
: " " ) ) ;
2026-07-14 22:36:27 -04:00
}
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- The ONLY place the study event is posted: arms bEventStudy with THIS instance's id (so the
//--- handler in OnChartEventHandler(), which matches on m_studyEventId, is the only member that
//--- runs it) and stamps the lost-event watchdog. sparam tags ("New Bar"/"Init"/"Resume"/...) are
//--- purely diagnostic.
bool ArmStudyEvent ( const long lparam , const string tag )
{
bEventStudy = EventChartCustom ( ChartID ( ) , m_studyEventId , lparam , 0 , tag ) ;
if ( bEventStudy )
m_studyArmedTick = GetTickCount ( ) ;
return bEventStudy ;
}
2026-07-14 22:36:27 -04:00
void StartTraining ( void )
{
if ( ! m_trainingStopRequested & & ! m_trainingPaused )
return ;
m_trainingStopRequested = false ;
m_trainingPaused = false ;
if ( ! bEventStudy )
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArmStudyEvent ( ( long ) dtStudied , " Resume " ) ;
2026-07-14 22:36:27 -04:00
Print ( ID + " : training (re)started by user (era " + IntegerToString ( m_eraCount ) + " ) " ) ;
}
2026-07-25 16:39:11 -04:00
//--- Has an era ever cleared the per-class recall floor and been checkpointed this run? This is the
//--- same quality bar the plateau ladder's auto-deploy requires (see PLATEAU_STAGE_DEPLOY), exposed so
//--- the panel can warn before a MANUAL deploy ships a model that ignores Buy or Sell.
bool HasRecallPassingCheckpoint ( void ) const { return m_bestPassedRecall ; }
//--- MANUAL deploy (panel "Deploy Model"): finalise whatever the run has found so far as THE model -
//--- exactly what the plateau ladder does on its own at stage 3, just triggered early by the operator.
//--- Restores the best checkpoint (not whatever era the run happened to be mid-way through), persists
//--- weights + calibration + shadow, and flips to live inference.
//--- Deliberately does NOT set m_trainingStopRequested: like every other deploy path, a deployed model
//--- still runs live inference AND online continual learning. A panel Stop is the thing that halts
//--- everything - see StopTraining()/OnlineLearnStep()'s gate. Reversible via RetrainDeployed().
bool DeployNow ( void )
{
if ( CheckPointer ( Net ) = = POINTER_INVALID | | ! m_isInitialized )
return false ;
if ( m_trainingComplete )
return true ; // already deployed - nothing to do
//--- Set BEFORE any save below: the flag is written INTO the .nnw, so persisting first would store
//--- "still training" and a restart would resume the era loop instead of running the deployed model.
m_trainingComplete = true ;
m_trainingPaused = false ;
m_trainingStopRequested = false ;
if ( m_trainRunActive | | m_haveOosCheckpoint )
FinalizeTrainRun ( ) ; // restores the best checkpoint, persists, ends the run
else
{
//--- Nothing trained this session (e.g. deploying a model that was just loaded from disk), so
//--- there is no in-memory checkpoint to restore - persist exactly what is loaded right now.
PersistDeployedModel ( ) ;
SaveChartSignals ( ) ;
}
RefreshLatestSignal ( ) ;
Print ( ID + " : model DEPLOYED by user at era " + IntegerToString ( m_eraCount ) +
" (balanced accuracy " + ( m_bestBalancedOos < 0 ? " n/a " : DoubleToString ( m_bestBalancedOos , 1 ) + " % " ) +
" , blended OOS " + DoubleToString ( dOosForecast , 1 ) + " %) - training stopped, now running live inference " +
( m_enableOnlineLearning ? " with online continual learning " : " " ) +
" . Use the panel's \" Retrain Model \" to resume training from here. " ) ;
feat: gate deployment on the null of the MAXIMUM, not the per-era null
EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM
over every era a run ranks. A 2-sigma one-sided test passes on noise with
probability 0.0228 per era, so over N eras the chance at least one clears
it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The
gate was near-certain to open on a long run whatever the data held.
It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance -
+1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the
call counts these runs produce that is p_family 0.92..0.9999.
Every OTHER best-of-N decision here already carries this correction, and
every one REJECTS on this data: the barrier-geometry winner (null of the
maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI
lag profile (null of the maximum over 21 lags). The one decision that
ships a model to a live account had none.
BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to
deploy:
z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n)
p_single = P(Z >= z)
p_family = 1 - (1-p_single)^N
against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN
snapshotted precision/chance/call-count, not the latest era's, because
the model that ships is the one that has to clear the bar.
N counts CANDIDATE eras (coverage measurable, at least one directional
call) - an era that called nothing directional could never have become
the best, so counting it would make the gate stricter than the search
that actually happened.
Conservative on purpose: consecutive eras share OOS bars and differ by
one gradient step, so they are nowhere near N independent draws and the
true family-wise error is below this bound. This gate decides what trades
real money and the house posture is reject-unless-demonstrated.
Effect at 2900 directional calls / N=112: required edge goes 1.76pp ->
2.92pp. A real edge clears it; +1.5pp does not.
Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and
the m_trainingComplete assignment - which must stay identical or the flag
persisted into the .nnw disagrees with the decision to stop, and a reload
runs inference on a model the ladder refused.
NOT applied to the two operator paths (era-cap deploy, panel Deploy
button). Those stay the operator's call; ReportSelectionGateVerdict()
logs the verdict beside them so an authorised deploy can never later be
misread as a validated one.
NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather
than pulling in Math\Stat. Verified against reference values to 6dp:
Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are
ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1".
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- Deliberately reported, not enforced: a manual deploy is the operator overriding the ladder, and
//--- that override stays available. But the ladder's own verdict on this checkpoint goes in the log
//--- next to it, so "I shipped this" is never later mistaken for "this passed". See
//--- DEPLOY_FAMILY_WISE_ALPHA and HasRecallPassingCheckpoint()'s panel warning.
ReportSelectionGateVerdict ( " manual deploy " ) ;
2026-07-25 16:39:11 -04:00
return true ;
}
//--- The inverse of DeployNow(), and the ONLY way back: while m_trainingComplete is set,
//--- ScheduleTrainingIfNeeded() routes every tick to the converged/inference branch, so StartTraining()
//--- alone can never revive a deployed model (it clears the stop flag, but the complete flag still wins
//--- that branch). Clearing it here re-arms the normal training path, continuing from the DEPLOYED
//--- weights rather than from scratch - "reset weights" is the separate, destructive button for that.
//--- The plateau ladder, best-checkpoint tracking and annealed gamma all reset themselves when Train()
//--- starts its next fresh run (see the !m_trainRunActive block), so a retrain does not inherit the
//--- exhausted stage that deployed the model and immediately re-deploy it.
void RetrainDeployed ( void )
{
if ( ! m_trainingComplete )
return ;
m_trainingComplete = false ;
m_trainingStopRequested = false ;
m_trainingPaused = false ;
//--- Persist the cleared flag immediately. Otherwise a terminal restart before the first era
//--- completes would reload the .nnw still marked complete and silently go back to inference-only,
//--- looking like the button did nothing.
PersistDeployedModel ( ) ;
if ( ! bEventStudy )
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArmStudyEvent ( ( long ) dtStudied , " Retrain " ) ;
2026-07-25 16:39:11 -04:00
Print ( ID + " : RETRAINING the deployed model from era " + IntegerToString ( m_eraCount ) +
" - keeping its current weights as the starting point (use \" Delete & Reset Weights \" to start from scratch instead). " ) ;
}
2026-07-26 12:36:56 -04:00
//--- Manual "rescan" of the drawn signal arrows: purges every arrow currently on the chart (namespaced
//--- delete - user drawings untouched) and re-infers the last SIGNAL_RESCAN_LOOKBACK_BARS bars from the
//--- CURRENTLY deployed weights, then re-runs the same end-of-era NMS declutter (PruneDirectionalClusters)
//--- used during training so the fresh set matches what a live re-render would have produced. Wired to
//--- the panel's Hide->Show Signals sequence: without this, "restore" only ever replays whatever was
//--- last saved to the .arrows sidecar, which for a long-deployed model can be a stale historical render
//--- from whenever it was last actually trained - years-old arrows crowding out anything recent. Chart-only
//--- (no persistent chart in the tester/optimizer) and a no-op until a model has something to infer with.
2026-07-26 12:52:56 -04:00
//--- This only does the cheap setup (buffer resize, arrow purge, cache alloc) and QUEUES the per-bar
//--- inference loop for AdvanceChartSignalRescan() to drain in time-boxed slices off the timer - see
//--- that method's comment for why the loop itself must never run in one blocking pass. Returns true
//--- once a rescan has been queued (check RescanPending() for completion), false if there was nothing
//--- to rescan (no deployed model, tester/optimizer context, etc).
bool StartChartSignalRescan ( void )
2026-07-26 12:36:56 -04:00
{
if ( MQLInfoInteger ( MQL_TESTER ) | | MQLInfoInteger ( MQL_OPTIMIZATION ) | | MQLInfoInteger ( MQL_FORWARD ) )
return false ;
if ( CheckPointer ( Net ) = = POINTER_INVALID | | ! m_isInitialized | | ! m_trainingComplete )
return false ;
if ( m_outputNeuronsCount ! = 1 & & m_outputNeuronsCount ! = 3 )
return false ;
int barsAvail = Bars ( m_symbol . Name ( ) , PERIOD_CURRENT ) ;
int barsNow = MathMin ( SIGNAL_RESCAN_LOOKBACK_BARS , barsAvail ) ;
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate
1dda479 clamped the training sweep. It left five other paths asking the indicators
for a depth they cannot serve, and on a live account the quiet ones are worse than
the stall was - a stalled chart is visible, a chart trading on a degraded feature
window is not.
ServableBars(want, context) is now the single gate, and all six go through it:
training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small
positive BarsCalculated is warm-up, which m_coldSweepTick owns)
label prebuild clamp - labels come from price/ADZigZag and would survive a
capped MA, but ResizeBuffers sizes EVERY buffer and a failed
CopyBuffer leaves m_MA EMPTY for the next reader, so this path
could silently re-break the block Train()'s clamp just fixed
live inference HOLD. Below `need` the swing block takes its degraded path and
inference runs on a different feature distribution than the model
was fitted on. This EA sizes real positions off that output, so
no signal beats a mismatched one
online learning HOLD, same reason and worse - this path WRITES to a live trading
model, so a mismatched (features, label) pair is not a wrong arrow,
it is a wrong weight update that compounds every bar
chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest
"Max bars in chart" is also 5000, so this one is genuinely
reachable; uncapped it repaints the window all-Neutral
research export clamp before the emptiness test, so a capped symbol exports the
depth it has rather than writing a CSV with a dead feature block -
an artefact that looks complete and is silently wrong
Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars
(16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so
the failure mode is unreachable rather than merely unlikely.
Not changed: a genuinely SHORT price history still takes the old degraded path at
every site. That is pre-existing behaviour and narrowing it would mute charts that
trade today, so it stays a separate decision rather than a side effect of this fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
//--- SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and the SMALLEST "Max bars in chart" setting MT5 offers
//--- is also 5000, so this call site genuinely can be capped - it would redraw the whole rescan
//--- window as Neutral (every feature window rejected) and read as "the model calls nothing".
barsNow = ServableBars ( barsNow , " chart rescan " ) ;
2026-07-26 12:36:56 -04:00
if ( barsNow < = m_historyBars )
return false ;
if ( ! ResizeBuffers ( barsNow ) | | ! RefreshData ( ) )
return false ;
EnsureShadowNet ( ) ;
2026-07-26 14:08:59 -04:00
//--- Drop only the arrows THIS rescan is about to re-judge - i.e. those within [now, oldest bar of
//--- the barsNow window] - not every namespaced arrow on the chart. PruneDirectionalClusters below
//--- only touches indices inside [0, barsNow), so anything older survives untouched either way; the
//--- previous version called ObjectsDeleteAll(0, SIG_ARROW_PREFIX) unconditionally, which wiped
//--- EVERY arrow ever drawn (years of history, going back to whenever this EA was first attached)
//--- and then only ever redrew the last SIGNAL_RESCAN_LOOKBACK_BARS (~7 months on H1) - anything
//--- older was gone permanently the moment Show Signals was clicked, with no way back short of the
//--- .arrows sidecar (itself only ever a snapshot from whatever the LAST save happened to catch).
//--- Reported 2026-07-26: a user's multi-year arrow history vanished down to whatever a stale
//--- .arrows file held, immediately after their first-ever successful Show Signals click. This scoped
//--- delete is the fix - older arrows are never in scope to be wiped in the first place.
datetime rescanCutoffTime = m_Time . GetData ( barsNow - 1 ) ;
for ( int oi = ObjectsTotal ( 0 , 0 , OBJ_ARROW ) - 1 ; oi > = 0 ; oi - - )
{
string onm = ObjectName ( 0 , oi , 0 , OBJ_ARROW ) ;
if ( StringFind ( onm , SIG_ARROW_PREFIX ) ! = 0 )
continue ;
if ( ( datetime ) ObjectGetInteger ( 0 , onm , OBJPROP_TIME ) > = rescanCutoffTime )
ObjectDelete ( 0 , onm ) ;
}
2026-07-26 12:36:56 -04:00
ArrayResize ( m_arrowSignalCache , barsNow ) ;
ArrayInitialize ( m_arrowSignalCache , -2.0 ) ;
2026-07-26 12:52:56 -04:00
m_rescanBarsNow = barsNow ;
m_rescanHi = barsNow - m_historyBars ;
m_rescanIndex = 0 ;
2026-07-26 13:58:05 -04:00
m_rescanRawBuy = 0 ;
m_rescanRawSell = 0 ;
m_rescanRawNeutral = 0 ;
2026-07-26 12:52:56 -04:00
m_rescanPending = ( m_rescanHi > 0 ) ;
m_rescanStartMs = GetTickCount ( ) ;
if ( m_rescanPending )
Print ( ID + " : rescanning last " + IntegerToString ( m_rescanHi ) + " bars against the deployed model (progressive, non-blocking)... " ) ;
return m_rescanPending ;
2026-07-26 12:36:56 -04:00
}
2026-07-26 12:55:31 -04:00
//--- true while a queued rescan (StartChartSignalRescan above) still has slices left for
//--- AdvanceChartSignalRescan to drain - polled by Warrior_EA.mq5's FinalizeSignalsRescanIfDone() to
//--- know when it's safe to (re)apply arrow visibility and report the Show Signals click as complete.
bool RescanPending ( void ) const { return m_rescanPending ; }
2026-07-14 22:36:27 -04:00
//--- forces a save of the network's current in-memory weights/state regardless of era-completion
//--- state; called from OnDeinit() so shutdown/chart-removal never loses more than the current tick
//--- of learning, and a subsequent restart's Train() resumes from m_eraCount rather than the last
//--- fully-completed era only.
2026-07-24 21:56:53 -04:00
//--- Heavy weight/state persistence ONLY (net weights + calibration sidecar + shadow net). Split out
//--- from PersistOnShutdown() so OnDeinit() can run the cheap chart save+purge FIRST: on the CPU-DLL
//--- box this recursive save (two full nets) is the slow/fragile step, and if it ever stalls past MT5's
//--- deinit budget or faults, the arrows and status panel must already be gone, not stranded on the
//--- chart (the reported "cleanup not going well - signals/panel stay after Abnormal termination").
bool PersistWeightsOnShutdown ( void )
2026-07-14 22:36:27 -04:00
{
if ( CheckPointer ( Net ) = = POINTER_INVALID | | ! m_isInitialized )
return false ;
2026-07-26 10:59:46 -04:00
//--- An inference-only run (any Strategy Tester pass - see m_inferenceOnly) trains NOTHING, so there
//--- is no new state to persist and this save can only do harm. It is exactly what corrupted the
//--- tester cache on 2026-07-26: when the seeded load failed, the !netLoaded path reset the in-memory
//--- state to a fresh untrained net (era 0), and this unconditional shutdown save then wrote THAT
//--- over the good seeded copy - which, because seeding only re-runs when the cache file is absent,
//--- silently poisoned every subsequent backtest. Skipping it makes the tester cache strictly
//--- read-only for single backtests: it can only ever be (re)written by the seed copy from
//--- production, never by a run's own in-memory state.
if ( m_inferenceOnly )
{
PrintVerbose ( ID + " : inference-only run - skipping the shutdown weight save (nothing was trained; the cached model is left exactly as seeded). " ) ;
return true ;
}
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in
ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp,
.stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn
chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA
after a reset but before an era completed re-created a .nnw from the
freshly-built, never-run net, so the next attach loaded an era-0 stub
instead of starting clean. For LSTM/HYBRID that stub is worse than nothing -
a layer that has never run a forward pass has m_iInputs<=0, so Save omits
every LSTM buffer (see 413ff7e). Skip the save when no era completed and no
model was loaded; that is exactly the post-reset and first-attach state.
Also sweep _shadowclone.tmp, which the reset did not cover.
Separately, ComputeLstmHiddenSize budgeted every topology against the
flattened input (historyBars x neuronsCount). True for LSTM, wrong for
HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature
map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults.
The quadratic is dominated by the inputs term, so overstating the fan-in
2.6x cost a full ladder step (16 units where the budget affords 32). New
virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this
rather than an AIType check. desc.window is advisory only - CNet never
passes it to the layer - but is now truthful for the same reason.
Derived values stay out of the weights-filename fingerprint and are adopted
from the .cfg, so existing models keep their saved width; only fresh ones
pick up the corrected budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
//--- Nothing trained and nothing loaded => there is no state to persist, and writing anyway is
//--- actively harmful. This is what defeated the panel's reset-weights button: ResetWeights
//--- correctly deletes the .nnw/.cfg/.stats/_ckpt/_shadow set and the .arrows sidecar, but if the
//--- EA is then detached before a single era completes, THIS save immediately re-created a .nnw
//--- from the freshly-built, never-run net - so the next attach loaded an era-0 stub instead of
//--- starting clean. For LSTM/HYBRID that stub is worse than useless: a layer that has never run a
//--- forward pass has m_iInputs<=0, so CNeuronLSTMOCL::Save omits every LSTM buffer (see the note
//--- in its Load()). m_eraCount==0 && !m_modelLoadedFromDisk is exactly the state ResetWeights
//--- leaves behind, and also the first-ever-attach state - both cases have nothing worth writing.
if ( m_eraCount = = 0 & & ! m_modelLoadedFromDisk )
{
PrintVerbose ( ID + " : no era completed and no model loaded - skipping the shutdown weight save (leaving the model files absent so the next attach starts genuinely clean). " ) ;
return true ;
}
2026-07-14 22:36:27 -04:00
double currentIndicatorParams [ ] ;
refactor(ExpertSignalAIBase): extract AutoTune param state into CADIndicatorTuner
CExpertSignalAIBase (4,326 lines, one class) carried 5 struct
definitions, 10 member fields, and 3 methods (Flatten/Unflatten/
PerturbRandom) purely for the AutoTuneIndicators search-space state -
entirely self-contained (never touches Net, Train()'s resumable state
machine, or anything else in the class). Extracted into a new
Expert/ADIndicatorTuner.mqh (CADIndicatorTuner), held as a single
m_indicatorTuner member.
TuneIndicatorsAndTrain() itself - the outer loop that actually
orchestrates Train()/Net/checkpointing around this tuner - turned out
to be exactly as tightly coupled to Train()'s resumable state machine
as Train() itself, so per the same caution already applied to Train()
in this refactor pass, it stays in CExpertSignalAIBase rather than
being pulled into the collaborator; it now calls the tuner's public
Flatten()/Unflatten()/PerturbRandom()/SaveAsBest()/RestoreBest()
instead of manipulating the structs inline.
All internal field-access renames (m_adCumDeltaParams.lookback ->
m_indicatorTuner.adCumDelta.lookback, etc., ~40 sites across the 5
InitAD*() indicator-setup methods) verified against a full grep sweep
- no leftover references to the old field/method names. Compiled
clean (MetaEditor, 0 errors/0 warnings).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 16:22:09 -04:00
m_indicatorTuner . Flatten ( currentIndicatorParams ) ;
2026-07-14 22:36:27 -04:00
bool ok = Net . Save ( m_activeFileName + " .nnw " , dError , dUndefine , dForecast , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , currentIndicatorParams ) ;
2026-07-23 19:36:34 -04:00
//--- calibration state (class priors + confidence scale) must travel with the weights so live
//--- trading behaves like training after a restart - see SaveModelStats().
2026-07-26 12:12:14 -04:00
if ( ! SaveModelStats ( m_activeFileName , m_activeFileCommon ) )
Print ( ID + " : ERROR - shutdown SaveModelStats failed for " + m_activeFileName + " . Calibration state not persisted. " ) ;
2026-07-25 02:01:05 -04:00
//--- Deliberately do NOT save the shadow net here. On the CPU-DLL box a second full-net write
2026-07-25 12:02:38 -04:00
//--- (~18MB) roughly DOUBLES the shutdown cost, and OnDeinit has a limited budget before MT5 reports
//--- "Abnormal termination" and skips the rest of the teardown. (An earlier revision also blamed that
//--- overrun for the "failed to allocate layer 0" reloads; that was a misdiagnosis - the real cause
//--- was a lost virtual override in the read path, see AI\Network.mqh's CLayer::CreateElement note.
//--- Halving the shutdown cost is still worth it on its own.) The shadow is NOT lost: it is saved
//--- every era (Train()), at FinalizeTrainRun(), and every ONLINE_LEARN_PERSIST_EVERY bars during
//--- online learning, and it self-heals (EnsureShadowNet re-blends from the main Net when
//--- missing/stale). Worst case a
2026-07-25 02:01:05 -04:00
//--- shutdown loses only the shadow's in-progress-era drift, which re-converges - a far better
//--- trade than risking the whole model to an over-budget shutdown.
2026-07-14 22:36:27 -04:00
if ( ! ok )
Print ( ID + " : ERROR - failed to persist weights on shutdown for " + m_activeFileName + " , error " + IntegerToString ( GetLastError ( ) ) ) ;
else
PrintVerbose ( ID + " : weights persisted on shutdown (era " + IntegerToString ( m_eraCount ) + " , trainingComplete= " + ( string ) m_trainingComplete + " ) " ) ;
2026-07-24 21:56:53 -04:00
return ok ;
}
//--- Persist the drawn arrows to disk, then remove THIS EA's chart visuals (arrows + status label).
//--- Called early in OnDeinit(), before the heavy weight save, so a later stall/fault in the save can
//--- never leave the chart littered. Idempotent - the destructor's PurgeChart() then simply no-ops.
//--- Deliberately NOT part of SaveWeightsNow(): a mid-session manual save must not wipe the chart.
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
//---
//--- 2026-08-01: there used to be a SECOND behaviour here, selected by a `preserveChartArrows` flag the
//--- caller derived from the deinit reason - on a recompile / input change / template swap / symbol
//--- change the arrows were deliberately LEFT on the chart, on the theory that an in-process reload
//--- should not flicker. It is gone, and the flag with it. Three reasons, in order of weight:
//--- 1. It is the reported defect. "The EA removes its panel and label but its signals stay" is what
//--- that branch does by design, and an operator has no way to tell a deliberate warm-reload
//--- preserve from a cleanup that failed.
//--- 2. It was only correct when the reload keeps the SAME config. Change an input that feeds the
//--- weights fingerprint - which is exactly what REASON_PARAMETERS means - and the preserved
//--- arrows belong to a model this chart is no longer running, with nothing to mark them stale.
//--- 3. The save/restore mechanism it was avoiding already handles this case, progressively and
//--- without blocking OnInit (LoadChartSignals + AdvanceChartSignalRestore). Keeping a second,
//--- subtly different route to the same outcome bought a few hundred milliseconds of flicker.
//--- One path now: persist, clear, restore on the next attach if a model for this config exists.
void ShutdownChartCleanup ( void )
2026-07-24 21:56:53 -04:00
{
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
PersistAndClearChartSignals ( ) ;
2026-07-24 21:56:53 -04:00
}
//--- Full shutdown persistence (weights + arrows), preserved for the panel's manual "save weights"
//--- button (SaveWeightsNow) - does NOT purge the chart. OnDeinit no longer calls this; it runs
//--- ShutdownChartCleanup() then PersistWeightsOnShutdown() so cleanup can't be starved by the save.
bool PersistOnShutdown ( void )
{
bool ok = PersistWeightsOnShutdown ( ) ;
//--- Persist the drawn arrows too so a re-add/recompile restores them without a retrain.
2026-07-24 11:52:19 -04:00
SaveChartSignals ( ) ;
2026-07-14 22:36:27 -04:00
return ok ;
}
//--- explicit manual save, identical persistence to PersistOnShutdown() but user-triggered from the panel
bool SaveWeightsNow ( void ) { return PersistOnShutdown ( ) ; }
//--- reloads this signal's current-config weights file from disk, discarding any unsaved in-memory
//--- training progress since the last successful save
bool LoadWeightsNow ( void )
{
if ( CheckPointer ( Net ) = = POINTER_INVALID )
return false ;
double loadedIndicatorParams [ ] ;
bool netLoaded = Net . Load ( m_activeFileName + " .nnw " , dError , dUndefine , dForecast , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , loadedIndicatorParams ) ;
if ( ! netLoaded )
{
Print ( ID + " : ERROR - failed to load weights from " + m_activeFileName + " .nnw, error " + IntegerToString ( GetLastError ( ) ) ) ;
return false ;
}
2026-07-27 11:28:23 -04:00
m_modelLoadedFromDisk = true ;
2026-07-29 12:00:40 -04:00
//--- the file may carry a superseded architecture - correct it before anything reads the net
EnforceTopologyContract ( ) ;
2026-07-23 19:36:34 -04:00
//--- restore the calibration state that pairs with these weights (priors + confidence scale) so a
//--- manual reload keeps live decisions calibrated exactly as the saved model was - see LoadModelStats().
LoadModelStats ( m_activeFileName , m_activeFileCommon ) ;
2026-07-14 22:36:27 -04:00
if ( ArraySize ( loadedIndicatorParams ) = = AD_TUNE_PARAM_COUNT )
{
2026-08-13 10:23:11 -04:00
//--- same no-change guard as the resume path - see AdoptIndicatorParams
2026-07-14 22:36:27 -04:00
if ( m_indicatorsPtr ! = NULL )
2026-08-13 10:23:11 -04:00
AdoptIndicatorParams ( loadedIndicatorParams , m_indicatorsPtr ) ;
else
m_indicatorTuner . Unflatten ( loadedIndicatorParams ) ;
2026-07-14 22:36:27 -04:00
}
//--- this just swapped dtStudied/m_eraCount/Net's weights out from under whatever a chunked
//--- Train() run (see its declaration comment) had cached for the era/trial it was mid-way
//--- through - discard that resumable state so the next Train() call starts a fresh era
//--- against the just-loaded dtStudied instead of resuming bar-loop bookkeeping computed
//--- against a now-stale one.
m_trainRunActive = false ;
m_eraResumePending = false ;
m_haveOosCheckpoint = false ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1 ; // the joint-checkpoint era stamp goes with the snapshot it describes
2026-07-14 22:36:27 -04:00
m_oosWindow . Clear ( ) ;
m_tuneTrialIndex = -1 ;
RefreshLatestSignal ( ) ;
Print ( ID + " : weights reloaded from disk (era " + IntegerToString ( m_eraCount ) + " , trainingComplete= " + ( string ) m_trainingComplete + " ) " ) ;
return true ;
}
//--- deletes this signal's current-config saved files only (weights, topology config, in-progress
//--- checkpoint) and rebuilds a fresh untrained topology in memory so training restarts from era 0.
//--- Never touches another signal type's or another symbol/timeframe's files - m_fileName already
//--- embeds symbol+period+id+output-count+opt-algo.
bool ResetWeights ( void )
{
bool stopped = m_trainingStopRequested ;
m_trainingStopRequested = true ; // hold off any in-flight Train() scheduling while we reset
//--- targets whichever file this run is actually training against (see InitNeuralNetwork): the
//--- shared production weights normally, or the local tester/optimizer cache during a backtest -
//--- so resetting from the panel during a visual-mode backtest can never wipe the live model.
int flags = m_activeFileCommon ? FILE_COMMON : 0 ;
string nnw = m_activeFileName + " .nnw " ;
string cfg = m_activeFileName + " .cfg " ;
string ckpt = m_activeFileName + " _ckpt.tmp " ;
2026-07-24 11:52:19 -04:00
//--- The calibration/online-learning sidecar (.stats: priors, confidence scale, CPU-inference marker,
//--- and the online watermark/guardrail - see SaveModelStats) and the deployed EMA shadow
//--- (_shadow.nnw - see SaveShadowNet) both pair with the weights being erased. Delete them too, or
//--- a fresh retrain would silently inherit the OLD model's calibration and blend into a stale shadow
//--- (EnsureShadowNet loads _shadow.nnw from disk before it ever clones the new Net).
string stats = m_activeFileName + " .stats " ;
string shadow = m_activeFileName + " _shadow.nnw " ;
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in
ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp,
.stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn
chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA
after a reset but before an era completed re-created a .nnw from the
freshly-built, never-run net, so the next attach loaded an era-0 stub
instead of starting clean. For LSTM/HYBRID that stub is worse than nothing -
a layer that has never run a forward pass has m_iInputs<=0, so Save omits
every LSTM buffer (see 413ff7e). Skip the save when no era completed and no
model was loaded; that is exactly the post-reset and first-attach state.
Also sweep _shadowclone.tmp, which the reset did not cover.
Separately, ComputeLstmHiddenSize budgeted every topology against the
flattened input (historyBars x neuronsCount). True for LSTM, wrong for
HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature
map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults.
The quadratic is dominated by the inputs term, so overstating the fan-in
2.6x cost a full ladder step (16 units where the budget affords 32). New
virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this
rather than an AIType check. desc.window is advisory only - CNet never
passes it to the layer - but is now truthful for the same reason.
Derived values stay out of the weights-filename fingerprint and are adopted
from the .cfg, so existing models keep their saved width; only fresh ones
pick up the corrected budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
//--- EnsureShadowNet clones the live net through this temp file (see CloneNetInto). A crash or a
//--- reset mid-clone leaves it on disk shaped for the model being erased; sweep it with the rest.
string shadowClone = m_activeFileName + " _shadowclone.tmp " ;
fix(reset): say what the reset actually did, per member and per file
The user reports "Delete & Reset Weights only wipes the first NN". I could not
find a code path that skips ensemble members, and I am not going to assert one:
the handler loops g_aiSignals[0..g_aiSignalCount), all four topologies register
unconditionally in OnInit, and SetIdentity gives each its own State\<id>\ folder
so the six deleted paths are genuinely distinct per member. What IS true is that
the whole success path was SILENT - six FileDelete calls per member printing only
on failure, and one chart-wide Alert - so a four-member reset and a one-member
reset produce byte-identical output. The symptom could be neither confirmed nor
refuted from a log. That is the defect I can fix today.
- COMPILED <timestamp> (__DATETIME__) beside the build tag. The hand-edited tag
had sat at scan-nofwd-v5 across a week of commits, so it could not answer the
question it exists for. The compile stamp cannot be forgotten. Tag bumped to
reset-census-v6.
- RegistryLine() (public): ID, active file path, common/local, era, deployed vs
training, ensemble index. The reset handler prints a numbered census of the
whole registry BEFORE the confirm dialog. If that says 1 on an AI_HYBRID chart
the fault is registration, not the reset - and RegisterAISignal already has a
loud MAX_AI_SIGNALS message for exactly that.
- The confirmation dialog now names the count, so a wrong registry is visible
before anything is deleted rather than after.
- ResetWeights prints one line per member: N deleted / N already absent / N
FAILED, plus a per-suffix breakdown. "absent" on a member that should have had
a .nnw is a completely different fault from "deleted"; they were identical.
- ResetWeights' return value was discarded. A member whose BuildFreshTopology
fails has had its files deleted and has no network - and the Alert still said
"weights reset". Counted now, with an INCOMPLETE alert when they disagree.
- Same for dbm.ResetDatabase(), whose bool was also dropped. The DB is one shared
file for every signal on the chart, so there is nothing per-member to loop -
the log now says that explicitly, since it is the question being asked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:57:26 -04:00
//--- SAY WHAT HAPPENED TO EVERY FILE. Until now this block printed only on a delete FAILURE, so the
//--- entire success path of a four-member ensemble reset was six silent operations per member and
//--- one chart-wide Alert - which is precisely why "it only wiped the first model" could not be
//--- settled from a log. Absent files are reported too: "absent" on a member that should have had a
//--- .nnw is a completely different fault from "deleted", and they were indistinguishable.
int filesDeleted = 0 , filesAbsent = 0 , filesFailed = 0 ;
string wipeReport = " " ;
string targets [ 6 ] ;
targets [ 0 ] = nnw ;
targets [ 1 ] = cfg ;
targets [ 2 ] = ckpt ;
targets [ 3 ] = stats ;
targets [ 4 ] = shadow ;
targets [ 5 ] = shadowClone ;
for ( int fi = 0 ; fi < 6 ; fi + + )
{
ResetLastError ( ) ;
string leaf = targets [ fi ] ;
//--- name the SUFFIX only in the per-file summary; the full path is printed once, below it
string shortName = StringSubstr ( targets [ fi ] , StringLen ( m_activeFileName ) ) ;
if ( ! FileIsExist ( leaf , flags ) )
{
filesAbsent + + ;
wipeReport + = StringFormat ( " %s%s=absent " , ( wipeReport = = " " ? " " : " " ) , shortName ) ;
continue ;
}
if ( FileDelete ( leaf , flags ) )
{
filesDeleted + + ;
wipeReport + = StringFormat ( " %s%s=deleted " , ( wipeReport = = " " ? " " : " " ) , shortName ) ;
}
else
{
filesFailed + + ;
wipeReport + = StringFormat ( " %s%s=FAILED(%d) " , ( wipeReport = = " " ? " " : " " ) ,
shortName , GetLastError ( ) ) ;
Print ( ID + " : ERROR - failed to delete " + leaf + " , error " + IntegerToString ( GetLastError ( ) ) ) ;
}
}
PrintFormat ( " %s: RESET WIPE of %s - %d deleted, %d already absent, %d FAILED | %s " ,
ID , m_activeFileName , filesDeleted , filesAbsent , filesFailed , wipeReport ) ;
fix: clear stale signal arrows when a fresh model starts at era 0
Arrow cleanup existed on two paths - the panel's reset-weights, and the
topology-mismatch discard - but both are gated on there being a saved .nnw to
delete. The third case had no cleanup at all: a fresh topology at era 0 with no
weights behind it, which is what a changed config produces. A new fingerprint
makes a new m_fileName, so the previous model's files are not "discarded", they
are simply not this model's files, and nothing ever cleared the chart.
That is not cosmetic. Arrows outlive the model that drew them twice over:
1. The chart objects live in the CHART, not the sidecar, so they survive a
remove/re-add, a recompile, a restart and a fresh deploy no matter what
happens to any file on disk.
2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for
SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the
dead model's calls and writes them out under the NEW model's filename -
laundering them into the new model's history where nothing can separate
them afterwards.
Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it
cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the
namespaced chart objects and logs why - and called it from all three paths.
The call sits at the BuildFreshTopology() call site, not inside it: the genetic
tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never
touch the chart. All three sites run after m_fileName has its config fingerprint
appended, so they target the right sidecar.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 10:13:01 -04:00
//--- The drawn signal arrows and their .arrows sidecar belong to the model being erased, exactly
//--- like the .stats/_shadow sidecars above - see ClearPersistedChartSignals().
ClearPersistedChartSignals ( " weights reset from the panel " ) ;
2026-07-14 22:36:27 -04:00
m_eraCount = 0 ;
m_trainingComplete = false ;
2026-07-27 11:28:23 -04:00
m_modelLoadedFromDisk = false ;
2026-07-14 22:36:27 -04:00
dtStudied = 0 ;
dError = -1 ;
dUndefine = 0 ;
dForecast = 0 ;
dPrevSignal = 0 ;
2026-07-21 00:03:45 -04:00
m_nmsLiveBuyTime = 0 ;
m_nmsLiveSellTime = 0 ;
2026-07-21 12:30:29 -04:00
m_nmsLiveBuyAccept = false ;
m_nmsLiveSellAccept = false ;
m_nmsLiveKeptTime = 0 ;
m_nmsLiveKeptDir = Neutral ;
m_nmsLiveKeptConf = 0 ;
2026-07-14 22:36:27 -04:00
dOosError = -1 ;
dOosForecast = 0 ;
m_oosSamples = 0 ;
2026-07-25 00:02:34 -04:00
//--- fresh model => wipe the compounded/persistent accuracy history too (it is only reset here; a
//--- normal restart restores it from .stats, and it survives era-to-era). See m_cumIsCorrect.
m_cumIsCorrect = 0 ;
m_cumIsTotal = 0 ;
m_cumOosCorrect = 0 ;
m_cumOosTotal = 0 ;
2026-08-16 21:08:41 -04:00
//--- ENSEMBLE: the panel's combined-vote accuracy is the same kind of lifetime counter (see
//--- g_ensCumOosCorrect's declaration comment), so a reset here must wipe it too, or the panel
//--- would keep quoting a win-rate measured partly against weights that no longer exist.
if ( m_ensembleMember )
{
g_ensCumOosCorrect = 0 ;
g_ensCumOosTotal = 0 ;
}
2026-07-14 22:36:27 -04:00
//--- discard any in-progress chunked run/tuning state - it references buffers/checkpoints from
//--- before this reset and must never be resumed into the freshly rebuilt topology below
m_trainRunActive = false ;
m_eraResumePending = false ;
m_haveOosCheckpoint = false ;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (b77e7b4): Train() runs at most one
era per call and a member that finished era N is held until every member
reaches N, so when the last member scores the vote no member's weights
have advanced past end-of-era-N. That makes the deferred simultaneous
capture a guarantee rather than a race. Each snapshot is era-STAMPED and
deploy requires every stamp to equal the winning era - otherwise a member
whose capture failed would still hold an older snapshot and the deployed
quartet would again be one nothing measured. Partial capture rolls the
era back out of "best" so the search continues instead of freezing
behind a checkpoint that does not exist.
Statistics mirror the per-member gate one for one - same coverage floor
(MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction
chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over
the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs:
the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one
member called. Two-sidedness is required of the vote itself - a vote that
never goes short IS the always-long model the chance reference prices in.
Members keep their own per-era statistics and their own learning-rate
dynamics (regression restore, eta decay); those are per-net training
mechanics, not deployment decisions. The shared ladder is mirrored onto
each member so per-era log lines report the state that actually governs
them. Solo charts are untouched on every path.
Verified: full MetaEditor compile, 0 errors 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1 ; // the joint-checkpoint era stamp goes with the snapshot it describes
2026-07-14 22:36:27 -04:00
m_oosWindow . Clear ( ) ;
m_syncWaitStartTick = 0 ;
m_tuneTrialIndex = -1 ;
//--- an explicit reset restarts from era 0 against a fresh topology - re-verify history sync and
//--- rebuild the label cache too, and drop any in-flight continual-learning OOS simulation, since
//--- both would otherwise reference bars/weights from before this reset.
m_warmupPassesRemaining = 3 ;
m_labelCacheBars = 0 ;
m_labelCacheAnchorTime = 0 ;
m_labelCachePrebuilt = false ;
m_labelPrebuildActive = false ;
m_prebuildSeedPending = false ;
if ( m_simOosRunActive )
{
delete m_simOosNet ;
m_simOosNet = NULL ;
m_simOosRunActive = false ;
}
2026-07-27 09:24:53 -04:00
//--- Re-seed before building a fresh topology so weight init is genuinely random, not dominated
//--- by whatever fixed/deterministic seed the genetic tuner's last candidate evaluation left in
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- the MQL5 RNG state. Matches the tuner's own post-tune rebuild path
2026-07-27 09:24:53 -04:00
//--- (line ~5071) and Warrior_EA.mq5's OnInit.
MathSrand ( GetTickCount ( ) ) ;
2026-07-14 22:36:27 -04:00
bool rebuilt = BuildFreshTopology ( ) ;
if ( ! rebuilt )
Print ( ID + " : ERROR - failed to rebuild fresh topology after weights reset " ) ;
else
{
2026-07-24 21:36:02 -04:00
//--- Stamp the config fingerprint with isInitialized=FALSE, NOT m_isInitialized. The compare in
//--- InitNeuralNetwork (and its own fresh-start .cfg write) always run before m_isInitialized is
//--- set true at the end of init, so they always use false. ResetWeights is invoked from the panel
//--- AFTER init, where m_isInitialized is true - passing it here would make this the ONLY .cfg on
//--- disk with isInitialized=true, so the very next attach spuriously fails the compare
//--- ("Configuration mismatch. Deleting file") and needlessly discards the weights we just reset.
//--- This flag is runtime lifecycle state, not a topology/input parameter, so it must never gate reuse.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
SaveTopologyConfiguration ( m_activeFileName , m_initialNeuronsCount , m_hiddenLayersCount , m_neuronsReduction , m_minNeuronsCount , m_optimizationAlgo , m_historyBars , m_outputNeuronsCount , m_neuronsCount , LEGACY_STUDY_PERIOD_SLOT , m_minTrainYear , false , LEGACY_CONVERGE_WR_SLOT , m_fractalPeriods , m_convFilterCount , m_lstmHiddenSize , m_activeFileCommon ) ;
2026-07-14 22:36:27 -04:00
Print ( ID + " : weights reset - training will restart from era 0 (current config only: " + m_activeFileName + " ) " ) ;
}
m_trainingStopRequested = stopped ;
if ( ! stopped & & ! bEventStudy )
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArmStudyEvent ( 0 , " Reset " ) ;
2026-07-14 22:36:27 -04:00
return rebuilt ;
}
} ;
//+------------------------------------------------------------------+
//+------------------------------------------------------------------+
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| IMPLEMENTATION |
2026-07-14 22:36:27 -04:00
//| |
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| CExpertSignalAIBase's method bodies live in these partial files. |
//| They MUST be included here, after the class declaration above, |
//| and nowhere else. Order between them does not matter - they are |
//| all out-of-class definitions of an already-declared class. |
2026-07-14 22:36:27 -04:00
//| |
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| The class was ~8200 lines in one file; this splits the bodies by |
//| responsibility so a change to, say, chart drawing no longer means |
//| scrolling past the era loop. Nothing was rewritten in the move. |
//+------------------------------------------------------------------+
# include "AIBase\Training.mqh"
2026-08-01 11:27:28 -04:00
# include "AIBase\Lifecycle.mqh"
# include "AIBase\Topology.mqh"
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
# include "AIBase\Labels.mqh"
# include "AIBase\OnlineLearning.mqh"
# include "AIBase\AutoTune.mqh"
# include "AIBase\Inference.mqh"
# include "AIBase\Persistence.mqh"
# include "AIBase\ChartUI.mqh"
# include "AIBase\Features.mqh"
feat: excursion-size head (Stage 1, measurement only)
Direction is closed - normalised asymmetry fails on three instruments
with a working positive control, and the classifier's own best-of-999
era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is
a different question and RANGE clears at ~4x its null.
Checked the denomination before building on that, since the source memo
warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is
predictable" is a claim about travel RELATIVE to current ATR, not a
restatement of "ATR is autocorrelated". It is exactly the part a fixed
multiple (stop 3.31*ATR, target 1.64*ATR) discards.
A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches
ladder rung k) upward and downward. Survival parameterisation rather than
regressing the multiple, because it needs nothing new from CNet: sigmoid
outputs and the per-neuron delta the `total != 3` branch already applies
(a quantile head would need a linear activation and a pinball gradient in
Network.mqh, Network.cl and the DirectML path, on a class four topologies
share). Targets are free - m_ladderUpAt already records first-touch age
per rung with 0 meaning never reached.
Separate net, not extra outputs on the classifier: more outputs would
change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push
the count off 3 - the exact condition backProp uses to select the joint
softmax gradient the 3-class head depends on. The classifier is
bit-for-bit unaffected and this is removable without trace.
STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the
constant per-rung base rate - the baseline a fixed ATR multiple already
assumes - with both predictors fitted IS and evaluated OOS, so neither
gets a look at the test set. Positive skill justifies Stage 2 (drive
SL/TP and sizing off ExcursionQuantile, which is defined and deliberately
uncalled). Zero or negative means ATR already carries everything and
Stage 2 must not be built.
Trains only on primary occurrences: the replay queue oversamples for
CLASS balance, and a direction-balanced sample is a biased SIZE sample.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
# include "AIBase\Excursion.mqh"
feat(gate): cross-instrument pooled certification
The deploy bottleneck is CERTIFICATION, not training. A 4,738-bar OOS window at
L=75.6 holds ~63 independent observations; certifying a 3pp edge at 2 sigma needs
~1,036. More bars of the same symbol barely help - they overlap. Other symbols do
not.
WHAT POOLS. Not win rates: symbols have different derived geometries, different
break-evens and different drifts, so averaging raw rates across them is
meaningless. What pools is each symbol's EXCESS OVER ITS OWN CHANCE RATE,
combined by inverse-variance weighting (fixed-effects meta-analysis). Each symbol
keeps its own model, geometry and chance rate; only the evidence is combined.
THE CORRELATION PROBLEM, bracketed rather than assumed away. SP500 and NAS100 are
~0.9 correlated and pooling them as independent inflates the evidence. Nothing
here can measure that without sharing return series, so instead of guessing a
correction the gate reports both ends:
SE_INDEP = sqrt(1/SUM(1/var_i)) all members independent
SE_CORR = SUM(w_i * sqrt(var_i)) all members perfectly correlated
The truth is always between. THE GATE USES SE_CORR, so a pass cannot be an
artifact of correlated instruments - that bound already assumes the worst. The
ratio is logged as the diversification credit the gate declines to claim, so the
cost of that conservatism is visible instead of hidden.
SCOPE, deliberately limited: the pooled result is REPORTED, never folded into
tradeableOK. The local gate certifies the model that actually trades this symbol;
the pool answers the different question of whether the strategy has an edge at
all. Letting a cross-symbol result license a local deploy would ship a model that
never cleared its own bar - so it cannot.
Mechanics: one file per instrument (no concurrent-write path to get wrong), every
FileOpen carrying FILE_SHARE_READ|FILE_SHARE_WRITE, records skipped rather than
reinterpreted on a version mismatch, 12h staleness cutoff so a stopped chart
cannot vote, and pooling refused below 3 instruments. Poolability requires
matching timeframe and ratio; differing SYMBOL is the entire point. Publishing is
unconditional - a pool that only hears from winners is a selection effect, not a
meta-analysis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:45:37 -04:00
# include "AIBase\PooledGate.mqh"
2026-07-16 00:56:33 -04:00
//+------------------------------------------------------------------+