refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| Warrior_EA |
|
|
|
|
|
//| AnimateDread |
|
|
|
|
|
//| |
|
|
|
|
|
//| Chart arrows, arrow persistence/restore, status panel, chart cle|
|
|
|
|
|
//| |
|
|
|
|
|
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
|
|
|
|
|
//| This holds CExpertSignalAIBase method BODIES only. The class |
|
|
|
|
|
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
|
|
|
|
|
//| #includes this file at the bottom, after the declaration. Do not |
|
|
|
|
|
//| include it anywhere else and do not compile it on its own. |
|
|
|
|
|
//| |
|
|
|
|
|
//| Split out purely to make the 8216-line original navigable; the |
|
|
|
|
|
//| code inside was moved verbatim, not rewritten. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
#ifndef WARRIOR_AIBASE_CHARTUI_MQH
|
|
|
|
|
#define WARRIOR_AIBASE_CHARTUI_MQH
|
|
|
|
|
//+------------------------------------------------------------------+
|
fix: clear stale signal arrows when a fresh model starts at era 0
Arrow cleanup existed on two paths - the panel's reset-weights, and the
topology-mismatch discard - but both are gated on there being a saved .nnw to
delete. The third case had no cleanup at all: a fresh topology at era 0 with no
weights behind it, which is what a changed config produces. A new fingerprint
makes a new m_fileName, so the previous model's files are not "discarded", they
are simply not this model's files, and nothing ever cleared the chart.
That is not cosmetic. Arrows outlive the model that drew them twice over:
1. The chart objects live in the CHART, not the sidecar, so they survive a
remove/re-add, a recompile, a restart and a fresh deploy no matter what
happens to any file on disk.
2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for
SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the
dead model's calls and writes them out under the NEW model's filename -
laundering them into the new model's history where nothing can separate
them afterwards.
Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it
cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the
namespaced chart objects and logs why - and called it from all three paths.
The call sits at the BuildFreshTopology() call site, not inside it: the genetic
tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never
touch the chart. All three sites run after m_fileName has its config fingerprint
appended, so they target the right sidecar.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 10:13:01 -04:00
|
|
|
//| Erase this model's drawn arrows, its .arrows sidecar, and any |
|
|
|
|
|
//| deferred restore still in flight. |
|
|
|
|
|
//| |
|
|
|
|
|
//| Must run on EVERY path that discards or replaces the trained |
|
|
|
|
|
//| weights, because arrows outlive the model that drew them in two |
|
|
|
|
|
//| separate ways: |
|
|
|
|
|
//| |
|
|
|
|
|
//| 1. The chart objects themselves are saved in the CHART, not in |
|
|
|
|
|
//| the sidecar, so they survive a remove/re-add, a recompile, a |
|
|
|
|
|
//| terminal restart and a fresh deploy regardless of what |
|
|
|
|
|
//| happens to any file on disk. |
|
|
|
|
|
//| 2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart |
|
|
|
|
|
//| for SIG_ARROW_PREFIX objects. So a stale arrow left on screen |
|
|
|
|
|
//| is not merely cosmetic - the next save writes it back out |
|
|
|
|
|
//| under the CURRENT model's filename, laundering a dead model's |
|
|
|
|
|
//| calls into the new model's history where nothing can tell |
|
|
|
|
|
//| them apart afterwards. |
|
|
|
|
|
//| |
|
|
|
|
|
//| Keyed on m_fileName + FILE_COMMON, NOT m_activeFileName/flags: |
|
|
|
|
|
//| SaveChartSignals always writes the shared, chart-owned file and |
|
|
|
|
|
//| refuses to run in a backtest, so the same tester guard is applied |
|
|
|
|
|
//| here rather than the m_activeFileCommon indirection the weight |
|
|
|
|
|
//| files use. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::ClearPersistedChartSignals(const string reason)
|
|
|
|
|
{
|
|
|
|
|
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
|
|
|
|
|
return;
|
|
|
|
|
//--- Cancel any deferred restore FIRST. Otherwise its already-parsed queue keeps drawing the old
|
|
|
|
|
//--- model's arrows from the next timer slice onward - after the file was deleted - and the next
|
|
|
|
|
//--- save writes them straight back out. See AdvanceChartSignalRestore.
|
|
|
|
|
m_arrowRestorePending = false;
|
|
|
|
|
m_arrowRestoreIndex = 0;
|
|
|
|
|
ArrayFree(m_arrowRestoreTime);
|
|
|
|
|
ArrayFree(m_arrowRestoreCode);
|
|
|
|
|
ArrayFree(m_arrowRestorePrice);
|
|
|
|
|
ArrayFree(m_arrowRestoreTf);
|
|
|
|
|
string arrows = m_fileName + ".arrows";
|
|
|
|
|
ResetLastError();
|
|
|
|
|
if(FileIsExist(arrows, FILE_COMMON) && !FileDelete(arrows, FILE_COMMON))
|
|
|
|
|
Print(ID + ": ERROR - failed to delete " + arrows + ", error " + IntegerToString(GetLastError()));
|
|
|
|
|
ResetLastError();
|
|
|
|
|
//--- Clear them off the chart too, so the reset is visibly complete instead of leaving stale arrows
|
|
|
|
|
//--- on screen until the next restart. Namespaced delete - user drawings are untouched.
|
|
|
|
|
ObjectsDeleteAll(0, SIG_ARROW_PREFIX);
|
|
|
|
|
ChartRedraw(0);
|
|
|
|
|
PrintVerbose(ID + ": cleared drawn signal arrows and " + arrows + " - " + reason);
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//| Persist the currently-drawn directional arrows to a sidecar file |
|
|
|
|
|
//| so they survive an EA remove/re-add/recompile without a retrain. |
|
|
|
|
|
//| Captures time, arrow code (217 Buy / 218 Sell), price, and hide |
|
|
|
|
|
//| state; chart-only (no persistent chart in a backtest). Skips |
|
|
|
|
|
//| writing when there is nothing to save so it never clobbers a good |
|
|
|
|
|
//| file with an empty one (e.g. a transient no-arrow state). |
|
fix(deinit): O(n^2) arrow prune blew the shutdown budget and littered 3 charts
Reported as "the perceptron correctly cleaned its chart on deinit, the
other 3 did not, abnormal termination". Measured from the 2026-08-01 log,
time from "OnDeinit: shutting down" to MetaTrader force-terminating:
PAI 3.75 s -> survived, chart cleaned
CONV 4.71 s -> Abnormal termination
LSTM 4.28 s -> Abnormal termination
HYBRID 4.16 s -> Abnormal termination
In all four the last line printed is the inference census, which is the
end of StopTraining() - so the overrun is inside ShutdownChartCleanup(),
i.e. between saving the arrows and purging them.
The cost is the prune loop at the end of SaveChartSignals():
for(int i = 0; i < prunedCount; i++)
ObjectDelete(0, SIG_ARROW_PREFIX + TimeToString(pruned[i]));
ObjectDelete is O(objects) on a crowded chart, so this is O(n^2). It was
harmless while the model called a direction on ~6% of bars. After the
triple-barrier relabel the models call on 83-94% of bars, the chart
carries many thousands of arrows, and the loop overran MetaTrader's
OnDeinit budget - so PurgeChart() never ran and the arrows stayed on
screen. The slow tidy-up starved the fast one.
The work was pure waste at that moment: ShutdownChartCleanup purges every
arrow with a single bulk ObjectsDeleteAll immediately afterwards.
Deleting them one at a time first has no effect except to prevent the
bulk delete from happening at all.
SaveChartSignals takes a pruneChartObjects flag, and the two shutdown
call sites pass false:
- ShutdownChartCleanup passes `preserveChartArrows`, which is exactly
right: prune when the arrows are STAYING (chart and sidecar must
agree), skip when they are about to be purged wholesale.
- FinalizeTrainRun passes !m_trainingStopRequested. Removing a chart
MID-ERA reaches StopTraining -> FinalizeTrainRun, which took the
expensive path a second time, even earlier, before anything had been
cleared. Same defect one call site up; it only escaped notice because
the observed removals happened to land between eras.
Normal convergence and the live per-era path are unchanged - they still
prune, which is what keeps the chart object count bounded.
This also restores the invariant the 2026-07 fix intended ("chart cleanup
runs BEFORE the heavy weight save so a stall cannot leave the chart
littered"). That fix moved cleanup ahead of the WEIGHT save, but cleanup
had since grown its own slow step ahead of its own fast one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:38:36 -04:00
|
|
|
//| |
|
|
|
|
|
//| pruneChartObjects=false says "the caller is about to bulk-delete |
|
|
|
|
|
//| every arrow anyway" - see the prune block at the end for why that |
|
|
|
|
|
//| distinction is what keeps OnDeinit inside its budget. |
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
//| |
|
|
|
|
|
//| RETURNS true when the sidecar provably holds the current arrow set |
|
|
|
|
|
//| - including the "chart has no arrows" case, where there is nothing |
|
|
|
|
|
//| to lose. Only a genuine write failure returns false. The shutdown |
|
|
|
|
|
//| path (PersistAndClearChartSignals) keys the chart purge on this: |
|
|
|
|
|
//| arrows may only be removed from the chart once they exist on disk, |
|
|
|
|
|
//| because for a CONVERGED model the chart objects are the ONLY copy |
|
|
|
|
|
//| (nothing redraws them - PruneDirectionalClusters runs per training |
|
|
|
|
|
//| era, and a deployed model has no eras left). |
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//+------------------------------------------------------------------+
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
bool CExpertSignalAIBase::SaveChartSignals(bool pruneChartObjects = true)
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
{
|
|
|
|
|
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
return true;
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
long times[];
|
|
|
|
|
int codes[];
|
|
|
|
|
double prices[];
|
|
|
|
|
long tfs[];
|
|
|
|
|
int n = 0;
|
|
|
|
|
int total = ObjectsTotal(0);
|
|
|
|
|
//--- Pre-size the arrays ONCE to an upper bound (all chart objects), then fill and trim. The old
|
|
|
|
|
//--- ArrayResize(n+1)-per-arrow reallocated the buffers on every match -> O(n^2), which on a chart
|
|
|
|
|
//--- with thousands of accumulated arrows took long enough to blow MT5's OnDeinit time budget and
|
|
|
|
|
//--- force-terminate shutdown before the chart got cleaned. This keeps the whole scan O(n).
|
|
|
|
|
//--- room for the chart's own arrows PLUS any still sitting in the deferred-restore queue (see below)
|
|
|
|
|
int queueRemaining = m_arrowRestorePending ? (ArraySize(m_arrowRestoreTime) - m_arrowRestoreIndex) : 0;
|
|
|
|
|
if(queueRemaining < 0)
|
|
|
|
|
queueRemaining = 0;
|
|
|
|
|
ArrayResize(times, total + queueRemaining);
|
|
|
|
|
ArrayResize(codes, total + queueRemaining);
|
|
|
|
|
ArrayResize(prices, total + queueRemaining);
|
|
|
|
|
ArrayResize(tfs, total + queueRemaining);
|
|
|
|
|
for(int i = 0; i < total; i++)
|
|
|
|
|
{
|
|
|
|
|
string nm = ObjectName(0, i);
|
|
|
|
|
if(StringFind(nm, SIG_ARROW_PREFIX) != 0)
|
|
|
|
|
continue;
|
|
|
|
|
if(ObjectGetInteger(0, nm, OBJPROP_TYPE) != OBJ_ARROW)
|
|
|
|
|
continue;
|
|
|
|
|
times[n] = (long)ObjectGetInteger(0, nm, OBJPROP_TIME);
|
|
|
|
|
codes[n] = (int)ObjectGetInteger(0, nm, OBJPROP_ARROWCODE);
|
|
|
|
|
prices[n] = ObjectGetDouble(0, nm, OBJPROP_PRICE);
|
|
|
|
|
tfs[n] = g_signalsVisible ? OBJ_ALL_PERIODS : OBJ_NO_PERIODS;
|
|
|
|
|
n++;
|
|
|
|
|
}
|
|
|
|
|
//--- CRITICAL for init/deinit sync: a deferred restore may still be in flight (see
|
|
|
|
|
//--- AdvanceChartSignalRestore), meaning part of the saved history has NOT been drawn yet and so is
|
|
|
|
|
//--- invisible to the chart scan above. Saving at that moment - which OnDeinit does, via
|
|
|
|
|
//--- ShutdownChartCleanup - would persist only the drawn subset and silently truncate the user's
|
|
|
|
|
//--- history. Fold the undrawn remainder of the queue back in so the file always represents
|
|
|
|
|
//--- chart-plus-queue, whatever point the restore reached. These entries are by construction not yet on
|
|
|
|
|
//--- the chart, so no de-duplication is needed (and a same-bar duplicate would merely be overwritten on
|
|
|
|
|
//--- the next load, since the object name is derived from the bar time).
|
|
|
|
|
for(int q = m_arrowRestoreIndex; q < m_arrowRestoreIndex + queueRemaining; q++)
|
|
|
|
|
{
|
|
|
|
|
times[n] = (long)m_arrowRestoreTime[q];
|
|
|
|
|
codes[n] = m_arrowRestoreCode[q];
|
|
|
|
|
prices[n] = m_arrowRestorePrice[q];
|
|
|
|
|
tfs[n] = g_signalsVisible ? OBJ_ALL_PERIODS : OBJ_NO_PERIODS;
|
|
|
|
|
n++;
|
|
|
|
|
}
|
|
|
|
|
if(n <= 0)
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
return true; // nothing to save - leave any existing file intact (and nothing on the chart to lose)
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//--- Keep only the MAX_PERSISTED_ARROWS most recent, by TIME (see the define's comment for why scan
|
|
|
|
|
//--- order can't be used). Selection is threshold-based rather than a full parallel-array sort: sort a
|
|
|
|
|
//--- COPY of the times, read off the cut-off, then compact the four arrays in one pass. Compaction is
|
|
|
|
|
//--- safe in place because the write index never runs ahead of the read index.
|
|
|
|
|
long pruned[];
|
|
|
|
|
int prunedCount = 0;
|
|
|
|
|
if(n > MAX_PERSISTED_ARROWS)
|
|
|
|
|
{
|
|
|
|
|
long sortedTimes[];
|
|
|
|
|
ArrayResize(sortedTimes, n);
|
|
|
|
|
ArrayCopy(sortedTimes, times, 0, 0, n);
|
|
|
|
|
ArraySort(sortedTimes); // ascending
|
|
|
|
|
long cutoff = sortedTimes[n - MAX_PERSISTED_ARROWS];
|
|
|
|
|
ArrayResize(pruned, n);
|
|
|
|
|
int w = 0;
|
|
|
|
|
for(int i = 0; i < n; i++)
|
|
|
|
|
{
|
|
|
|
|
//--- ">= cutoff" can match more than the cap when several arrows share the cut-off timestamp;
|
|
|
|
|
//--- the w < cap test keeps the kept set at exactly the cap in that case.
|
|
|
|
|
if(times[i] >= cutoff && w < MAX_PERSISTED_ARROWS)
|
|
|
|
|
{
|
|
|
|
|
times[w] = times[i];
|
|
|
|
|
codes[w] = codes[i];
|
|
|
|
|
prices[w] = prices[i];
|
|
|
|
|
tfs[w] = tfs[i];
|
|
|
|
|
w++;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
pruned[prunedCount++] = times[i];
|
|
|
|
|
}
|
|
|
|
|
n = w;
|
|
|
|
|
}
|
|
|
|
|
//--- Staged through a temp file + atomic rename (System\AtomicFile.mqh), so an interrupted write
|
|
|
|
|
//--- keeps the previous arrow set instead of truncating it, and never blocks LoadChartSignals()
|
|
|
|
|
//--- on another instance.
|
|
|
|
|
string arrowsName = m_fileName + ".arrows";
|
|
|
|
|
string arrowsTmpName = "";
|
|
|
|
|
int handle = AtomicWriteBegin(arrowsName, FILE_COMMON, arrowsTmpName);
|
|
|
|
|
if(handle == INVALID_HANDLE)
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
{
|
|
|
|
|
//--- Was a silent `return`. Silence here is what made the "arrows stay on the chart" report
|
|
|
|
|
//--- unresolvable for three sessions: every failure mode of this function looked identical to
|
|
|
|
|
//--- success from the outside, and the only externally visible symptom (a littered chart) is two
|
|
|
|
|
//--- calls downstream. Name the file and the error so the next occurrence is one grep.
|
|
|
|
|
Print(ID + ": ERROR - could not open " + arrowsTmpName + " to persist " + IntegerToString(n) +
|
|
|
|
|
" chart signal arrows, error " + IntegerToString(GetLastError()) +
|
|
|
|
|
" - the arrows stay on the chart (they are the only copy).");
|
|
|
|
|
return false;
|
|
|
|
|
}
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//--- Results ARE checked now: the pruning step below is documented as running only after a
|
|
|
|
|
//--- successful write, but nothing used to verify that, so a failed/partial write still deleted
|
|
|
|
|
//--- the chart objects and lost those arrows in both places.
|
|
|
|
|
bool ok = (FileWriteInteger(handle, 0x57534152) > 0); // 'WSAR' magic
|
|
|
|
|
if(ok && FileWriteInteger(handle, n) <= 0)
|
|
|
|
|
ok = false;
|
|
|
|
|
for(int i = 0; ok && i < n; i++)
|
|
|
|
|
{
|
|
|
|
|
if(FileWriteLong(handle, times[i]) <= 0 || FileWriteInteger(handle, codes[i]) <= 0 ||
|
|
|
|
|
FileWriteDouble(handle, prices[i]) <= 0 || FileWriteLong(handle, tfs[i]) <= 0)
|
|
|
|
|
ok = false;
|
|
|
|
|
}
|
|
|
|
|
if(!AtomicWriteEnd(handle, arrowsName, arrowsTmpName, FILE_COMMON, ok, __FUNCTION__))
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
return false; // AtomicWriteEnd already logged which half failed
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//--- Only after the file is safely written: drop the pruned arrows from the chart too, so the chart and
|
|
|
|
|
//--- the sidecar stay in agreement and the object count stops growing without bound. Done last so a
|
|
|
|
|
//--- failed write above leaves both the file AND the chart untouched.
|
fix(deinit): O(n^2) arrow prune blew the shutdown budget and littered 3 charts
Reported as "the perceptron correctly cleaned its chart on deinit, the
other 3 did not, abnormal termination". Measured from the 2026-08-01 log,
time from "OnDeinit: shutting down" to MetaTrader force-terminating:
PAI 3.75 s -> survived, chart cleaned
CONV 4.71 s -> Abnormal termination
LSTM 4.28 s -> Abnormal termination
HYBRID 4.16 s -> Abnormal termination
In all four the last line printed is the inference census, which is the
end of StopTraining() - so the overrun is inside ShutdownChartCleanup(),
i.e. between saving the arrows and purging them.
The cost is the prune loop at the end of SaveChartSignals():
for(int i = 0; i < prunedCount; i++)
ObjectDelete(0, SIG_ARROW_PREFIX + TimeToString(pruned[i]));
ObjectDelete is O(objects) on a crowded chart, so this is O(n^2). It was
harmless while the model called a direction on ~6% of bars. After the
triple-barrier relabel the models call on 83-94% of bars, the chart
carries many thousands of arrows, and the loop overran MetaTrader's
OnDeinit budget - so PurgeChart() never ran and the arrows stayed on
screen. The slow tidy-up starved the fast one.
The work was pure waste at that moment: ShutdownChartCleanup purges every
arrow with a single bulk ObjectsDeleteAll immediately afterwards.
Deleting them one at a time first has no effect except to prevent the
bulk delete from happening at all.
SaveChartSignals takes a pruneChartObjects flag, and the two shutdown
call sites pass false:
- ShutdownChartCleanup passes `preserveChartArrows`, which is exactly
right: prune when the arrows are STAYING (chart and sidecar must
agree), skip when they are about to be purged wholesale.
- FinalizeTrainRun passes !m_trainingStopRequested. Removing a chart
MID-ERA reaches StopTraining -> FinalizeTrainRun, which took the
expensive path a second time, even earlier, before anything had been
cleared. Same defect one call site up; it only escaped notice because
the observed removals happened to land between eras.
Normal convergence and the live per-era path are unchanged - they still
prune, which is what keeps the chart object count bounded.
This also restores the invariant the 2026-07 fix intended ("chart cleanup
runs BEFORE the heavy weight save so a stall cannot leave the chart
littered"). That fix moved cleanup ahead of the WEIGHT save, but cleanup
had since grown its own slow step ahead of its own fast one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:38:36 -04:00
|
|
|
//---
|
|
|
|
|
//--- SKIPPED ON SHUTDOWN (pruneChartObjects == false), and that is not an optimisation - it is the fix
|
|
|
|
|
//--- for a reproducible "Abnormal termination" that left three of four charts littered on 2026-08-01.
|
|
|
|
|
//--- This loop is O(prunedCount) ObjectDelete calls, and ObjectDelete is itself O(objects) on a
|
|
|
|
|
//--- crowded chart, so it degrades to O(n^2). That was tolerable while the model called a direction on
|
|
|
|
|
//--- ~6% of bars; after the triple-barrier relabel the models call on 83-94% of bars, the chart carries
|
|
|
|
|
//--- many thousands of arrows, and the loop blew MetaTrader's OnDeinit budget - measured at 4.2-4.7 s
|
|
|
|
|
//--- against a PAI chart that survived in 3.75 s. MetaTrader then force-terminated OnDeinit BEFORE
|
|
|
|
|
//--- PurgeChart() ran, so the arrows stayed on screen: the slow tidy-up starved the fast one.
|
|
|
|
|
//--- At shutdown the caller purges every arrow with a single bulk ObjectsDeleteAll immediately after
|
|
|
|
|
//--- this returns, so deleting them one at a time first is work whose only effect is to prevent the
|
|
|
|
|
//--- bulk delete from happening at all.
|
|
|
|
|
if(prunedCount > 0 && pruneChartObjects)
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
{
|
|
|
|
|
for(int i = 0; i < prunedCount; i++)
|
|
|
|
|
ObjectDelete(0, SIG_ARROW_PREFIX + TimeToString((datetime)pruned[i]));
|
|
|
|
|
ChartRedraw(0);
|
|
|
|
|
PrintVerbose(ID + ": pruned " + IntegerToString(prunedCount) + " old chart signal arrows (keeping the " +
|
|
|
|
|
IntegerToString(MAX_PERSISTED_ARROWS) + " most recent)");
|
|
|
|
|
}
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
m_lastArrowsSaved = n;
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| Shutdown path: hand the drawn arrows over to the save/restore |
|
|
|
|
|
//| mechanism and take them OFF the chart, rather than abandoning |
|
|
|
|
|
//| them there for the next EA (or the user) to find. |
|
|
|
|
|
//| |
|
|
|
|
|
//| Reported 2026-08-01: the panel and status label disappeared on |
|
|
|
|
|
//| removal but the signal arrows stayed. Leaving them is wrong in |
|
|
|
|
|
//| both directions - a removed EA that keeps drawing on the chart |
|
|
|
|
|
//| looks broken, and the leftovers are not inert: SaveChartSignals() |
|
|
|
|
|
//| rebuilds the sidecar by SCANNING the chart, so the next model to |
|
|
|
|
|
//| attach adopts the dead one's calls as its own history (see |
|
|
|
|
|
//| ClearPersistedChartSignals' header). |
|
|
|
|
|
//| |
|
|
|
|
|
//| ORDER MATTERS, and it is write-then-clear, never the reverse. For |
|
|
|
|
|
//| a converged model the chart objects are the only copy of its |
|
|
|
|
|
//| signal history: nothing redraws them, because the renderer |
|
|
|
|
|
//| (PruneDirectionalClusters) runs once per TRAINING era and a |
|
|
|
|
|
//| deployed model has none left. So the purge is conditional on the |
|
|
|
|
|
//| sidecar write having actually succeeded - a chart that stays |
|
|
|
|
|
//| littered because the disk write failed is a far better outcome |
|
|
|
|
|
//| than one that is clean because the history was destroyed, and the |
|
|
|
|
|
//| failure now says so in the log instead of looking like this bug. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::PersistAndClearChartSignals(void)
|
|
|
|
|
{
|
|
|
|
|
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
|
|
|
|
|
{
|
|
|
|
|
PurgeChart(); // tester charts are throwaway - nothing to persist, just leave nothing behind
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
m_lastArrowsSaved = 0;
|
|
|
|
|
//--- pruneChartObjects=false: the purge below removes every arrow in one bulk call, so deleting the
|
|
|
|
|
//--- over-cap ones individually first is pure cost - and at O(objects) per ObjectDelete it is the cost
|
|
|
|
|
//--- that overran MT5's deinit budget on 2026-08-01 and skipped the cleanup entirely.
|
|
|
|
|
bool saved = SaveChartSignals(false);
|
|
|
|
|
if(!saved)
|
|
|
|
|
{
|
|
|
|
|
Print(ID + ": WARNING - could not persist the drawn signal arrows, so they are being LEFT on the chart "
|
|
|
|
|
"rather than discarded (they are the only copy). Clear them with the panel's reset-weights, or "
|
|
|
|
|
"fix the write error logged above.");
|
|
|
|
|
ClearStatusLabel();
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
int removed = PurgeChart();
|
|
|
|
|
Print(ID + ": chart signals - persisted " + IntegerToString(m_lastArrowsSaved) + " arrow(s) to " +
|
|
|
|
|
m_fileName + ".arrows and cleared " + IntegerToString(removed) +
|
|
|
|
|
" from the chart; they are restored automatically the next time this configuration is attached.");
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| Recreate the arrows saved by SaveChartSignals(). Runs at init so |
|
|
|
|
|
//| a re-added / recompiled / restarted chart shows the deployed |
|
|
|
|
|
//| model's signals again without retraining. Restores the saved hide |
|
|
|
|
|
//| state, so the show/hide toggle survives too. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::LoadChartSignals(void)
|
|
|
|
|
{
|
|
|
|
|
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
|
|
|
|
|
return;
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
//--- ARROWS BELONG TO A MODEL. No weights loaded for this configuration means this chart is starting from
|
|
|
|
|
//--- era 0, so it must start visually clean: a fresh run that inherits a previous model's arrows shows
|
|
|
|
|
//--- calls it never made, and the first save would then write them back out as its OWN history (this
|
|
|
|
|
//--- function's sidecar is rebuilt by SCANNING the chart - see SaveChartSignals). The fresh-topology
|
|
|
|
|
//--- branch in InitNeuralNetwork already clears them for the ordinary case; this is the same rule stated
|
|
|
|
|
//--- once, at the single point where arrows are brought back, so it holds no matter which of that
|
|
|
|
|
//--- function's several early exits actually ran.
|
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach
Chart objects live in the MT5 chart PROFILE, not in this EA's files.
They survive a terminal restart, a recompile, and deleting every
.nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION
removes them - and MetaTrader force-terminates OnDeinit at roughly
4,500 ms, so a run killed mid-cleanup orphans them permanently with no
owner left to clean up after. That is the "deleted every file,
recompiled, restarted, old arrows and a stale panel still there"
report: nothing was wrong with the files and deleting them could not
have helped.
Both halves are fixed.
STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight
run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state
re-seed) and then write two full nets per chart. On four charts that is
the bulk of the budget, spent to preserve a PARTIAL era that was never
scored, never checkpointed and never deployable. FlushTrainRun()
discards it instead - drop the resumable bookkeeping, leave the net
neutral (unfreeze BN, flush the batch, batch size 1), skip the save -
and training resumes from the last completed era, which the era-end
save and the periodic autosave have already put on disk. What is
discarded is bounded by one era.
A CONVERGED model keeps the old finalise-and-save path: its weights can
carry online-learning updates made since the last era boundary, and for
a deployed model no further era boundary is coming to persist them.
MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model
loaded, sidecar missing - so the common paths returned leaving whatever
the previous instance stranded. LoadChartSignals now sweeps the arrow
namespace unconditionally before restoring, so the post-init chart
holds exactly what the sidecar holds whichever branch runs, and the
panel gets the same treatment before Create() (CAppDialog namespaces
its controls, so a killed Destroy strands the lot and the next attach
draws a second panel on the corpse).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
|
|
|
//--- ORPHAN SWEEP, UNCONDITIONAL AND FIRST. Chart objects live in the MT5 chart PROFILE, not in this
|
|
|
|
|
//--- EA's files: they survive a terminal restart, a recompile, and deleting every .nnw/.cfg/.stats/
|
|
|
|
|
//--- .arrows on disk. The only thing that removes them is a deinit that runs to completion - and
|
|
|
|
|
//--- MetaTrader force-terminates OnDeinit at roughly 4,500 ms, so a run killed mid-cleanup orphans them
|
|
|
|
|
//--- PERMANENTLY, with no owner left to clean up after.
|
|
|
|
|
//--- Every purge here used to sit behind a branch (no model loaded / sidecar missing), so the common
|
|
|
|
|
//--- paths returned with whatever the previous instance had stranded still on screen. That is the
|
|
|
|
|
//--- "I deleted every file, recompiled, restarted, and the old arrows are still there" report: nothing
|
|
|
|
|
//--- was wrong with the files, and no amount of deleting them could have helped.
|
|
|
|
|
//--- Sweeping unconditionally before the restore makes the post-init chart deterministic - it holds
|
|
|
|
|
//--- exactly what the sidecar holds, and nothing else, whichever branch below runs.
|
|
|
|
|
ObjectsDeleteAll(0, SIG_ARROW_PREFIX);
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
if(!m_modelLoadedFromDisk)
|
|
|
|
|
{
|
|
|
|
|
ClearPersistedChartSignals("no saved model for this configuration - starting with a clean chart");
|
|
|
|
|
return;
|
|
|
|
|
}
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
string fn = m_fileName + ".arrows";
|
|
|
|
|
if(!FileIsExist(fn, FILE_COMMON))
|
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach
Chart objects live in the MT5 chart PROFILE, not in this EA's files.
They survive a terminal restart, a recompile, and deleting every
.nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION
removes them - and MetaTrader force-terminates OnDeinit at roughly
4,500 ms, so a run killed mid-cleanup orphans them permanently with no
owner left to clean up after. That is the "deleted every file,
recompiled, restarted, old arrows and a stale panel still there"
report: nothing was wrong with the files and deleting them could not
have helped.
Both halves are fixed.
STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight
run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state
re-seed) and then write two full nets per chart. On four charts that is
the bulk of the budget, spent to preserve a PARTIAL era that was never
scored, never checkpointed and never deployable. FlushTrainRun()
discards it instead - drop the resumable bookkeeping, leave the net
neutral (unfreeze BN, flush the batch, batch size 1), skip the save -
and training resumes from the last completed era, which the era-end
save and the periodic autosave have already put on disk. What is
discarded is bounded by one era.
A CONVERGED model keeps the old finalise-and-save path: its weights can
carry online-learning updates made since the last era boundary, and for
a deployed model no further era boundary is coming to persist them.
MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model
loaded, sidecar missing - so the common paths returned leaving whatever
the previous instance stranded. LoadChartSignals now sweeps the arrow
namespace unconditionally before restoring, so the post-init chart
holds exactly what the sidecar holds whichever branch runs, and the
panel gets the same treatment before Create() (CAppDialog namespaces
its controls, so a killed Destroy strands the lot and the next attach
draws a second panel on the corpse).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
|
|
|
{
|
|
|
|
|
ChartRedraw(0);
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
return;
|
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach
Chart objects live in the MT5 chart PROFILE, not in this EA's files.
They survive a terminal restart, a recompile, and deleting every
.nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION
removes them - and MetaTrader force-terminates OnDeinit at roughly
4,500 ms, so a run killed mid-cleanup orphans them permanently with no
owner left to clean up after. That is the "deleted every file,
recompiled, restarted, old arrows and a stale panel still there"
report: nothing was wrong with the files and deleting them could not
have helped.
Both halves are fixed.
STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight
run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state
re-seed) and then write two full nets per chart. On four charts that is
the bulk of the budget, spent to preserve a PARTIAL era that was never
scored, never checkpointed and never deployable. FlushTrainRun()
discards it instead - drop the resumable bookkeeping, leave the net
neutral (unfreeze BN, flush the batch, batch size 1), skip the save -
and training resumes from the last completed era, which the era-end
save and the periodic autosave have already put on disk. What is
discarded is bounded by one era.
A CONVERGED model keeps the old finalise-and-save path: its weights can
carry online-learning updates made since the last era boundary, and for
a deployed model no further era boundary is coming to persist them.
MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model
loaded, sidecar missing - so the common paths returned leaving whatever
the previous instance stranded. LoadChartSignals now sweeps the arrow
namespace unconditionally before restoring, so the post-init chart
holds exactly what the sidecar holds whichever branch runs, and the
panel gets the same treatment before Create() (CAppDialog namespaces
its controls, so a killed Destroy strands the lot and the next attach
draws a second panel on the corpse).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
|
|
|
}
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//--- share flags: read-only, see CopySharedFile().
|
|
|
|
|
int handle = FileOpen(fn, FILE_COMMON | FILE_BIN | FILE_READ | FILE_SHARE_READ | FILE_SHARE_WRITE);
|
|
|
|
|
if(handle == INVALID_HANDLE)
|
|
|
|
|
return;
|
|
|
|
|
if(FileReadInteger(handle) != 0x57534152)
|
|
|
|
|
{
|
|
|
|
|
FileClose(handle);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
int n = FileReadInteger(handle);
|
|
|
|
|
//--- Sanity-cap a corrupt/garbage count so a bad header can never queue millions of object creations.
|
|
|
|
|
if(n < 0 || n > MAX_RESTORED_ARROWS)
|
|
|
|
|
{
|
|
|
|
|
Print(ID + ": .arrows header declares " + IntegerToString(n) + " arrows - refusing to restore (limit " +
|
|
|
|
|
IntegerToString(MAX_RESTORED_ARROWS) + "); the file looks corrupt. Chart signals will rebuild as the model runs.");
|
|
|
|
|
FileClose(handle);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
ArrayResize(m_arrowRestoreTime, n);
|
|
|
|
|
ArrayResize(m_arrowRestoreCode, n);
|
|
|
|
|
ArrayResize(m_arrowRestorePrice, n);
|
|
|
|
|
ArrayResize(m_arrowRestoreTf, n);
|
|
|
|
|
int parsed = 0;
|
|
|
|
|
for(int i = 0; i < n && !FileIsEnding(handle); i++)
|
|
|
|
|
{
|
|
|
|
|
m_arrowRestoreTime[parsed] = (datetime)FileReadLong(handle);
|
|
|
|
|
m_arrowRestoreCode[parsed] = FileReadInteger(handle);
|
|
|
|
|
m_arrowRestorePrice[parsed] = FileReadDouble(handle);
|
|
|
|
|
m_arrowRestoreTf[parsed] = FileReadLong(handle);
|
|
|
|
|
parsed++;
|
|
|
|
|
}
|
|
|
|
|
FileClose(handle);
|
|
|
|
|
//--- Trim to what actually parsed (a truncated file yields fewer records than the header claimed).
|
|
|
|
|
if(parsed != n)
|
|
|
|
|
{
|
|
|
|
|
ArrayResize(m_arrowRestoreTime, parsed);
|
|
|
|
|
ArrayResize(m_arrowRestoreCode, parsed);
|
|
|
|
|
ArrayResize(m_arrowRestorePrice, parsed);
|
|
|
|
|
ArrayResize(m_arrowRestoreTf, parsed);
|
|
|
|
|
}
|
|
|
|
|
//--- Apply the retention cap on the way IN as well, so a pre-cap file (one run had 2896 arrows) is
|
|
|
|
|
//--- trimmed on its first load instead of waiting for the next save - otherwise the very restore this
|
|
|
|
|
//--- cap exists to bound would still rebuild every one of them. Identical selection rule to
|
|
|
|
|
//--- SaveChartSignals: most recent by TIME, since file order is chart-scan order, i.e. arbitrary.
|
|
|
|
|
if(parsed > MAX_PERSISTED_ARROWS)
|
|
|
|
|
{
|
|
|
|
|
long sortedTimes[];
|
|
|
|
|
ArrayResize(sortedTimes, parsed);
|
|
|
|
|
for(int i = 0; i < parsed; i++)
|
|
|
|
|
sortedTimes[i] = (long)m_arrowRestoreTime[i];
|
|
|
|
|
ArraySort(sortedTimes); // ascending
|
|
|
|
|
long cutoff = sortedTimes[parsed - MAX_PERSISTED_ARROWS];
|
|
|
|
|
int w = 0;
|
|
|
|
|
for(int i = 0; i < parsed; i++)
|
|
|
|
|
if((long)m_arrowRestoreTime[i] >= cutoff && w < MAX_PERSISTED_ARROWS)
|
|
|
|
|
{
|
|
|
|
|
m_arrowRestoreTime[w] = m_arrowRestoreTime[i];
|
|
|
|
|
m_arrowRestoreCode[w] = m_arrowRestoreCode[i];
|
|
|
|
|
m_arrowRestorePrice[w] = m_arrowRestorePrice[i];
|
|
|
|
|
m_arrowRestoreTf[w] = m_arrowRestoreTf[i];
|
|
|
|
|
w++;
|
|
|
|
|
}
|
|
|
|
|
Print(ID + ": .arrows sidecar holds " + IntegerToString(parsed) + " arrows - restoring the " +
|
|
|
|
|
IntegerToString(w) + " most recent (retention cap " + IntegerToString(MAX_PERSISTED_ARROWS) +
|
|
|
|
|
"); the file is rewritten capped on the next save.");
|
|
|
|
|
parsed = w;
|
|
|
|
|
ArrayResize(m_arrowRestoreTime, parsed);
|
|
|
|
|
ArrayResize(m_arrowRestoreCode, parsed);
|
|
|
|
|
ArrayResize(m_arrowRestorePrice, parsed);
|
|
|
|
|
ArrayResize(m_arrowRestoreTf, parsed);
|
|
|
|
|
}
|
|
|
|
|
m_arrowRestoreIndex = 0;
|
|
|
|
|
m_arrowRestorePending = (parsed > 0);
|
|
|
|
|
m_arrowRestoreStartMs = GetTickCount();
|
|
|
|
|
//--- Deliberately does NOT draw anything here - see AdvanceChartSignalRestore's declaration comment for
|
|
|
|
|
//--- why this must not block OnInit. Drawing starts on the next timer/tick slice.
|
|
|
|
|
if(m_arrowRestorePending)
|
|
|
|
|
PrintVerbose(ID + ": queued " + IntegerToString(parsed) + " chart signal arrows for progressive restore");
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| See the declaration comment - draws a time-boxed slice of the |
|
|
|
|
|
//| arrows queued by LoadChartSignals(), so a chart carrying |
|
|
|
|
|
//| thousands of them fills in progressively instead of freezing the |
|
|
|
|
|
//| terminal during init. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::AdvanceChartSignalRestore(void)
|
|
|
|
|
{
|
|
|
|
|
if(!m_arrowRestorePending)
|
|
|
|
|
return;
|
|
|
|
|
int total = ArraySize(m_arrowRestoreTime);
|
|
|
|
|
uint sliceStart = GetTickCount();
|
|
|
|
|
int drawnThisSlice = 0;
|
|
|
|
|
while(m_arrowRestoreIndex < total)
|
|
|
|
|
{
|
|
|
|
|
//--- check the budget every 64 objects rather than every one: GetTickCount() itself is not free, and
|
|
|
|
|
//--- at ~microseconds per object a per-iteration clock read would dominate the work being measured.
|
|
|
|
|
if(drawnThisSlice > 0 && (drawnThisSlice & 63) == 0 && GetTickCount() - sliceStart >= ARROW_RESTORE_BUDGET_MS)
|
|
|
|
|
break;
|
|
|
|
|
datetime t = m_arrowRestoreTime[m_arrowRestoreIndex];
|
|
|
|
|
int code = m_arrowRestoreCode[m_arrowRestoreIndex];
|
|
|
|
|
double price = m_arrowRestorePrice[m_arrowRestoreIndex];
|
|
|
|
|
long tf = m_arrowRestoreTf[m_arrowRestoreIndex];
|
|
|
|
|
m_arrowRestoreIndex++;
|
|
|
|
|
drawnThisSlice++;
|
|
|
|
|
string nm = SIG_ARROW_PREFIX + TimeToString(t);
|
|
|
|
|
//--- Deliberately NO ObjectFind() pre-check. ObjectFind scans the chart's object list, so calling it
|
|
|
|
|
//--- per arrow made the restore O(n^2) - with ~2900 arrows that is ~4.2M name comparisons, which is
|
|
|
|
|
//--- what actually froze the terminal (observed 2026-07-26; previously masked because the filename
|
|
|
|
|
//--- bug made LoadChartSignals always miss its file). ObjectCreate already returns false when the
|
|
|
|
|
//--- name exists, and re-applying the properties below is harmless and exactly what a refresh does.
|
|
|
|
|
ObjectCreate(0, nm, OBJ_ARROW, 0, t, 0);
|
|
|
|
|
ObjectSetDouble(0, nm, OBJPROP_PRICE, price);
|
|
|
|
|
ObjectSetInteger(0, nm, OBJPROP_ARROWCODE, code);
|
|
|
|
|
ObjectSetInteger(0, nm, OBJPROP_COLOR, code == 217 ? clrBlue : clrRed);
|
|
|
|
|
ObjectSetInteger(0, nm, OBJPROP_ANCHOR, code == 217 ? ANCHOR_TOP : ANCHOR_BOTTOM);
|
|
|
|
|
ObjectSetInteger(0, nm, OBJPROP_TIMEFRAMES, tf); // restore the saved show/hide state
|
|
|
|
|
}
|
|
|
|
|
ChartRedraw(0);
|
|
|
|
|
if(m_arrowRestoreIndex < total)
|
|
|
|
|
return; // more slices to come
|
|
|
|
|
//--- done - release the buffers and report once
|
|
|
|
|
m_arrowRestorePending = false;
|
|
|
|
|
Print(ID + ": restored " + IntegerToString(total) + " chart signal arrows in " +
|
|
|
|
|
IntegerToString((int)(GetTickCount() - m_arrowRestoreStartMs)) + " ms (progressive, non-blocking)");
|
|
|
|
|
ArrayFree(m_arrowRestoreTime);
|
|
|
|
|
ArrayFree(m_arrowRestoreCode);
|
|
|
|
|
ArrayFree(m_arrowRestorePrice);
|
|
|
|
|
ArrayFree(m_arrowRestoreTf);
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| Time-boxed slice of the per-bar re-inference queued by |
|
|
|
|
|
//| StartChartSignalRescan(). Same doctrine as |
|
|
|
|
|
//| AdvanceChartSignalRestore() above: one feedForward per bar over up |
|
|
|
|
|
//| to SIGNAL_RESCAN_LOOKBACK_BARS bars is real, non-trivial compute |
|
|
|
|
|
//| (not a cheap object-property write like the arrow restore), so |
|
|
|
|
|
//| the slice is bounded by elapsed time rather than a fixed count - |
|
|
|
|
|
//| checked every bar since a single feedForward can itself approach |
|
|
|
|
|
//| the whole budget on a slow (CPU-DLL, no OpenCL) backend. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::AdvanceChartSignalRescan(void)
|
|
|
|
|
{
|
|
|
|
|
if(!m_rescanPending)
|
|
|
|
|
return;
|
|
|
|
|
uint sliceStart = GetTickCount();
|
|
|
|
|
EnsureShadowNet();
|
|
|
|
|
CNet *deployNet = (CheckPointer(m_shadowNet) != POINTER_INVALID) ? m_shadowNet : Net;
|
|
|
|
|
while(m_rescanIndex < m_rescanHi)
|
|
|
|
|
{
|
|
|
|
|
if(GetTickCount() - sliceStart >= ARROW_RESTORE_BUDGET_MS)
|
|
|
|
|
break;
|
|
|
|
|
int i = m_rescanIndex;
|
|
|
|
|
m_rescanIndex++;
|
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
|
|
|
//--- Same builder the live signal uses, so a restored arrow means what the deployed model would
|
|
|
|
|
//--- actually have said on that bar rather than a lookalike assembled by a parallel loop.
|
|
|
|
|
if(!BuildFeatureWindow(i))
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
continue;
|
|
|
|
|
deployNet.feedForward(TempData);
|
|
|
|
|
deployNet.getResults(TempData);
|
|
|
|
|
if(m_outputNeuronsCount == 1)
|
|
|
|
|
m_arrowSignalCache[i] = TempData[0];
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
double rawSignal = ApplyClassificationSoftmax();
|
|
|
|
|
switch(DoubleToSignal(rawSignal))
|
|
|
|
|
{
|
|
|
|
|
case Buy:
|
|
|
|
|
m_rescanRawBuy++;
|
|
|
|
|
break;
|
|
|
|
|
case Sell:
|
|
|
|
|
m_rescanRawSell++;
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
m_rescanRawNeutral++;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
m_arrowSignalCache[i] = AdjustedSignalFromSoftmax();
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
if(m_rescanIndex < m_rescanHi)
|
|
|
|
|
return; // more slices to come
|
|
|
|
|
//--- done - render the declustered set once, same as the old blocking pass did
|
|
|
|
|
PruneDirectionalClusters(m_rescanBarsNow);
|
|
|
|
|
SaveChartSignals();
|
|
|
|
|
m_rescanPending = false;
|
|
|
|
|
//--- Tally what the model actually called BEFORE decluttering, so an empty-looking chart is
|
|
|
|
|
//--- distinguishable in the log between "the model called Neutral almost everywhere" (a real,
|
|
|
|
|
//--- if extreme, calibration outcome) and "arrows were computed but never rendered" (a bug) -
|
|
|
|
|
//--- both look identical on the chart otherwise.
|
|
|
|
|
int buyCount = 0, sellCount = 0, neutralCount = 0, unscored = 0;
|
|
|
|
|
for(int k = 0; k < m_rescanHi; k++)
|
|
|
|
|
{
|
|
|
|
|
if(m_arrowSignalCache[k] == -2.0)
|
|
|
|
|
{
|
|
|
|
|
unscored++;
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
switch(DoubleToSignal(m_arrowSignalCache[k]))
|
|
|
|
|
{
|
|
|
|
|
case Buy:
|
|
|
|
|
buyCount++;
|
|
|
|
|
break;
|
|
|
|
|
case Sell:
|
|
|
|
|
sellCount++;
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
neutralCount++;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
Print(ID + ": rescanned " + IntegerToString(m_rescanBarsNow) + " bars against the deployed model and rebuilt the chart signal arrows in " +
|
|
|
|
|
IntegerToString((int)(GetTickCount() - m_rescanStartMs)) + " ms (progressive, non-blocking) - post-calibration calls (pre-decluttering): " +
|
|
|
|
|
IntegerToString(buyCount) + " Buy, " + IntegerToString(sellCount) + " Sell, " + IntegerToString(neutralCount) + " Neutral, " +
|
|
|
|
|
IntegerToString(unscored) + " unscored" +
|
|
|
|
|
(m_outputNeuronsCount == 3 ? " | RAW network argmax (before logit-prior correction): " +
|
|
|
|
|
IntegerToString(m_rescanRawBuy) + " Buy, " + IntegerToString(m_rescanRawSell) + " Sell, " +
|
|
|
|
|
IntegerToString(m_rescanRawNeutral) + " Neutral" : ""));
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| Compounded/persistent accuracy line for the simple panels. |
|
|
|
|
|
//| See the declaration comment (class body) for why this exists. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
string CExpertSignalAIBase::ComputeCompoundedAccuracyLine(void)
|
|
|
|
|
{
|
|
|
|
|
if(m_cumIsTotal <= 0 && m_cumOosTotal <= 0)
|
fix: the imbalance correction never ran during the auto-tune search
Neutral collapse on all four topologies by era 5 with a 2:6 barrier
(recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on
"measuring...". One root cause, and it was not the barrier.
The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is
exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1%
of Neutral coming from the vertical barrier - so the new m*k horizon
scaling is right, arguably generous.
What was broken: Train()'s era-start block wrapped UpdateClassPriors() in
`if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode,
and AutoTuneIndicators ships ON, so on a default configuration EVERY era
of the search ran with unmeasured priors. ApplyLogitAdjustment() requires
measured priors; without them it calls ClearLogitAdjustment() and returns.
So the entire search trained under PLAIN cross-entropy. With a 52.5%
majority class the optimum of plain CE is "always predict Neutral", and
that is precisely what all four models found. The panel followed: its
counters only advance on bars the model CALLED Buy or Sell, so a
collapsed model leaves them at zero and the line reads "measuring..."
forever.
This was latent, not new. It has been true for every auto-tuned run, but
it was invisible while the labels were near-balanced - last night's
accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to
collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight
to survive noise) moved Neutral to the majority and exposed it.
The guard's stated fear cannot happen. These priors are measured from the
LABEL distribution, and the tuner only perturbs indicator periods
(MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and
TP_Mode - none of which the search touches - so every candidate sees
byte-identical labels and identical priors. There is nothing to
contaminate. What the guard actually protected was the .stats write, and
that is gated separately: eval candidates never checkpoint and never
persist.
Also, because this is the THIRD quiet no-op to cost a run in this
codebase (after the fictional oversampling log line and the shadow-blend
skip):
- ApplyLogitAdjustment() now WARNS when it declines to install, instead
of silently clearing. A mechanism that cannot announce it is not
running is indistinguishable from one that is.
- The panel distinguishes "measuring..." (before era 1, nothing scored
yet - an honest warm-up) from "no directional calls yet" (eras trained,
zero calls - a finding, not a wait).
Both builds compile 0 errors / 0 warnings. No retrain forced by this
commit itself, but the collapsed models must be discarded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
|
|
|
{
|
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
|
|
|
//--- These counters are LIFETIME and persisted (they are what the panel presents as the product's
|
|
|
|
|
//--- accuracy), so they only advance on bars the model CALLED. They sit at zero in two very
|
|
|
|
|
//--- different situations and the panel must not describe both as a wait.
|
|
|
|
|
//--- This previously read "measuring..." forever, and the 2026-08-01 report was "they seem to be
|
|
|
|
|
//--- signaling but the label stays stuck" - at the time an auto-tune GA was running every era as a
|
|
|
|
|
//--- throwaway candidate, so nothing was counted for hours while the chart filled with arrows. That
|
|
|
|
|
//--- particular gap is closed at the source: tuning is now a filter pass that finishes in seconds
|
|
|
|
|
//--- (see TuneIndicatorsByFilter), so a run reaches real, counted eras almost immediately.
|
|
|
|
|
//--- Before era 1 nothing has been scored, so "measuring..." is honest; after eras have run, zero
|
|
|
|
|
//--- directional calls is a FINDING (a Neutral collapse) and must not read as a wait.
|
fix: the imbalance correction never ran during the auto-tune search
Neutral collapse on all four topologies by era 5 with a 2:6 barrier
(recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on
"measuring...". One root cause, and it was not the barrier.
The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is
exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1%
of Neutral coming from the vertical barrier - so the new m*k horizon
scaling is right, arguably generous.
What was broken: Train()'s era-start block wrapped UpdateClassPriors() in
`if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode,
and AutoTuneIndicators ships ON, so on a default configuration EVERY era
of the search ran with unmeasured priors. ApplyLogitAdjustment() requires
measured priors; without them it calls ClearLogitAdjustment() and returns.
So the entire search trained under PLAIN cross-entropy. With a 52.5%
majority class the optimum of plain CE is "always predict Neutral", and
that is precisely what all four models found. The panel followed: its
counters only advance on bars the model CALLED Buy or Sell, so a
collapsed model leaves them at zero and the line reads "measuring..."
forever.
This was latent, not new. It has been true for every auto-tuned run, but
it was invisible while the labels were near-balanced - last night's
accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to
collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight
to survive noise) moved Neutral to the majority and exposed it.
The guard's stated fear cannot happen. These priors are measured from the
LABEL distribution, and the tuner only perturbs indicator periods
(MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and
TP_Mode - none of which the search touches - so every candidate sees
byte-identical labels and identical priors. There is nothing to
contaminate. What the guard actually protected was the .stats write, and
that is gated separately: eval candidates never checkpoint and never
persist.
Also, because this is the THIRD quiet no-op to cost a run in this
codebase (after the fictional oversampling log line and the shadow-blend
skip):
- ApplyLogitAdjustment() now WARNS when it declines to install, instead
of silently clearing. A mechanism that cannot announce it is not
running is indistinguishable from one that is.
- The panel distinguishes "measuring..." (before era 1, nothing scored
yet - an honest warm-up) from "no directional calls yet" (eras trained,
zero calls - a finding, not a wait).
Both builds compile 0 errors / 0 warnings. No retrain forced by this
commit itself, but the collapsed models must be discarded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
|
|
|
if(m_eraCount > 0)
|
|
|
|
|
return "Buy/Sell calls correct: no directional calls yet";
|
fix(ui): unique chart tag, product-grade panel, responsive under load
Three separate reports from one deploy.
1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights
fingerprint omits the topology type on purpose - the file path already
separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a
value that is constant within a folder buys nothing while re-keying
every trained model into a forced retrain. So the files were never at
risk, but the tag could not do its one job. Prefixing the short id
makes it unique on the display side only; the hex half still greps
straight to the .nnw inside the folder the prefix names.
2. The default panel read like a training console. Six lines down to
three, each answering a question an owner actually has. The deploy
internals (best score, eras-since-best, ladder stage) were developer
diagnostics describing a recall floor that no longer decides anything,
and were already in the era-end journal line. In-sample accuracy left
the panel too: it grades the model on bars it trained on, so it always
flatters, and showing it beside the honest number invites reading the
wrong one. New compile-time DebuggingMode constant - deliberately not
an input - carries the IS/OOS pair and the resolved model path into
the journal instead. No extra Inputs row, no extra Market description
line, no user-reachable firehose.
3. Panel drag and buttons stuttered under training load, exactly as the
2026-07-26 note raising the chunk budget to 200ms warned they might.
Backed off to the documented 120ms - worst-case click latency is that
budget - and the derived topology (~292k weights to ~29k) makes the
throughput this costs far cheaper than when that note was written.
Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the
whole chart, so its cost scales with accumulated arrows, and 5 Hz was
the larger half of the stutter. Era-end still force-refreshes.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
|
|
|
return "Buy/Sell calls correct: measuring...";
|
fix: the imbalance correction never ran during the auto-tune search
Neutral collapse on all four topologies by era 5 with a 2:6 barrier
(recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on
"measuring...". One root cause, and it was not the barrier.
The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is
exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1%
of Neutral coming from the vertical barrier - so the new m*k horizon
scaling is right, arguably generous.
What was broken: Train()'s era-start block wrapped UpdateClassPriors() in
`if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode,
and AutoTuneIndicators ships ON, so on a default configuration EVERY era
of the search ran with unmeasured priors. ApplyLogitAdjustment() requires
measured priors; without them it calls ClearLogitAdjustment() and returns.
So the entire search trained under PLAIN cross-entropy. With a 52.5%
majority class the optimum of plain CE is "always predict Neutral", and
that is precisely what all four models found. The panel followed: its
counters only advance on bars the model CALLED Buy or Sell, so a
collapsed model leaves them at zero and the line reads "measuring..."
forever.
This was latent, not new. It has been true for every auto-tuned run, but
it was invisible while the labels were near-balanced - last night's
accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to
collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight
to survive noise) moved Neutral to the majority and exposed it.
The guard's stated fear cannot happen. These priors are measured from the
LABEL distribution, and the tuner only perturbs indicator periods
(MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and
TP_Mode - none of which the search touches - so every candidate sees
byte-identical labels and identical priors. There is nothing to
contaminate. What the guard actually protected was the .stats write, and
that is gated separately: eval candidates never checkpoint and never
persist.
Also, because this is the THIRD quiet no-op to cost a run in this
codebase (after the fictional oversampling log line and the shadow-blend
skip):
- ApplyLogitAdjustment() now WARNS when it declines to install, instead
of silently clearing. A mechanism that cannot announce it is not
running is indistinguishable from one that is.
- The panel distinguishes "measuring..." (before era 1, nothing scored
yet - an honest warm-up) from "no directional calls yet" (eras trained,
zero calls - a finding, not a wait).
Both builds compile 0 errors / 0 warnings. No retrain forced by this
commit itself, but the collapsed models must be discarded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
|
|
|
}
|
fix(ui): unique chart tag, product-grade panel, responsive under load
Three separate reports from one deploy.
1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights
fingerprint omits the topology type on purpose - the file path already
separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a
value that is constant within a folder buys nothing while re-keying
every trained model into a forced retrain. So the files were never at
risk, but the tag could not do its one job. Prefixing the short id
makes it unique on the display side only; the hex half still greps
straight to the .nnw inside the folder the prefix names.
2. The default panel read like a training console. Six lines down to
three, each answering a question an owner actually has. The deploy
internals (best score, eras-since-best, ladder stage) were developer
diagnostics describing a recall floor that no longer decides anything,
and were already in the era-end journal line. In-sample accuracy left
the panel too: it grades the model on bars it trained on, so it always
flatters, and showing it beside the honest number invites reading the
wrong one. New compile-time DebuggingMode constant - deliberately not
an input - carries the IS/OOS pair and the resolved model path into
the journal instead. No extra Inputs row, no extra Market description
line, no user-reachable firehose.
3. Panel drag and buttons stuttered under training load, exactly as the
2026-07-26 note raising the chunk budget to 200ms warned they might.
Backed off to the documented 120ms - worst-case click latency is that
budget - and the derived topology (~292k weights to ~29k) makes the
throughput this costs far cheaper than when that note was written.
Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the
whole chart, so its cost scales with accumulated arrows, and 5 Hz was
the larger half of the stutter. Era-end still force-refreshes.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
|
|
|
//--- OUT-OF-SAMPLE only on the panel. The in-sample figure is the model graded on bars it trained on,
|
|
|
|
|
//--- so it is always the flattering one and is never what forward trading delivers - showing both
|
|
|
|
|
//--- invites a buyer to read the higher number as the product's accuracy. The IS/OOS pair is genuinely
|
|
|
|
|
//--- useful (the gap between them is the over-fitting read), so it is not discarded, just moved to the
|
|
|
|
|
//--- journal under DebuggingMode where diagnosing it belongs. Falls back to in-sample, explicitly
|
|
|
|
|
//--- labelled, only in the brief window before the first out-of-sample era has been scored.
|
|
|
|
|
if(m_cumOosTotal > 0)
|
|
|
|
|
{
|
|
|
|
|
//--- Labelled "lifetime" deliberately. These counters are monotonic and never reset per era (only
|
|
|
|
|
//--- on a fresh model / reset-weights), and they persist across restarts via .stats, so this is the
|
|
|
|
|
//--- average over EVERY era ever trained - not the current era's performance. At era 200+ a single
|
|
|
|
|
//--- new era shifts it by a fraction of a percent, so a model that started badly and has since
|
|
|
|
|
//--- recovered still reads low here.
|
2026-08-10 12:05:49 -04:00
|
|
|
//---
|
|
|
|
|
//--- ALWAYS SHOWN WITH ITS BREAK-EVEN. This counter is a WIN RATE (m_cumOosCorrect advances on
|
|
|
|
|
//--- oTradeWon - did the implied trade reach its target before its stop), NOT label agreement, and
|
|
|
|
|
//--- a win rate is meaningless without the barrier that produced it. With the measured geometry a
|
|
|
|
|
//--- trade risks slMult*ATR to make tpMult*ATR, so under a driftless walk ANY directional call wins
|
|
|
|
|
//--- slMult/(slMult+tpMult) of the time for free. On the shipped 3.33/1.62 pair that is 67.3%, and
|
|
|
|
|
//--- all four topologies sat at 65% - i.e. at chance and BELOW break-even - while the panel
|
|
|
|
|
//--- announced "65% correct", which reads as skill to anyone who does not know the barrier
|
|
|
|
|
//--- (2026-08-10: the four models agreeing on one number despite completely different trade counts
|
|
|
|
|
//--- was the tell). The deploy gate already benchmarks against this null (chancePrecPct); the panel
|
|
|
|
|
//--- did not, and the panel is what a buyer reads.
|
2026-08-10 13:51:53 -04:00
|
|
|
//--- ONE LINE. The break-even was a second, wrapped line explaining the risk/reward that sets it;
|
|
|
|
|
//--- that is journal detail, not panel detail, and it cost more vertical space than it was worth.
|
|
|
|
|
//--- The COMPARISON stays, in three words, because without it the win rate reads as skill when it
|
|
|
|
|
//--- is the geometry's own base rate - which is exactly how four models sitting at chance came to
|
|
|
|
|
//--- look like four models at 65% (see the deploy gate's chancePrecPct for the same test).
|
2026-08-10 12:05:49 -04:00
|
|
|
double slMultBe, tpMultBe;
|
|
|
|
|
BarrierMultiples(slMultBe, tpMultBe);
|
|
|
|
|
int winPct = (int)MathRound(m_cumOosCorrect * 100.0 / m_cumOosTotal);
|
2026-08-10 13:51:53 -04:00
|
|
|
string need = "";
|
2026-08-10 12:05:49 -04:00
|
|
|
if(slMultBe > 0.0 && tpMultBe > 0.0)
|
2026-08-10 13:51:53 -04:00
|
|
|
need = ", need " + IntegerToString((int)MathRound(100.0 * slMultBe / (slMultBe + tpMultBe))) + "%";
|
|
|
|
|
return "Buy/Sell calls correct: " + IntegerToString(winPct) + "% (unseen data" + need + ")";
|
fix(ui): unique chart tag, product-grade panel, responsive under load
Three separate reports from one deploy.
1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights
fingerprint omits the topology type on purpose - the file path already
separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a
value that is constant within a folder buys nothing while re-keying
every trained model into a forced retrain. So the files were never at
risk, but the tag could not do its one job. Prefixing the short id
makes it unique on the display side only; the hex half still greps
straight to the .nnw inside the folder the prefix names.
2. The default panel read like a training console. Six lines down to
three, each answering a question an owner actually has. The deploy
internals (best score, eras-since-best, ladder stage) were developer
diagnostics describing a recall floor that no longer decides anything,
and were already in the era-end journal line. In-sample accuracy left
the panel too: it grades the model on bars it trained on, so it always
flatters, and showing it beside the honest number invites reading the
wrong one. New compile-time DebuggingMode constant - deliberately not
an input - carries the IS/OOS pair and the resolved model path into
the journal instead. No extra Inputs row, no extra Market description
line, no user-reachable firehose.
3. Panel drag and buttons stuttered under training load, exactly as the
2026-07-26 note raising the chunk budget to 200ms warned they might.
Backed off to the documented 120ms - worst-case click latency is that
budget - and the derived topology (~292k weights to ~29k) makes the
throughput this costs far cheaper than when that note was written.
Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the
whole chart, so its cost scales with accumulated arrows, and 5 Hz was
the larger half of the stutter. Era-end still force-refreshes.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
|
|
|
}
|
|
|
|
|
return "Buy/Sell calls correct: " +
|
|
|
|
|
IntegerToString((int)MathRound(m_cumIsCorrect * 100.0 / m_cumIsTotal)) +
|
|
|
|
|
"% (training data so far)";
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| See the declaration comment (class body) for why this exists. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::UpdateTrainingStatusLabel(const string &progressLine, double neuron0, double neuron1, double neuron2, double signalValue, bool forceRefresh)
|
|
|
|
|
{
|
|
|
|
|
// Cache regardless of whether this particular call actually redraws below - see
|
|
|
|
|
// m_lastDisplayNeuron0's declaration comment for why the era-end forced refresh needs these.
|
|
|
|
|
m_lastDisplayNeuron0 = neuron0;
|
|
|
|
|
m_lastDisplayNeuron1 = neuron1;
|
|
|
|
|
m_lastDisplayNeuron2 = neuron2;
|
|
|
|
|
m_lastDisplaySignal = signalValue;
|
|
|
|
|
// Throttle: SetStatusLabel()'s ChartRedraw() is real work that gets slower as more chart objects
|
|
|
|
|
// accumulate over a long backtest - calling it every single bar across three passes (instead of
|
|
|
|
|
// pass 1 alone, the original frequency) is what actually stalled a run for 1.5+ hours without
|
|
|
|
|
// finishing era 0, not the shuffle itself. ~5 updates/sec is still visually live. forceRefresh
|
|
|
|
|
// bypasses this - see this method's declaration comment for why the era-end call needs to.
|
fix(ui): unique chart tag, product-grade panel, responsive under load
Three separate reports from one deploy.
1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights
fingerprint omits the topology type on purpose - the file path already
separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a
value that is constant within a folder buys nothing while re-keying
every trained model into a forced retrain. So the files were never at
risk, but the tag could not do its one job. Prefixing the short id
makes it unique on the display side only; the hex half still greps
straight to the .nnw inside the folder the prefix names.
2. The default panel read like a training console. Six lines down to
three, each answering a question an owner actually has. The deploy
internals (best score, eras-since-best, ladder stage) were developer
diagnostics describing a recall floor that no longer decides anything,
and were already in the era-end journal line. In-sample accuracy left
the panel too: it grades the model on bars it trained on, so it always
flatters, and showing it beside the honest number invites reading the
wrong one. New compile-time DebuggingMode constant - deliberately not
an input - carries the IS/OOS pair and the resolved model path into
the journal instead. No extra Inputs row, no extra Market description
line, no user-reachable firehose.
3. Panel drag and buttons stuttered under training load, exactly as the
2026-07-26 note raising the chunk budget to 200ms warned they might.
Backed off to the documented 120ms - worst-case click latency is that
budget - and the derived topology (~292k weights to ~29k) makes the
throughput this costs far cheaper than when that note was written.
Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the
whole chart, so its cost scales with accumulated arrows, and 5 Hz was
the larger half of the stutter. Era-end still force-refreshes.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
|
|
|
// 2026-07-30 raised 200 -> 400. The second half of the same responsiveness complaint the training
|
|
|
|
|
// chunk budget addresses, and on a long-running chart the larger half: ChartRedraw() repaints the
|
|
|
|
|
// WHOLE chart, so its cost scales with the accumulated signal arrows, and firing it five times a
|
|
|
|
|
// second is what makes dragging the panel stutter rather than any single call being slow. Nothing
|
|
|
|
|
// on the simple panel changes fast enough to need 5 Hz - era and accuracy move once per era, and
|
|
|
|
|
// the progress percentage is smooth at 2.5 Hz. The era-end call passes forceRefresh and bypasses
|
|
|
|
|
// this entirely, so no state transition is ever delayed by it.
|
|
|
|
|
const uint STATUS_LABEL_THROTTLE_MS = 400;
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
uint nowTick = GetTickCount();
|
|
|
|
|
if(!forceRefresh && m_lastStatusLabelUpdateTick != 0 && nowTick - m_lastStatusLabelUpdateTick < STATUS_LABEL_THROTTLE_MS)
|
|
|
|
|
return;
|
|
|
|
|
m_lastStatusLabelUpdateTick = nowTick;
|
|
|
|
|
//--- Simple panel (default, VerboseMode off): only what a non-technical user can act on - how far
|
|
|
|
|
//--- training has progressed, how reliable its LIVE Buy/Sell calls are (hit-rate = live precision:
|
|
|
|
|
//--- the same bars the deployed EA would actually trade, so it is a true forward-trading expectation,
|
|
|
|
|
//--- not the Neutral-inflated headline accuracy), and what it is signalling right now. Every raw NN
|
|
|
|
|
//--- internal (in-sample error, MSE, softmax neurons, confusion counts, recall) is diagnostic and is
|
|
|
|
|
//--- shown only under VerboseMode in the detailed block below.
|
|
|
|
|
if(!VerboseMode)
|
|
|
|
|
{
|
|
|
|
|
ENUM_SIGNAL curSig = DoubleToSignal(signalValue);
|
|
|
|
|
string sigPlain = (curSig == Buy) ? "Buy" : (curSig == Sell) ? "Sell" : "Neutral (no trade)";
|
2026-08-10 07:22:29 -04:00
|
|
|
//--- Progress of the pass ACTUALLY RUNNING, published by each pass (m_passProgressPct). This used
|
|
|
|
|
//--- to be computed here as (m_isTrainCursor+1)/max(m_isTrainQueueCount,1) - pass 2's counters -
|
|
|
|
|
//--- which during pass 1 is (0+1)/max(0,1) = 100%. So an era spent its entire scan pass showing
|
|
|
|
|
//--- "100%", i.e. the phase that can take minutes reported itself as finished. Reported by the
|
|
|
|
|
//--- user 2026-08-10 as "started learning at 100% and stuck there", and it cost real diagnostic
|
|
|
|
|
//--- time: the one number on screen said the opposite of what was happening.
|
|
|
|
|
int progressPct = m_passProgressPct;
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//--- Compounded, persistent DIRECTIONAL win-rate (Buy/Sell only, Neutral excluded - see
|
|
|
|
|
//--- m_cumIsCorrect) - a stable number that keeps refining across eras and restarts, not the noisy
|
|
|
|
|
//--- per-era metric that sat at "measuring", and not the Neutral-inflated all-class rate. IS =
|
|
|
|
|
//--- in-sample (bars it trained on), OOS = out-of-sample (held-out; the honest generalization read -
|
|
|
|
|
//--- the gap between them shows over/under-fitting).
|
|
|
|
|
string accLine = ComputeCompoundedAccuracyLine();
|
fix(ui): unique chart tag, product-grade panel, responsive under load
Three separate reports from one deploy.
1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights
fingerprint omits the topology type on purpose - the file path already
separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a
value that is constant within a folder buys nothing while re-keying
every trained model into a forced retrain. So the files were never at
risk, but the tag could not do its one job. Prefixing the short id
makes it unique on the display side only; the hex half still greps
straight to the .nnw inside the folder the prefix names.
2. The default panel read like a training console. Six lines down to
three, each answering a question an owner actually has. The deploy
internals (best score, eras-since-best, ladder stage) were developer
diagnostics describing a recall floor that no longer decides anything,
and were already in the era-end journal line. In-sample accuracy left
the panel too: it grades the model on bars it trained on, so it always
flatters, and showing it beside the honest number invites reading the
wrong one. New compile-time DebuggingMode constant - deliberately not
an input - carries the IS/OOS pair and the resolved model path into
the journal instead. No extra Inputs row, no extra Market description
line, no user-reachable firehose.
3. Panel drag and buttons stuttered under training load, exactly as the
2026-07-26 note raising the chunk budget to 200ms warned they might.
Backed off to the documented 120ms - worst-case click latency is that
budget - and the derived topology (~292k weights to ~29k) makes the
throughput this costs far cheaper than when that note was written.
Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the
whole chart, so its cost scales with accumulated arrows, and 5 Hz was
the larger half of the stutter. Era-end still force-refreshes.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
|
|
|
//--- The checkpoint/deploy internals - best score so far, whether it clears the deploy gate,
|
|
|
|
|
//--- eras-since-best, plateau-ladder stage - used to occupy two more panel lines. They are
|
|
|
|
|
//--- development diagnostics: a buyer cannot act on "ladder stage 2/3", and the panel is the
|
|
|
|
|
//--- product's face. Emitted to the journal under DebuggingMode instead, so nothing is lost while
|
|
|
|
|
//--- working on the EA - but from the ERA-END summary (Training.mqh), not from here: this method is
|
|
|
|
|
//--- throttled to ~5 calls/second, so a Print() at this site would put five lines per second into
|
|
|
|
|
//--- the journal, and the state it reports only changes once per era anyway. Deliberately not
|
|
|
|
|
//--- folded into VerboseMode either: that switch is the supported power-user view, and this is
|
|
|
|
|
//--- internal state whose meaning shifts as the selection rule does (it described a per-class
|
|
|
|
|
//--- recall floor that no longer decides anything).
|
|
|
|
|
//--- Three lines, and each one answers a question an owner actually has: what is it doing and how
|
|
|
|
|
//--- far along, how well has it been calling direction, and what is it saying right now.
|
2026-08-10 07:22:29 -04:00
|
|
|
//--- Naming the phase as well as the percentage: an era is three passes over ~38k bars each, so
|
|
|
|
|
//--- "34%" alone cannot distinguish steady progress from a stall, and all three passes look
|
|
|
|
|
//--- identical from outside. m_passLabel is set by whichever pass is running.
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
string simple = StringFormat(
|
2026-08-10 07:22:29 -04:00
|
|
|
DisplayName() + " - learning (era %d, %s %d%%)\n" +
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
"%s\n" +
|
|
|
|
|
"Current signal: %s",
|
2026-08-10 07:22:29 -04:00
|
|
|
m_eraCount, m_passLabel, progressPct, accLine, sigPlain);
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
SetStatusLabel(simple);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// Recall/precision-by-class and the continual-learning OOS sim are still tracked (used for the
|
|
|
|
|
// convergence gate elsewhere) but dropped from the on-chart status text - it made the panel too
|
|
|
|
|
// tall/wordy for a one-line-per-metric display; the counts below are enough at a glance.
|
|
|
|
|
string classLine = StringFormat(
|
|
|
|
|
"Predicted -> Buy: %d Sell: %d Neutral: %d\n" +
|
|
|
|
|
"Actual -> Buy: %d Sell: %d Neutral: %d",
|
|
|
|
|
m_countBuySignals, m_countSellSignals, m_countNeutralSignals,
|
|
|
|
|
m_trueBuyCount, m_trueSellCount, m_trueNeutralCount
|
|
|
|
|
);
|
|
|
|
|
string oosErrStr = (m_oosSamples > 0 ? DoubleToString(dOosError, 2) : "N/A");
|
|
|
|
|
string s;
|
|
|
|
|
if(m_outputNeuronsCount == 1)
|
|
|
|
|
s = StringFormat(
|
|
|
|
|
ID + " : Study -> Era %d\n" +
|
|
|
|
|
"%s\n" +
|
|
|
|
|
"IS %d%% Acc: %.2f%% MSE: %.2f AvgErr: %.2f\n" +
|
|
|
|
|
"OOS %d%% Acc: %.2f%% Mismatch: %s Samples: %d\n" +
|
|
|
|
|
"Signal Neuron: %.5f\n" +
|
|
|
|
|
"Forecast: %s -> %.2f\n" +
|
|
|
|
|
"%s",
|
|
|
|
|
m_eraCount, progressLine,
|
|
|
|
|
100 - m_oosSplitPct, dForecast,
|
|
|
|
|
dError,
|
|
|
|
|
Net.getRecentAverageError(),
|
|
|
|
|
m_oosSplitPct, dOosForecast, oosErrStr, m_oosSamples,
|
|
|
|
|
neuron0,
|
|
|
|
|
EnumToString(DoubleToSignal(signalValue)), signalValue,
|
|
|
|
|
classLine
|
|
|
|
|
);
|
|
|
|
|
else
|
|
|
|
|
if(m_outputNeuronsCount == 3)
|
|
|
|
|
{
|
|
|
|
|
//--- Headline OOS accuracy above blends in Neutral (usually the majority class, and "don't
|
|
|
|
|
//--- trade" rather than a call that can be right or wrong the way Buy/Sell are) - a model can
|
|
|
|
|
//--- score well on it while its actual Buy/Sell calls are unreliable. This is the number that
|
|
|
|
|
//--- predicts trading performance, so it gets its own line rather than staying log-only.
|
|
|
|
|
string oosDirRecall = "OOS recall Buy: " + (m_lastBuyRecallPct < 0 ? "n/a" : IntegerToString(m_lastBuyRecallPct) + "%") +
|
|
|
|
|
" Sell: " + (m_lastSellRecallPct < 0 ? "n/a" : IntegerToString(m_lastSellRecallPct) + "%");
|
|
|
|
|
//--- Live-trade precision line: precision on ONLY the calls that clear the confidence floor under
|
|
|
|
|
//--- the live/prior-corrected rule, with the fire count in parentheses. This is the metric that
|
|
|
|
|
//--- matches forward trading - the deployed EA takes exactly these bars (see AdjustedSignalFromSoftmax).
|
|
|
|
|
string oosLivePrec = "Live win rate Buy: " + (m_lastBuyFiredPrecPct < 0 ? "n/a" : IntegerToString(m_lastBuyFiredPrecPct) + "%") +
|
|
|
|
|
" (" + IntegerToString(m_lastBuyFired) + ") Sell: " + (m_lastSellFiredPrecPct < 0 ? "n/a" : IntegerToString(m_lastSellFiredPrecPct) + "%") +
|
|
|
|
|
" (" + IntegerToString(m_lastSellFired) + ")";
|
|
|
|
|
s = StringFormat(
|
|
|
|
|
ID + " : Study -> Era %d\n" +
|
|
|
|
|
"%s\n" +
|
|
|
|
|
"IS %d%% Acc: %.2f%% MSE: %.2f AvgErr: %.2f\n" +
|
|
|
|
|
"OOS %d%% Acc: %.2f%% Mismatch: %s Samples: %d\n" +
|
|
|
|
|
"%s\n" +
|
|
|
|
|
"%s\n" +
|
|
|
|
|
"Buy: %.5f Sell: %.5f Neutral: %.5f\n" +
|
|
|
|
|
"Forecast: %s -> %.2f\n" +
|
|
|
|
|
"%s",
|
|
|
|
|
m_eraCount, progressLine,
|
|
|
|
|
100 - m_oosSplitPct, dForecast,
|
|
|
|
|
dError,
|
|
|
|
|
Net.getRecentAverageError(),
|
|
|
|
|
m_oosSplitPct, dOosForecast, oosErrStr, m_oosSamples,
|
|
|
|
|
oosDirRecall,
|
|
|
|
|
oosLivePrec,
|
|
|
|
|
neuron0, neuron1, neuron2,
|
|
|
|
|
EnumToString(DoubleToSignal(signalValue)), signalValue,
|
|
|
|
|
classLine
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
s = "Invalid neuron count!";
|
|
|
|
|
SetStatusLabel(s);
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::DrawObject(datetime time, double signal, double high, double low)
|
|
|
|
|
{
|
|
|
|
|
double price = 0;
|
|
|
|
|
int arrow = 0;
|
|
|
|
|
color clr = 0;
|
|
|
|
|
ENUM_ARROW_ANCHOR anch = ANCHOR_BOTTOM;
|
|
|
|
|
switch(DoubleToSignal(signal))
|
|
|
|
|
{
|
|
|
|
|
case Buy:
|
|
|
|
|
price = low;
|
|
|
|
|
arrow = 217;
|
|
|
|
|
clr = clrBlue;
|
|
|
|
|
anch = ANCHOR_TOP;
|
|
|
|
|
break;
|
|
|
|
|
case Sell:
|
|
|
|
|
price = high;
|
|
|
|
|
arrow = 218;
|
|
|
|
|
clr = clrRed;
|
|
|
|
|
anch = ANCHOR_BOTTOM;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
if(price == 0 || arrow == 0)
|
|
|
|
|
return;
|
|
|
|
|
string name = SIG_ARROW_PREFIX + TimeToString(time);
|
|
|
|
|
//--- Deliberately NO ObjectFind() pre-check - same fix as AdvanceChartSignalRestore's ObjectCreate
|
|
|
|
|
//--- call (see that method's comment): ObjectFind scans the ENTIRE chart object list, so calling it
|
|
|
|
|
//--- once per drawn arrow makes PruneDirectionalClusters's era-end sweep O(n^2) in the arrow count -
|
|
|
|
|
//--- exactly the pattern that froze the terminal once already (2026-07-26, arrow restore). This
|
|
|
|
|
//--- sweep runs ONCE PER ERA, completely unchunked (no TRAIN_TIME_BUDGET_MS yield), so its cost
|
|
|
|
|
//--- directly stalls the panel/chart for however long it takes - raising the training budget can't
|
|
|
|
|
//--- fix that, only cutting this O(n^2) cost can. ObjectCreate already returns false harmlessly when
|
|
|
|
|
//--- the name exists (ignored below, same as the restore path); re-applying the properties is exactly
|
|
|
|
|
//--- what a refresh does regardless of whether the object is new or already there.
|
|
|
|
|
ObjectCreate(0, name, OBJ_ARROW, 0, time, 0);
|
|
|
|
|
ObjectSetDouble(0, name, OBJPROP_PRICE, price);
|
|
|
|
|
ObjectSetInteger(0, name, OBJPROP_ARROWCODE, arrow);
|
|
|
|
|
ObjectSetInteger(0, name, OBJPROP_COLOR, clr);
|
|
|
|
|
ObjectSetInteger(0, name, OBJPROP_ANCHOR, anch);
|
|
|
|
|
ObjectSetInteger(0, name, OBJPROP_TIMEFRAMES, g_signalsVisible ? OBJ_ALL_PERIODS : OBJ_NO_PERIODS);
|
|
|
|
|
ObjectSetString(0, name, OBJPROP_TOOLTIP, EnumToString(DoubleToSignal(signal)) + " " + DoubleToString(signal, 5));
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::DeleteObject(datetime time)
|
|
|
|
|
{
|
|
|
|
|
string name = SIG_ARROW_PREFIX + TimeToString(time);
|
|
|
|
|
//--- No ObjectFind() pre-check either - same O(n^2)-avoidance reasoning as DrawObject() above.
|
|
|
|
|
//--- ObjectDelete() already returns false (silently, no dialog) when the name doesn't exist.
|
|
|
|
|
ObjectDelete(0, name);
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
//| End-of-era renderer + non-max suppression over m_arrowSignalCache.|
|
|
|
|
|
//| This is the SOLE place directional arrows are drawn during |
|
|
|
|
|
//| training (the scan passes only RECORD predictions into the cache -|
|
|
|
|
|
//| they never draw), so the chart only ever shows the declustered |
|
|
|
|
|
//| set, never the raw mid-era clusters. Walks strictly oldest -> |
|
|
|
|
|
//| newest (bar index high -> low). For each scored bar: |
|
|
|
|
|
//| - Neutral: delete any arrow. |
|
|
|
|
|
//| - Same-direction within m_signalClusterWindow of the previous |
|
|
|
|
|
//| SEEN same-direction bar: suppress (collapses a whole contiguous |
|
|
|
|
|
//| run to its first bar - advance last-seen either way). |
|
|
|
|
|
//| - Cross-direction within the window of the last KEPT opposite |
|
|
|
|
|
//| signal (flicker at one turn zone; real opposite pivots are a |
|
|
|
|
|
//| leg apart): keep only the higher-confidence side. |
|
|
|
|
|
//| - Otherwise: KEEP -> draw the arrow. |
|
|
|
|
|
//| Order-independent by design: pass 2 records IS bars in SHUFFLED |
|
|
|
|
|
//| order, so this cache pass is the only place NMS/rendering can be |
|
|
|
|
|
//| applied correctly. See m_signalClusterWindow. |
|
|
|
|
|
//+------------------------------------------------------------------+
|
|
|
|
|
void CExpertSignalAIBase::PruneDirectionalClusters(int bars)
|
|
|
|
|
{
|
|
|
|
|
if(m_signalClusterWindow <= 0) // NMS off: the passes drew inline, nothing to render here
|
|
|
|
|
return;
|
|
|
|
|
int cacheSize = ArraySize(m_arrowSignalCache);
|
|
|
|
|
if(cacheSize <= 0)
|
|
|
|
|
return;
|
|
|
|
|
int hi = MathMin(bars, cacheSize);
|
|
|
|
|
int lastBuyIdx = -1; // last SEEN bar per direction (same-direction contiguous collapse)
|
|
|
|
|
int lastSellIdx = -1;
|
|
|
|
|
int keptIdx = -1; // last KEPT bar of EITHER direction (cross-direction resolution)
|
|
|
|
|
double keptConf = 0.0;
|
|
|
|
|
ENUM_SIGNAL keptDir = Neutral;
|
|
|
|
|
for(int idx = hi - 1; idx >= 0; idx--) // high index = oldest bar -> iterate forward in time
|
|
|
|
|
{
|
|
|
|
|
double sv = m_arrowSignalCache[idx];
|
|
|
|
|
if(sv == -2.0) // not scored this era: leave whatever's there
|
|
|
|
|
continue;
|
|
|
|
|
datetime t = m_Time.GetData(idx);
|
|
|
|
|
ENUM_SIGNAL sig = DoubleToSignal(sv);
|
|
|
|
|
if(sig != Buy && sig != Sell) // scored Neutral: ensure no arrow
|
|
|
|
|
{
|
|
|
|
|
DeleteObject(t);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
// 1) Same-direction contiguous collapse: suppress if within the window of the previous SEEN
|
|
|
|
|
// same-direction bar; advance last-seen either way so a whole run collapses to its first bar.
|
|
|
|
|
int lastSame = (sig == Buy) ? lastBuyIdx : lastSellIdx;
|
|
|
|
|
bool sameContinuation = (lastSame >= 0 && (lastSame - idx) <= m_signalClusterWindow);
|
|
|
|
|
if(sig == Buy)
|
|
|
|
|
lastBuyIdx = idx;
|
|
|
|
|
else
|
|
|
|
|
lastSellIdx = idx;
|
|
|
|
|
if(sameContinuation)
|
|
|
|
|
{
|
|
|
|
|
DeleteObject(t);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
// 2) Cross-direction resolution: a fresh cluster within the window of the last KEPT opposite
|
|
|
|
|
// signal is flicker at one turn zone (real opposite pivots are a whole leg apart) - keep only the
|
|
|
|
|
// higher-confidence side. Confidence = |signed signal| = the winning softmax probability.
|
|
|
|
|
double conf = MathAbs(sv);
|
|
|
|
|
if(keptIdx >= 0 && keptDir != sig && (keptIdx - idx) <= m_signalClusterWindow)
|
|
|
|
|
{
|
|
|
|
|
if(conf > keptConf)
|
|
|
|
|
DeleteObject(m_Time.GetData(keptIdx)); // this bar is stronger: drop the weaker opposite arrow
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
DeleteObject(t); // the kept opposite is stronger: suppress this bar
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
}
|
2026-08-10 14:26:12 -04:00
|
|
|
// 3) ALTERNATION, identical to NmsLiveAccept's rule 3 and pass 3's. MUST match both: this is the
|
|
|
|
|
// drawn history, and an arrow set that does not obey the same rule as the traded set shows the
|
|
|
|
|
// user calls the EA would never have taken.
|
|
|
|
|
if(BothDirectionsTradeable() && keptIdx >= 0 && keptDir == sig)
|
|
|
|
|
{
|
|
|
|
|
DeleteObject(t);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
// KEEP: this is the render step - the arrow is drawn here, not by the scan passes.
|
|
|
|
|
DrawObject(t, sv, m_High.GetData(idx), m_Low.GetData(idx));
|
|
|
|
|
keptIdx = idx; // first bar of a fresh, window-clear cluster
|
|
|
|
|
keptConf = conf;
|
|
|
|
|
keptDir = sig;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
//+------------------------------------------------------------------+
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
//| PurgeChart - Removes this EA's own visual objects from the chart. |
|
|
|
|
|
//| Returns how many signal arrows it actually removed. |
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
//+------------------------------------------------------------------+
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
int CExpertSignalAIBase::PurgeChart(void)
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
{
|
|
|
|
|
long chartID = 0;
|
|
|
|
|
//--- Delete ONLY what this EA created: our namespaced signal arrows plus the status-label objects. A
|
|
|
|
|
//--- blanket ObjectsDeleteAll(chartID) here (the old behavior) also wiped the user's own manual chart
|
|
|
|
|
//--- drawings and any other indicator's objects - unacceptable on a client's chart. Called from the
|
|
|
|
|
//--- destructor (clean removal); NOT from InitIndicators anymore, so arrows survive re-inits.
|
fix: purge every EA object namespace on init and after deinit teardown
Leftover objects survived deinit because the cleanup list had drifted.
PurgeChart()'s own comment said it removed "our namespaced signal arrows
plus the status-label objects" while the code removed arrows ONLY, and
the panel prefix was swept at OnInit and nowhere else - so an ordinary
deinit left the status line, and any panel straggler, on the chart.
Three scattered call sites and a comment cannot be kept in step. There is
now ONE list - WarriorChartPrefixes() - covering arrows, status label and
panel, and one sweep, WarriorPurgeChartObjects(), used by every path.
Add a prefix there when a new object family appears and every cleanup
picks it up.
Two call sites added:
OnInit, before ANYTHING is drawn (including the status label it would
otherwise delete). Chart objects live in the chart PROFILE, not in the
EA, so they outlive the process: a deinit force-terminated at
MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5
replaced while attached all strand objects no later deinit will ever
own - and deleting the EA's files does not remove them, which is why
they read as corruption. Arrows are included: LoadChartSignals restores
them from their sidecar moments later and already opens with its own
arrow sweep, so this only removes orphans the sidecar does not account
for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds
that sidecar by scanning the chart.
OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control
tree and ClearStatusLabel clears text rather than guaranteeing object
removal; either can leave a straggler and nothing looked afterwards.
Bounded work - three prefix deletes and one object-list scan - so it
respects the ordering rule that keeps the cheap visible cleanup ahead
of the heavy save. Arrows excluded: ShutdownChartCleanup already
persisted and removed them and re-deleting would race that write.
The two are complementary: the deinit sweep closes the ordinary case, the
OnInit purge closes the case where MetaTrader never let us finish. Only
the second can help after a starved shutdown.
Both sweeps rescan by name across EVERY object type and delete what the
bulk call missed. ObjectsDeleteAll's return has already been observed
disagreeing with a by-name scan of the same chart microseconds apart, and
object commands are queued on the chart rather than applied inline, so a
returned count is not evidence the objects are gone.
Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so
the name cannot drift away from the list that cleans it up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
|
|
|
//--- Arrows are counted separately from the rest because the caller's report is about arrows, and
|
|
|
|
|
//--- because that count is compared against SaveChartSignals' own - two numbers that only mean
|
|
|
|
|
//--- something side by side.
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
int removed = ObjectsDeleteAll(chartID, SIG_ARROW_PREFIX);
|
|
|
|
|
if(removed < 0)
|
|
|
|
|
removed = 0;
|
fix: purge every EA object namespace on init and after deinit teardown
Leftover objects survived deinit because the cleanup list had drifted.
PurgeChart()'s own comment said it removed "our namespaced signal arrows
plus the status-label objects" while the code removed arrows ONLY, and
the panel prefix was swept at OnInit and nowhere else - so an ordinary
deinit left the status line, and any panel straggler, on the chart.
Three scattered call sites and a comment cannot be kept in step. There is
now ONE list - WarriorChartPrefixes() - covering arrows, status label and
panel, and one sweep, WarriorPurgeChartObjects(), used by every path.
Add a prefix there when a new object family appears and every cleanup
picks it up.
Two call sites added:
OnInit, before ANYTHING is drawn (including the status label it would
otherwise delete). Chart objects live in the chart PROFILE, not in the
EA, so they outlive the process: a deinit force-terminated at
MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5
replaced while attached all strand objects no later deinit will ever
own - and deleting the EA's files does not remove them, which is why
they read as corruption. Arrows are included: LoadChartSignals restores
them from their sidecar moments later and already opens with its own
arrow sweep, so this only removes orphans the sidecar does not account
for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds
that sidecar by scanning the chart.
OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control
tree and ClearStatusLabel clears text rather than guaranteeing object
removal; either can leave a straggler and nothing looked afterwards.
Bounded work - three prefix deletes and one object-list scan - so it
respects the ordering rule that keeps the cheap visible cleanup ahead
of the heavy save. Arrows excluded: ShutdownChartCleanup already
persisted and removed them and re-deleting would race that write.
The two are complementary: the deinit sweep closes the ordinary case, the
OnInit purge closes the case where MetaTrader never let us finish. Only
the second can help after a starved shutdown.
Both sweeps rescan by name across EVERY object type and delete what the
bulk call missed. ObjectsDeleteAll's return has already been observed
disagreeing with a by-name scan of the same chart microseconds apart, and
object commands are queued on the chart rather than applied inline, so a
returned count is not evidence the objects are gone.
Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so
the name cannot drift away from the list that cleans it up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
|
|
|
//--- ...and now everything ELSE this EA owns, through the one shared list (WarriorChartPrefixes). This
|
|
|
|
|
//--- function's comment used to claim it removed "arrows plus the status-label objects" while the code
|
|
|
|
|
//--- removed arrows only, and the panel prefix was swept at OnInit and nowhere else - so a normal deinit
|
|
|
|
|
//--- left the status line on the chart. That is the reported "leftover objects on deinit".
|
|
|
|
|
int otherLeft = 0;
|
|
|
|
|
int otherRemoved = WarriorPurgeChartObjects(chartID, true, otherLeft);
|
|
|
|
|
if(otherRemoved > 0 || otherLeft > 0)
|
|
|
|
|
PrintVerbose(ID + ": purged " + IntegerToString(otherRemoved) +
|
|
|
|
|
" non-arrow EA object(s) (status line / panel)" +
|
|
|
|
|
(otherLeft > 0 ? " - " + IntegerToString(otherLeft) + " needed a by-name delete after the bulk call" : ""));
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
//--- VERIFY, don't assume. This one call was the entire cleanup, its return value was discarded, and
|
|
|
|
|
//--- nothing downstream ever looked at the chart again - so "the arrows are still there after removal"
|
|
|
|
|
//--- was indistinguishable from "the arrows were never there", which is exactly why that report survived
|
|
|
|
|
//--- three sessions. The sweep below is a typed scan (OBJ_ARROW only, so it walks a handful of objects on
|
|
|
|
|
//--- a normal chart, not the whole object list) and finds nothing whenever the bulk delete did its job,
|
|
|
|
|
//--- which is the overwhelmingly common case. When it DOES find something, it finishes the job and says
|
|
|
|
|
//--- so - naming the failure instead of leaving it to be re-reported as a visual symptom.
|
|
|
|
|
//--- Names are collected before any deletion: deleting while enumerating by index would renumber the very
|
|
|
|
|
//--- list being walked, and object commands are queued on the chart rather than applied inline.
|
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
|
|
|
//--- WIDENED 2026-08-09. The sweep used to filter on OBJ_ARROW, matching how the arrows are created -
|
|
|
|
|
//--- and that is exactly why it could not explain the run that prompted this: PAI and LSTM both logged
|
|
|
|
|
//--- "persisted 30/10 arrow(s) ... and cleared 0 from the chart", i.e. SaveChartSignals counted objects
|
|
|
|
|
//--- by prefix microseconds earlier and this function then found nothing to delete, with the WARNING
|
|
|
|
|
//--- below staying silent because its rescan was blind in the same way the bulk delete was. Two scans
|
|
|
|
|
//--- disagreeing about the same chart is a fact about the chart, not about arrows, so the rescan now
|
|
|
|
|
//--- walks EVERY object type. If the objects are there under some other type, this both reports it and
|
|
|
|
|
//--- finishes the job; if the chart really is empty, the counts below say so and the discrepancy is in
|
|
|
|
|
//--- SaveChartSignals' count instead. Either way one run settles it rather than another guess.
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
string leftovers[];
|
|
|
|
|
int found = 0;
|
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
|
|
|
int objectsTotal = ObjectsTotal(chartID, -1, -1);
|
|
|
|
|
int prefixMatches = 0;
|
|
|
|
|
if(objectsTotal > 0)
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
{
|
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
|
|
|
ArrayResize(leftovers, objectsTotal);
|
|
|
|
|
for(int i = 0; i < objectsTotal; i++)
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
{
|
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
|
|
|
string nm = ObjectName(chartID, i, -1, -1);
|
|
|
|
|
if(StringFind(nm, SIG_ARROW_PREFIX) != 0)
|
|
|
|
|
continue;
|
|
|
|
|
prefixMatches++;
|
|
|
|
|
leftovers[found++] = nm;
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
for(int i = 0; i < found; i++)
|
|
|
|
|
ObjectDelete(chartID, leftovers[i]);
|
|
|
|
|
if(found > 0)
|
|
|
|
|
Print(ID + ": WARNING - ObjectsDeleteAll(\"" + SIG_ARROW_PREFIX + "\") reported " + IntegerToString(removed) +
|
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
|
|
|
" removed but left " + IntegerToString(found) + " signal object(s) on the chart; deleted them by name. " +
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
"The bulk prefix delete is not doing its job on this terminal build - that is the root cause of any " +
|
|
|
|
|
"'arrows stay on the chart' report, not the shutdown ordering.");
|
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
|
|
|
//--- The both-zero case is the one that needs saying out loud: it means the chart carried no objects
|
|
|
|
|
//--- under our prefix at the moment of the purge, so an "arrows are still on screen" report cannot be
|
|
|
|
|
//--- this function's doing and m_lastArrowsSaved must have been counted from somewhere else (the
|
|
|
|
|
//--- deferred-restore queue is the one such source - see SaveChartSignals). Only printed when the save
|
|
|
|
|
//--- claimed arrows, so a genuinely empty chart stays quiet.
|
|
|
|
|
if(removed == 0 && found == 0 && m_lastArrowsSaved > 0 && !m_purgeMismatchWarned)
|
|
|
|
|
{
|
|
|
|
|
m_purgeMismatchWarned = true;
|
|
|
|
|
Print(ID + ": WARNING - the sidecar was just written with " + IntegerToString(m_lastArrowsSaved) +
|
|
|
|
|
" arrow(s) but the chart holds no \"" + SIG_ARROW_PREFIX + "\" object at all (" +
|
|
|
|
|
IntegerToString(objectsTotal) + " objects of any kind, " + IntegerToString(prefixMatches) +
|
|
|
|
|
" matching the prefix). The saved count did NOT come from the chart, so anything still drawn "
|
|
|
|
|
"was not created under this prefix and no purge here can remove it.");
|
|
|
|
|
}
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
ClearStatusLabel();
|
fix(chart): arrows survived the EA that drew them - persist, then clear
Reported: on deinit the panel and status label go, the signal arrows stay.
Two independent causes, both fixed here.
1. It was partly deliberate. ShutdownChartCleanup carried a second
behaviour selected by a `preserveChartArrows` flag derived from the
deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the
arrows were left on the chart on purpose, to avoid a reload flicker.
That branch IS the reported symptom, an operator cannot tell it apart
from a cleanup that failed, and it was outright wrong whenever the
reload changed the config - REASON_PARAMETERS means exactly that, and
the preserved arrows then belonged to a model the chart no longer
runs, with nothing marking them stale. It is gone, along with the flag
and m_purgeChartOnDestruct. One path now: persist, clear, restore on
the next attach.
2. Whatever remains was unfalsifiable. PurgeChart was a single
ObjectsDeleteAll(prefix) whose return value was discarded, with no
caller ever looking at the chart again - so "the arrows are still
there" and "the arrows were never there" produced identical evidence,
which is why the report survived three sessions. It now verifies:
after the bulk delete it walks the OBJ_ARROW-typed list (a handful of
objects, not the whole chart), deletes any surviving WarSig_ by name,
and says so. Costs one typed scan when the bulk delete works, which is
the normal case; names the root cause when it does not.
Every failure mode of SaveChartSignals was also silent - it returned void
and had three bare early returns. It returns bool now, logs the open
error with the filename, and the shutdown purge is CONDITIONAL on it: for
a converged model the chart objects are the only copy of its signal
history (nothing redraws them - the renderer runs per training era and a
deployed model has none left), so a chart left littered because the disk
write failed beats a clean chart bought by destroying the history. Either
way the log now says which happened.
Also states the user's rule once, where arrows come back rather than
across InitNeuralNetwork's several exits: no weights loaded for this
config => clear the sidecar and start visually clean. A fresh run must
not inherit calls it never made, and the first save would otherwise adopt
them (the sidecar is rebuilt by scanning the chart).
Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
|
|
|
ChartRedraw(chartID);
|
|
|
|
|
return removed + found;
|
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
|
|
|
}
|
|
|
|
|
#endif // WARRIOR_AIBASE_CHARTUI_MQH
|