A DATA-INTEGRITY BUG, pre-existing, surfaced by the clearer failure message in
869cd1b putting two identical timestamps next to each other:
13:42:36.584 (EURUSD) AltRawSave: atomic rename raw_EIA_WPSR.csv.savetmp -> ... failed
13:42:36.584 (XTIUSD) AltRawSave: atomic rename raw_EIA_WPSR.csv.savetmp -> ... failed
Same file, same millisecond, two charts, a third winning the race. That is not
reader/writer contention - it is THREE WRITERS on one destination, and
AtomicWriteBegin derived the staging name from the destination alone:
tmpName = finalName + ".savetmp"
So all three opened the SAME temp with FILE_WRITE and wrote it from offset 0 at
once. The published file could be an interleaved mixture of two charts' output,
and the atomic rename publishes that mixture faithfully - the swap guarantees a
reader never sees a HALF-WRITTEN file, and does nothing about a HALF-CORRECT one.
Alt-data is the exposed case: several charts fetch the same series and write the
same Common file.
Keying the temp on symbol+period makes staging private. The rename stays the only
contended operation, and a rename IS atomic, so a loser now publishes nothing
rather than half of itself. It also makes deferred promotion sound for the first
time: the temp promoted later is THIS chart's complete content, never a fragment
of someone else's.
SharedFileCopy.mqh uses the same shape but its destination is agent/terminal-local
and keyed by symbol+fingerprint, so charts cannot collide there. Left alone.
Note the two bugs are independent and both fixes are real. Confirmed in situ at
13:45:45, on the reader/writer one:
CTrainPoolWriter::Publish: atomic rename TrainPool\USDCAD_16388.bin failed (5004)
Warrior: deferred promotion of TrainPool\USDCAD_16388.bin succeeded - the peer
chart that held it has closed it, and the content written earlier is now live
without rewriting the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
REPLACES the in-line retry from ad4ae58, which was the wrong shape and did not
work. Measured after deploying it:
atomic rename ... failed (error 5004) after 4 attempts
MQL5 exposes no FILE_SHARE_DELETE, so a rename CANNOT succeed while any reader
holds the destination open - it is not a lock that waiting longer wins. The
retry assumed a peer holds a pool file for "tens of ms"; USDCAD_16388.bin is
134 MB and a peer reading it holds the handle for SECONDS. The loop lost every
time and bought nothing but 75ms of tick latency on the failure path.
The content is already written and correct - only the SWAP is blocked. So try the
rename once, and on failure remember the temp and promote it from OnTimer, where
I/O belongs. Once the reader closes, a single FileMove lands it. That beats the
old fallback of waiting for the next full publish, which rewrites all 134 MB and
may be an era away.
* pending list is bounded (8) and deduplicated - AtomicWriteBegin reuses one
temp name per file, so a second failure for the same file must not take a
second slot. A full list falls back to the previous next-publish behaviour.
* a successful write FORGETS any queued promotion for that name, so a stale
temp can never overwrite fresher content.
* a vanished temp (a later publish succeeded outright) is dropped, not retried.
* a landed promotion is LOGGED. Silence is what made me misread the last
attempt as working when there had simply been no contention in the window.
Compiled clean; NOT yet run - and note that verification needs a collision to
occur, which happened ~27 times across a whole day. Absence of the message in any
one window is not evidence either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
THE EXIT KNOB. Exit_On_Reversal_Vote (default false) replaces the deleted
Signal_ThresholdClose with one boolean: false pins the close threshold to an
arithmetically unreachable 101, true pins it to the SAME threshold the entry
uses - the seed at first, then the derived value, republished together whenever
it moves. A second threshold was always redundant; "the bot now says the other
way" is one question.
It also arms CExpertSignalCustom::m_holdToBarrier, which was DEAD CODE:
HoldToBarrier(bool) had no caller anywhere in the build, so the flag had been
permanently false and the disabled close threshold was carrying the whole
hold-to-barrier policy alone. Both halves now move together.
Default stays false because the reason is statistical: the gate certifies
P(label agrees | vote fired) against a label that runs to the barrier, so an
early close trades something never measured. Turning it on is a different
strategy, not a tightening of this one.
THE PIN. The live threshold now moves only when an era's weights become the
checkpoint, and freezes once g_ensDeployApproved. Every era still derives its own
rung - that is how the best one is found - but the rung that TRADES belongs to
the checkpoint, exactly as the weights do. Two reasons, one measured and one
structural: the per-era rung moves on 6-34% of steps (the live run flapped
SP500 15 -> 10 -> 15 within a minute of starting), and without the pin a later
era's rung could end up applied to an earlier era's deployed model. A ladder
restart releases the pin, since clearing the checkpoint clears what it pinned.
The era line now prints the rung its own numbers came from, so it stays honest
when that differs from the pinned one.
THE ATOMIC RENAME retried zero times. Six charts share the TrainPool and AltData
directories, so a publish regularly lands while a peer chart holds the
destination open and FileMove returns 5004 - 27 times in one day on the live
fleet. Nothing was lost (the temp keeps the new content, the old file stays
intact) but the row did not update until the next publish. Now four attempts at
25ms, on the FAILURE PATH ONLY - a successful rename never sleeps - and skipped
in the tester, where the contention cannot happen and Sleep would distort a pass.
A rescued retry is logged, so worsening contention is visible.
Retrain-neutral. Compiled clean; NOT yet run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SaveRaw() and RebuildFeatures() hand-rolled the same FileOpen(tmp)->write
->FileClose->FileMove(FILE_REWRITE) swap AtomicWriteBegin/AtomicWriteEnd
already generalize. AtomicWriteBegin hardcoded FILE_BIN (every prior
caller wrote binary payloads); these two write plain ANSI CSV lines via
FileWriteString, so AtomicWriteBegin now takes an optional modeFlags
param (default FILE_BIN, unchanged for the 4 existing callers) and the
two AltDataFetch sites pass FILE_TXT|FILE_ANSI.
The //| box blocks were excluded from 0b06f8e and 5efdb48 and were what
remained: 160 of them ran to 10+ lines, the longest to 88. Compressed to their
leading topic sentences - 5 lines for a function header, 8 for a file header -
keeping the box format and the standard MQL5 name/author lines verbatim.
Verified at the BYTE level this time, across every in-scope file: the list of
non-comment lines is byte-identical to HEAD and braces balance. The first check
compared a locale-decoded 'git show' against a UTF-8 read and flagged 25 files
that had not changed at all - every BOM and every non-ASCII line mismatched.
47,696 -> 40,665 lines in scope; comment share 38% -> 26%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same pass as 0b06f8e, applied file by file: comment runs of 4+ lines compressed
to their leading topic sentences, capped at 4 lines, whole sentences only.
Warning sentences (NEVER / MUST / trap / would-have) survive the budget.
Every file was checked the same way before committing: the list of non-comment
lines is byte-identical to HEAD, and braces balance. No code was touched.
Panel/, Enumerations/ and the already-terse System headers needed little or
nothing - PooledGate, TradeChecks, BinomialStats and Random came through with
no blocks over the threshold at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate's NormalUpperTail was a hand-rolled Abramowitz & Stegun 26.2.17
approximation. Its own comment gave the reason - "drags a chain of headers
behind it" - and that turned out to be one file: Math\Stat\Normal.mqh
includes only Math.mqh, which includes nothing. Swapped for Cody's rational
approximation in the library (~18 significant digits vs |error| < 7.5e-8).
No past verdict changes: at the z the gate operates on, the difference is
orders of magnitude below DEPLOY_FAMILY_WISE_ALPHA.
Adopting it needed the four bare macros in AI\Network.mqh gone first.
"#define b1 AdamBeta1" collides with an identifier in Math.mqh, so the
include would have macro-expanded the library's own local and failed to
compile - the same landmine that made the original author rename the
approximation's coefficients to ntB1..ntB5 rather than use the reference's
b1..b5. lr, b2 and momentum are the same class of hazard: single-token
global macros in a 52k-line codebase. All four now resolve to the input
names they always aliased, which is a pure textual identity - verified zero
bare occurrences remain.
Also:
- SelectionSort over the buffered signals was O(n^2) with an O(n^2) count of
StructToTime calls, because the comparison rebuilt both datetimes from the
six int date fields every time. Now materialises the keys once and does an
insertion sort; ArraySort cannot permute a struct array. IsEarlier goes
with it, MakeDateTime becomes SignalTime.
- Seven FileOpen sites lacked FILE_SHARE_READ|FILE_SHARE_WRITE, including
AtomicWriteBegin, which stages every model save. All 43 sites now carry
them - an exclusive open fails outright when another process holds the
path, which here has meant a silently skipped save.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FileOpen(FILE_WRITE) truncates its target on open. CNet::Save already staged
the .nnw through a temp file + rename for that reason, but the three sidecars
written beside it did not:
.stats ExpertSignalAIBase.mqh:5918
.arrows ExpertSignalAIBase.mqh:6224
.cfg ExpertSignalAIBase.mqh:7329
Two defects followed.
1. An interrupted write published a truncated sidecar. For .cfg that is the
worst case: LoadAndCompareTopologyConfiguration() reads a short file as a
mismatch, which discards the trained model and restarts from era 0.
2. Windows file sharing is a mutual contract - a writer opened with no
FILE_SHARE_* blocks every concurrent open regardless of the reader's flags.
All three read paths carry FILE_SHARE_READ|FILE_SHARE_WRITE specifically so
a tester agent can read them while a live chart runs; an exclusive writer on
the same path defeated that.
Extracted CNet::Save's proven pattern into System\AtomicFile.mqh
(AtomicWriteBegin/AtomicWriteEnd) and routed all four writers through it. This
also encodes the FileMove gotcha once instead of per call site: the destination
location comes from FILE_COMMON inside the 4th arg, NOT inherited from the
source, and getting it wrong moves the file to the wrong sandbox silently.
Also fixed while in these functions:
- SaveTopologyConfiguration had 13 copy-pasted 6-line error blocks that each
returned WITHOUT FileClose(handle), leaking the handle on every write
failure. Collapsed to one ok-chain that closes exactly once. The on-disk
field order and types are unchanged (asserted during the rewrite) so existing
.cfg files still load.
- SaveChartSignals documented that pruning runs only after a successful write
("a failed write above leaves both the file AND the chart untouched") but
never checked any write result, so a partial write still deleted the chart
objects. Results are checked now, making the existing comment true.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
