First live run exposed both problems at once. It logged "+83 release(s)
recorded", and every one of those rows shared a single observed_time up to 30
hours after its event_time: they were the startup backfill, not release-time
observations. Their actual figures are whatever the terminal holds NOW - the
post-revision values this recorder exists to avoid - and the very first batch
proved that is not hypothetical: a Retail Sales row came back previous 3.5 /
revised_prev 3.4, and a Core CPI row already carried revision=1.
Backfill is still worth keeping (a fine snapshot of the revised series, and it
carries the event metadata) but must never be silently mixed with release-time
observations. Every row now records lag_sec and a capture class, so the
distinction cannot be lost by whoever loads the CSV later:
LIVE observed within InpLiveLagSeconds (default 600s) of release
BACKFILL seen long after the fact - MUST NOT be used for surprise research
The log now reports the split per poll and says so explicitly when a poll is
entirely backfill.
Second and worse, in LoadSeen: the FILE_CSV field walk was off by one and keyed
the seen-set on event_id instead of value_id. event_id identifies the event TYPE,
not the release, so after any restart every future release of every event already
in the file would have been skipped - permanently, and silently, exactly for the
recurring high-importance events (NFP, CPI) that matter most. Now reads whole
lines and indexes a split array by a NAME-CHECKED column position, which cannot
drift when the schema changes. Refuses to guess if value_id is absent.
Schema change is handled by rotating any file with a non-matching header to
<name>.<timestamp>.old rather than appending, since mixing layouts mis-parses
every old row.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The calendar is the only non-price source MQL5 carries with real content, and it
is the one this project cannot research from history:
MqlCalendarValue.actual_value returns the CURRENT figure, i.e. after every later
revision. Reading 2019's NFP today returns a number nobody could have known in
2019, so any surprise = actual - forecast feature built from history carries
lookahead - and the flattering kind, since it makes a model look most prescient
exactly on the events that were revised most.
The only sound fix is to write down what the terminal reported at the moment of
release and never touch that row again. Write-once is the entire contract here: a
row is appended the first time a value_id is seen carrying an actual figure, and
is never rewritten, because re-recording on a later poll would silently import
the revision this file exists to avoid. The seen-set is rebuilt from the file on
init so a restart cannot duplicate or re-import either.
Deliberately standalone - no includes from the EA tree, and not wired into
Warrior_EA. It may run for months on a spare chart, and coupling it to the
trading system would mean a refactor there can stop the recorder; a gap in a
write-once series cannot be backfilled by definition. It also keeps a
data-collection task from adding any failure mode to a system about to trade a
prop account.
Records actual/forecast/previous/revised_previous, revision number, impact,
importance, units, plus observation time and the quote at observation. Every
FileOpen carries FILE_SHARE_READ|FILE_SHARE_WRITE per the rule this codebase
learned the hard way (exclusive opens fail 5004 and look like "no data").
Compiles 0 errors, 0 warnings. Value starts at zero and accrues with time, which
is the argument for starting it now rather than when it is wanted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
