First live run exposed both problems at once. It logged "+83 release(s)
recorded", and every one of those rows shared a single observed_time up to 30
hours after its event_time: they were the startup backfill, not release-time
observations. Their actual figures are whatever the terminal holds NOW - the
post-revision values this recorder exists to avoid - and the very first batch
proved that is not hypothetical: a Retail Sales row came back previous 3.5 /
revised_prev 3.4, and a Core CPI row already carried revision=1.
Backfill is still worth keeping (a fine snapshot of the revised series, and it
carries the event metadata) but must never be silently mixed with release-time
observations. Every row now records lag_sec and a capture class, so the
distinction cannot be lost by whoever loads the CSV later:
LIVE observed within InpLiveLagSeconds (default 600s) of release
BACKFILL seen long after the fact - MUST NOT be used for surprise research
The log now reports the split per poll and says so explicitly when a poll is
entirely backfill.
Second and worse, in LoadSeen: the FILE_CSV field walk was off by one and keyed
the seen-set on event_id instead of value_id. event_id identifies the event TYPE,
not the release, so after any restart every future release of every event already
in the file would have been skipped - permanently, and silently, exactly for the
recurring high-importance events (NFP, CPI) that matter most. Now reads whole
lines and indexes a split array by a NAME-CHECKED column position, which cannot
drift when the schema changes. Refuses to guess if value_id is absent.
Schema change is handled by rotating any file with a non-matching header to
<name>.<timestamp>.old rather than appending, since mixing layouts mis-parses
every old row.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The calendar is the only non-price source MQL5 carries with real content, and it
is the one this project cannot research from history:
MqlCalendarValue.actual_value returns the CURRENT figure, i.e. after every later
revision. Reading 2019's NFP today returns a number nobody could have known in
2019, so any surprise = actual - forecast feature built from history carries
lookahead - and the flattering kind, since it makes a model look most prescient
exactly on the events that were revised most.
The only sound fix is to write down what the terminal reported at the moment of
release and never touch that row again. Write-once is the entire contract here: a
row is appended the first time a value_id is seen carrying an actual figure, and
is never rewritten, because re-recording on a later poll would silently import
the revision this file exists to avoid. The seen-set is rebuilt from the file on
init so a restart cannot duplicate or re-import either.
Deliberately standalone - no includes from the EA tree, and not wired into
Warrior_EA. It may run for months on a spare chart, and coupling it to the
trading system would mean a refactor there can stop the recorder; a gap in a
write-once series cannot be backfilled by definition. It also keeps a
data-collection task from adding any failure mode to a system about to trade a
prop account.
Records actual/forecast/previous/revised_previous, revision number, impact,
importance, units, plus observation time and the quote at observation. Every
FileOpen carries FILE_SHARE_READ|FILE_SHARE_WRITE per the rule this codebase
learned the hard way (exclusive opens fail 5004 and look like "no data").
Compiles 0 errors, 0 warnings. Value starts at zero and accrues with time, which
is the argument for starting it now rather than when it is wanted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dumps CopyTicksRange output to CSV in the COMMON files folder, where research/ already
reads its rate exports from. Chunked by hour range because a single unbounded request over
years is both slow and liable to ERR_HISTORY_SMALL_BUFFER; boundaries are half-open on
purpose since CopyTicksRange is inclusive at both ends and adjacent chunks would otherwise
duplicate any tick landing exactly on a split.
Keeps MqlTick.flags RAW rather than decoding to a direction. On FX/CFD only
TICK_FLAG_BID/ASK are ever set - TICK_FLAG_BUY/SELL and volume/volume_real are empty for
Forex - so signed trade direction does not exist in this feed and has to be synthesised
offline from quote dynamics. Exporting a decoded 'side' column would be inventing data.
Written as the reliable alternative to decoding StrategyQuant's .dat: that format's base
record parses cleanly (32 bytes, ms timestamp + bid + ask + one volume, prices x1e6,
verified against a known SP500 level) but the delta stream is a custom bit-aligned
dictionary scheme, and it carries only ONE volume field - so it offers nothing MT5 does not
already provide.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
