First live run exposed both problems at once. It logged "+83 release(s)
recorded", and every one of those rows shared a single observed_time up to 30
hours after its event_time: they were the startup backfill, not release-time
observations. Their actual figures are whatever the terminal holds NOW - the
post-revision values this recorder exists to avoid - and the very first batch
proved that is not hypothetical: a Retail Sales row came back previous 3.5 /
revised_prev 3.4, and a Core CPI row already carried revision=1.
Backfill is still worth keeping (a fine snapshot of the revised series, and it
carries the event metadata) but must never be silently mixed with release-time
observations. Every row now records lag_sec and a capture class, so the
distinction cannot be lost by whoever loads the CSV later:
LIVE observed within InpLiveLagSeconds (default 600s) of release
BACKFILL seen long after the fact - MUST NOT be used for surprise research
The log now reports the split per poll and says so explicitly when a poll is
entirely backfill.
Second and worse, in LoadSeen: the FILE_CSV field walk was off by one and keyed
the seen-set on event_id instead of value_id. event_id identifies the event TYPE,
not the release, so after any restart every future release of every event already
in the file would have been skipped - permanently, and silently, exactly for the
recurring high-importance events (NFP, CPI) that matter most. Now reads whole
lines and indexes a split array by a NAME-CHECKED column position, which cannot
drift when the schema changes. Refuses to guess if value_id is absent.
Schema change is handled by rotating any file with a non-matching header to
<name>.<timestamp>.old rather than appending, since mixing layouts mis-parses
every old row.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0c2b025c16