Same harness as the D1 screens, forward 30 H4 bars (~5 days), 199 perms, on
the surviving htf mid bars (19.8k-36.4k bars per symbol). Question: does the
wired alt/volume information carry to H4, for the chart-timeframe decision.
Answer: the signal survives but is roughly halved, and the cost side worsens
2.6x per step down. SP500 vix_chg5 clears the family bar with MI|vol 0.0112
(vs 0.031 at D1); volLevel50 (the EA activity feature) is incremental on all
four symbols at H4 - USDJPY family-clean, and it is EURUSD strongest
non-control signal there too. Gold gvz_chg5 stays incremental (0.0038 vs
0.0197 at D1 - a fifth of the strength). COT is null at H4 on FX (weekly
cadence pasted across 30 bars/week dilutes it below detection on EURUSD/JPY;
survives conditionally on SP500).
Cost table (median spread/ATR; the 1.74xATR geometry in spread units):
SP500 138->52->25, EURUSD 282->113->57, USDJPY 230->87->44, XAUUSD 92->34->16
for D1->H4->H1. Every step down multiplies the cost share ~2.6x.
Also fixes the disaggregated-COT column name for gold/WTI in the screen
(M_Money vs Lev_Money - the same crash the EA-side catalog documents).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POWER is the real find of the three: daily temperature -> degree days ->
natural-gas demand is the textbook gas fundamental, numeric and daily. But it
is point data needing construction into a national series (NOAA CPC ships that
ready-made), and its target symbol is not traded yet - fetch code written for
a chart nobody attaches first runs months later, unobserved, which is the
silent-FRED failure shape. Queued for the AvaTrade expansion, not refused.
FIRMS: re-raised, nothing changed since it was parked - point fire detections
behind the same unproven proxy chain. GIBS: imagery tiles, not numbers; our
CONV is 1D and NASA already sells the extracted products.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner decision (stated twice): available data gets wired; the networks judge
usefulness; the deploy gate remains the arbiter of what trades. Implemented:
MACRO block (6) on every symbol: 10y yield 20d change, curve slope, 5y
breakeven 20d change, Fed-ECB policy gap, CPI yoy, unemployment 12m change.
Screened null vs forward range on all four research symbols - recorded as
the honest prior in the catalog comment, wired regardless.
RISK block (3) extended to every symbol (FX majors, metals, energy, BTC all
now carry vix/vix_chg5/usd_chg5).
IVOL pair extended with the level alongside the change.
Vintage integrity kept where it is free: CPI is fetched as CPIAUCNS (NSA,
essentially never revised) so the plain-FRED backfill stays first-print-clean;
yields/curve/breakevens/policy rates are unrevised by nature. UNRATE is the
one exception (seasonal refits, ~0.1-0.2pp) - the EA cannot run the ALFRED
protocol, accepted and documented at the declaration site.
UpdateFred gains a staleDays parameter so the monthly series do not fire a
pointless fetch attempt every hour for three weeks after each print.
FeatureValue now takes the day and does its own as-of lookups - adding a
source no longer widens a parameter list. Feature counts: 12-15 per symbol;
symbol feature-order changed, safe only because no models exist yet.
export.py mirrors the new catalog for the five research symbols (13-15
features), smoke-tested: all five CSVs written, 6,072 daily rows each.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Screened yields (DGS2/DGS10), curve slope, inflation breakevens, Fed policy,
the Fed-ECB policy differential, and monthly US unemployment and CPI - all on
ALFRED first prints, 499 permutations, against forward 5-day range.
NOT ONE macro feature clears the family-wise bar on any symbol. The only thing
that clears anywhere is the trailing-range positive control, which is what it
is there to do. Best a-priori candidate, the Fed-ECB differential on EURUSD,
came in at MI 0.00170 p=0.088 - nothing. The two features flagged INCREMENTAL
(dgs2_chg5 on SP500) have null marginal MI and are isolated conditional cells
at the expected false-positive rate, not findings.
The `distinct` column quantifies the power argument instead of asserting it:
unemployment takes 51-66 distinct values across 3,745-6,159 bars, CPI 174-277,
against 6,159 for a continuous feature. A monthly series pasted onto daily bars
carries about 1% of the resolution, and it showed - the monthly features were
among the weakest in every table.
The contrast with the implied-vol screen is the useful part: the options
market FORWARD-LOOKING view of an instrument (gvz_chg5 on gold, MI|vol 0.0197)
carries real information about its range, while the economy BACKWARD-LOOKING
state carries none. Mismatched timescales - rate levels move over months,
5-day range moves daily.
Also makes load_bars fall back to htf/{SYM}_D1_mid.npz when the tick-derived
build is absent (the 2026-08-16 disk cleanup removed bars/ but htf/ survived),
with need_ticks=True turning that fallback into a loud failure for the
order-flow screen rather than silently testing flow features on OHLC data.
No EA change: nothing survived to wire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the screen_ivol survivors (41d726c). New per-symbol `ivolSeries` in the
catalog feeds a generic `ivol_chg5` feature from whichever CBOE vol index the
instrument owns, so one code path serves every symbol:
XAUUSD + ivol_chg5 (GVZ) - MI|vol 0.01971 p=0.002, 3.6x the positive
control and 4.6x the vix_chg5 gold had alone.
vix_chg5 KEPT: this appends, it does not replace.
EURUSD + vix_chg5 - screened, incremental p<=0.006, and its first
real feature ever (it had only exploratory EIA).
USDJPY + vix_chg5 - screened, incremental.
NAS100 / US30 / US2000 + ivol_chg5 (VXN / VXD / RVX) - exploratory by analogy.
XTIUSD / XBRUSD + ivol_chg5 (OVX) - exploratory, no oil bars to screen yet.
SP500 unchanged - its features already screened clean and VXN/VIX3M edging
out VIX is a correlated within-family best-of-N, not a real ranking.
On EURUSD/USDJPY the screen put VXD marginally above VIX, but they are
near-duplicates and the gap sits inside the noise, so the tie is broken by a
rule rather than by the number: take the series already in the fetch path.
Also fixes a real collector bug: fetch_vintaged built ALFRED realtime windows
out to 2028, and FRED rejects realtime_end after today - so every REVISED
series (unemployment, CPI, GDP: exactly the ones needing the vintage path) was
unreachable, while unrevised series never noticed because they bail earlier.
UNRATE and CPIAUCSL now return first prints correctly.
Adds screen_macro.py (rates, curve, breakevens, Fed/ECB policy differential,
plus monthly country stats) with a `distinct` column that reports the honest
effective sample size - a monthly series pasted onto D1 bars is a step
function, and that column is what decides whether it can clear a gate at all.
Not yet run: the Market Data bars directory is being regenerated right now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No free historical GEX exists: probed the CBOE chain endpoint with date/dt query
params (both silently ignored, returned today) and dated/historical paths (403),
and the CBOE index-history CSVs are 403 too. The forward recorder stays the only
path to GEX history.
But the options market publishes its per-instrument view of future range as the
CBOE vol indices, and FRED carries the whole family free with 15-25 years of
history - screenable today with the existing collector and harness. Fetched
GVZ (gold), OVX (oil), VXN, VXD, RVX, VIX3M.
HEADLINE - XAUUSD: gvz_chg5 (gold IV 5-day change) MI 0.02103, MI|vol 0.01971,
p=0.002. That is 3.6x the trailing-range positive control and 4.6x the vix_chg5
this project currently ships on gold - the second-largest incremental MI of the
whole campaign, on a symbol that carries exactly one screened feature today.
Vol-change is incremental on all four symbols: SP500 (known), USDJPY vxd_chg5
0.00492, and EURUSD vxd_chg5 0.00412 / vix_chg5 0.00379 - notable because
EURUSD has no screened features at all and its own trailing range is a weak
control there, so external vol carries information its own history does not.
Caveats recorded in the script and memory: SP500 within-family ordering
(VXN > VIX3M > VIX, all ~0.031-0.038 conditional) is a best-of-N artifact and
must not be cherry-picked; XAUUSD noise control misbehaved this run (MI|vol
0.00271 p=0.002), so anything under ~0.003 conditional on gold is unresolved -
gvz_chg5 at 7x that floor is unaffected; EVZ (euro IV) is DISCONTINUED since
2025-03 and must never be wired.
Nothing wired - the EA is mid-deploy and this would re-key every model again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spent 3 of 5 daily requests. All three were informative:
ETF data (SPY/QQQ/IWM) requires Basic - free tier is single stocks only.
Full-chain GEX (all expirations) requires Growth - free and Basic must
query one expiration per request, so even with history a full-chain
backfill would be 24-54 requests per day of history.
AAPL?expiration=2026-09-18 returned 200 with the right schema but a nearly
empty payload: 13 of 93 strikes carried any open interest, total call OI
4,296 against CBOE 373,253 for the same expiry, put OI zero, and every
near-the-money strike blank.
So the construction could not be validated - not because the math disagreed
but because there was nothing to compare against. From outside it is not
possible to tell free-tier degradation from their flow-signed methodology,
and finding out costs $1,499/month.
Verdict hardens: the free CBOE CDN is strictly better than Basic for this
project - complete chains, every expiry and strike, gamma and open interest
populated, unlimited, $0. Our own AAPL figures were internally coherent
(+0.929 Bn/1% total, Sep-18 expiry +0.154 Bn, near-money gammas 0.013-0.019).
GEX stays externally unvalidated; if that ever matters, use a different vendor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pricing checked: Free $0 (5/day), Basic $79 (250/day), Growth $299 (2,500/day),
Alpha $1,499 (unlimited) - and the Historical API is ALPHA-EXCLUSIVE. Basic and
Growth serve live data only.
The archive was the only thing worth buying from this vendor, so nothing in
budget helps: Basic would spend $79/month to make a once-a-day snapshot 15
seconds fresh instead of 15 minutes. Not subscribing.
The free key keeps one genuine use: a single live call to compare their GEX
against our CBOE-computed number, validating the recorder formula against a
commercial implementation (sign and magnitude only - they sign strikes from
classified tape, we use the standard open-interest assumption).
Recorded the EV argument for future sessions: the recorder banks this history
for free in ~12 months, and on this project base rate most alt-data families
die at the incremental gate. Paying four figures to test GEX a year early is a
poor trade. If revisited, price bulk ARCHIVE sellers, not analytics APIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Option open interest is a snapshot source - no free history exists anywhere -
so the series only accrues from the day recording starts. That is why this
ships BEFORE the redeploy: every day the EA is not running is a day of history
that cannot be recovered later.
Records one row per weekday after 21:00 UTC to gex_{CANONICAL}.csv: net/call/put
dollar GEX per 1% move, call and put OI, the three nearest expiries and the
front expiry code. Feeds NOTHING - wiring a feature that is missing across ~100%
of the training sample would waste input width and hand batch-norm a constant.
It becomes a screening candidate at ~250 rows, gated like every other feature.
Thesis: dealer gamma is a RANGE mechanism (long gamma -> hedging sells rallies
and buys dips, range compresses; short gamma amplifies both ways), and range is
this project's one proven channel.
Verified in situ against the live SPX chain before writing any MQL5: 29,362
contracts, 20,993 with nonzero gamma, 54 expiries, total +90.7 Bn/1% (calls
+305.7, puts -215.0), and 100% of net GEX inside 5% of spot. The CDN publishes
per-contract gamma directly, so no pricing model - and no model risk - enters
the recorded data. Also verified the CDN does NOT gate on User-Agent (the old
"CBOE is UA-gated" note in DESIGN.md was a different CBOE path), so plain
WebRequest reaches it.
Dropped a zero-gamma "flip level" field: the probe returned a crossing above
spot while total GEX was strongly positive, which is incoherent - a static
gamma snapshot cannot give a flip level without repricing. Recording a
plausible-looking wrong number is worse than recording nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EIA (user directive: "the NN might find patterns in it for both oil and regular
symbols"). Weekly Petroleum Status Report via the v2 API - crude stocks ex-SPR,
field production, refinery utilization - three features (1y percentile, 4w
change, utilization) on EVERY catalog symbol, not just oil. EIA screened NULL on
WTI's short 7y sample, so these ship as EXPLORATORY inputs: the deploy gate, not
the screen, decides whether a model trained on them trades. Publication stamp
observed+6d mirrors research/altdata/eia.py.
Symbol handling was hardcoded to three if-blocks; it is now a catalog of 24
instruments x alias lists covering The5ers/FTMO/AvaTrade/Dukascopy/OANDA/IC
Markets naming, with prefix matching for the broker suffix zoo (US500.cash,
XAUUSDm, EURUSD.r). Adding an instrument is one AddSpec row. COT caches are
named by CANONICAL so two brokers' names for one contract share a download.
Unrecognised symbol -> a chart dialog (Panel\AltDataMapDialog.mqh, CAppDialog +
dropdown) asks which instrument it is; the answer persists in symbol_map.cfg and
"No alternative data" is a recorded choice, not a nag. Non-blocking by design:
an unmapped symbol contributes 0 features and must never hold up a chart.
Also: UrlEncodePart now escapes '%' - SoQL like-predicates use it as the
wildcard and an unescaped one corrupts the query; docs/ gains the whitelist
URLs, an API-key backup, and the catalog reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VPIN-style toxicity (|imbalance|): null-to-marginal everywhere. But tick
ACTIVITY (count vs 20d mean) clears the family bar on ALL FOUR symbols for
forward range AND survives conditioning on trailing realized range
(SP500 MI|vol 0.025, XAUUSD 0.0088, USDJPY 0.0069, EURUSD 0.0049, all
p<=0.006). On EURUSD it beats the trailing-range positive control itself -
resolving the void-control anomaly: EURUSD D1 range IS predictable, just
not by its own trailing range. USDJPY spread_stress (max/mean) also
family-clean + incremental. Direction: nothing beyond the known SP500
leverage effect. Validates the EA's volume feature block for the RANGE
objective the tuner now optimizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
12 features (5 EIA petroleum + 3 COT managed-money + VIX/USD + controls)
vs forward 5-bar range and direction on 1,789 D1 bars resampled from the
decoded XTIUSD M1 file, 499 circular-shift perms. No feature clears the
family-wise bar on either target; every EIA fundamental is null even
marginally (best p=0.13). Sample is short (~7y) so a weak effect is not
excluded - but per the gate, no EIA feature ships. The EIA key stays in
keys.txt for future use (longer history / recorded surprise-vs-consensus).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MI|vol column = I(X; target | trailing-range tercile), same circular-shift
null. Range target 499 perms: SP500 vix_chg5 survives conditioning at 0.031
(3x trailing range's own within-tercile residual); VIX LEVEL emerges
conditionally (variance-risk-premium structure); USDJPY COT family survives.
Direction target: SP500 vol/VIX-chg clear marginally (equity leverage
effect) but drop to p~0.05-0.06 conditional = redundant with price vol;
USDJPY/XAUUSD/EURUSD direction null across all alt features.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- fred.py: ALFRED output_type=4 first prints, chunked realtime windows
(2000-vintage cap), unrevised-series fallback (published=observed+1d);
NFCI excluded from features (revised, no vintage archive)
- eia.py: 4 weekly petroleum series on disk (1982->now)
- screen.py: as-of joined alt features vs forward 5-bar range/ATR on D1,
3x3 MI, circular-shift null, family-wise max bar, +/- controls
First readings (199 perms): SP500 vix_chg5 MI 0.047 (1.5x the positive
control) + usd_chg5 clear family bar; USDJPY 4 COT positioning features
clear family bar BEATING the positive control; XAUUSD vix_chg5 tops control
but sub-family-bar; EURUSD positive control FAILS -> table void per the
excursion-target rule, needs investigation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15, ~6 minutes after attach: PAI-8fea (fractal target, 37
features, D1) converged at era 299 and the plateau deploy CLEARED the
family-wise gate for the first time in project history: dir-precision
73.1% vs 63% break-even, +10.4pp on 350 test calls = 4.04 sigma,
p_family = 0.0081.
This script asks the first two hostile questions offline:
- DRIFT: always-long at the same 2.64/1.66 geometry scores 64.1% on the
last 15% of D1 history (66.7% on 30%) - drift alone clears BE by
~1-3pp, but the model is +9pp above ALWAYS-LONG, so the pass is
selection, not drift.
- SWAP EXPOSURE: median 7-8 bars to the long target = ~10 nights of
financing ~ 0.15-0.2% notional vs a ~1.7% target -> a ~1-1.5pp BE
haircut against a +10.4pp margin. Survives.
Remaining before belief: replication on other D1 symbols, and closing
the live-semantics gap (certified wins assume hold-to-barrier; live
exit paths can cut on vote flips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User request: NNs predicting pivots (fractals for label density). Run on
the offline stack that demonstrably CAN learn (+2.6pp XAUUSD meta), free
of every historical in-EA training bug: pooled 4-symbol training,
per-symbol norm, scale-free causal features, target = side of current
price the next confirmed Bill Williams fractal lands on, real M1
ask/bid fills, threshold fitted on calib only, pre-registered 2-sigma
net-expectancy gate.
Result: train CE 0.682 (a whisper below the 0.693 coin), and on test no
symbol passes - EURUSD/USDJPY net zero, XAUUSD gross +0.22 pts vs a
larger spread (net -0.30), SP500 net +0.05 +/- 0.45. The gross-positive
tails are index drift plus sub-spread micro-reversion - the tick-flow
decay shape at swing scale. Seventh independent measurement of the same
fact: entry-time direction information is not in these features.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Faithful Python port of ADZigZag (stock MetaQuotes ZigZag 12/5/3,
verbatim rebrand) including the incremental prev_calculated branch, so
the indicator can be replayed bar by bar exactly as it draws live.
SP500 H1, 74,599 bars, fills at real M1 ask/bid:
- FINAL swings: 4,658 legs, mean 50.8 pts = 95 spreads. Perfect
foresight +50.2 pts/leg. The user premise (swings dwarf spread) is
fully confirmed.
- LIVE: 81% of drawn newest-pivots later repaint away entirely
(18,805 of 23,299). Holding the drawn direction at every bar close
grosses +0.38 pts/trade (t=0.8, zero cost charged) out of the
50.8-pt average swing - 0.7% of the line the chart ends up showing.
- LONG +1.66 gross / SHORT -0.90 = the index drift, nothing else;
long net +1.17 pts / 13-bar hold = ~1.5 bp, under one night financing.
The spread subtracts 0.49 pts of a swing that hands over 0.38: cost was
never the obstacle - pivot knowledge is.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User claim: swings dwarf the spread, so cost cannot be what blocks swing
trading at 1:2/1:3 RR. Measured on the validated M1 bid/ask book, SP500
H1, ATR-scaled causal ZigZag at 4 reversal thresholds:
- The premise is CONFIRMED: median swing 36-79 spreads, mean up to 110.
Perfect-foresight expectancy +28 to +59 pts/leg.
- The conclusion does not follow: trading every confirmed leg (enter on
the ZigZag confirmation close, real ask/bid fills, exit on the next
confirmation) grosses -0.2 to -0.5 pts/leg AT ZERO COST, on 3,681 to
13,871 legs. The confirmation retracement - the event that DEFINES a
pivot - consumes the entire swing before the spread is even charged.
- Long/short split is symmetric around the index drift (LONG +1.18,
SHORT -2.17 gross at 3xATR), i.e. no swing structure beyond drift.
- 3.0xATR reversal reproduces the EA ZigZag cadence exactly (median leg
17 bars, 49 legs/1000 bars vs the EA measured 17 and 44).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three additions to meta_pool.py, in the order the campaign needed them:
- memmap + float32-throughout (per-batch float64 cast): the 6.5 GB
4-symbol corpus OOMed the float64 pipeline on the training box;
- pool2: per-symbol standardization (each symbol by its own train-slice
mu/sd) + 64/32 capacity + l2 1e-3, after the naive pooled model
underfit to the prior (train CE pinned at base-rate entropy);
- curve: fixed-ladder precision-vs-threshold on calib and test side by
side - the dose-response diagnostic that closed the question.
RESULT recorded in memory: pooling transfers real skill (XAUUSD +2.6pp,
SP500 +1.2pp at fitted thresholds, >>2 sigma) but 0/8 fitted operating
points clear break-even, and the high-conviction tail is temporally
unstable - the precision-vs-threshold slope FLIPS SIGN between calib and
test on 3 of 4 symbols, so no ex-ante threshold rule exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Loads the EA's MetaExport .f32 datasets (UTF-16 sidecars), applies the EA's
own discipline offline: chronological 55/15/30 split with horizon-length
purges, operating point fitted on the calibration slice only via
coverage x (precision - BE) with the 25% floor, test slice touched once,
deployability at the 2-sigma edge floor. Small leaky-ReLU MLP + Adam in
numpy; `stats` / `eval <tag>` / `pool` commands.
First run on XAUUSD_16388 validated the plumbing and exposed the data:
the 2.5h gold tester run only covered 2004-07..2006-10 (3,214 candidates)
because gold tick volume is huge - and corpus builds do not need ticks at
all (journaling is bar-open-keyed, labels come from bar history later), so
"Open prices only" modeling builds the same corpus ~100x faster.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
seasonal.py gets a frame-based entry (analyse_frame) and a reusable
report() so the identical statistics - circular-rotation family-wise
null, max-|t| bar, split-half - can run on instruments whose book is
synthesised from M1 bars. breadth_seasonal.py runs it on the five
SQX-decoded instruments (FTSE100, UK100, WTI x2 feeds, USDCAD) that
share no data path with the four originals; the duplicate-market pairs
(FTSE100/UK100, WTI_d/WTI_5) double as replication checks. Caveats
stated in the module docstring: synthesised flat spread (move/spread
is approximate, no intraday spread shape) and file-time clock labels;
drift/t columns are spread-free and unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the 2026-08-09 training audit. FORCES A RETRAIN of every
Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be
redeployed alongside the .ex5 - they carry new exports.
F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online
SGD (one weight update per bar), which is the mechanical source of the
era-to-era whipsaw every downstream guard was built to cope with. The O(n^2)
outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv /
AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the
optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so
there is one Adam/SGD implementation instead of four that can drift.
- the LSTM needs no outer-product kernel (WeightsGradient already holds the
sample's full dW) but could NOT simply be left un-zeroed between samples:
CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a
separate accumulator plus an elementwise add.
- batch-norm gamma/beta accumulate in host arrays, not new BatchOptions
slots - BN_OPT_STRIDE is baked into every persisted .nnw.
- scoped to pass 2; online learning keeps immediate updates. Every save /
checkpoint / scoring boundary flushes, scaling by the real sample count.
- degrades to per-sample updates (one log line) on a tier that cannot
accumulate, so old devices and DLL-free builds are unaffected.
- verified offline: DirectML/batch_accum_check.cpp drives the real exports
against an independent reference; at B=1 the accumulator matches the
shipped unbatched kernel's own gradient to 1.1e-16. Math only - the
in-situ check remains the per-layer dW/W report on a real era.
F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a
conv/LSTM front end had already reduced it, so an LSTM's dense stack was
charged for 1,280 inputs when it receives 64. Confirmed from the deployed
.cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now
budgeted against the front-end output and capped at it (never fan out), with
the derivation reordered so both stages settle first.
N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing
direction and Wyckoff stage into one scalar across a sign discontinuity. Split
into direction + [0,1] magnitude, the same convention the base OHLC block uses.
Information-preserving; 13 readings now occupy 16 inputs.
Compiled clean (0 errors, 0 warnings); both DLLs rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four of the six findings from research/training_pipeline_audit_2026-08-09.md
(F4 mini-batching and F6 feature re-encode deliberately deferred - see the
report's implementation-status section for why):
- F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%,
which is 15-bit - provably non-uniform on every full-history era over 32,768
queued samples. New 30-bit ShuffleRandomIndex().
- F2: plateau warm restarts were a no-op whenever eta already sat at its
ceiling (the normal state of a non-regressing plateau) - the ladder was just
a 24-era countdown. Restarts now overshoot to 5x the ceiling
(PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience
window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has
real range.
- F3: checkpoint restores put weights back but kept the rejected trajectory's
Adam moments, so the optimizer immediately pushed back toward the rolled-back
state (the restore->regress->restore oscillation). CNet::ResetOptimizerState()
zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta
untouched) on every mid-run restore, every boosted restart, and the
deploy-time restore that online learning continues from.
- F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk,
so the selection metric the checkpoint ranking and deploy gate read is a pure
function of the checkpoint instead of partly measuring BN drift. Defensive
unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and
the OOS continual-learning simulation stay adaptive by design.
Compiled clean (0 errors, 0 warnings) via the staged-tree recipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one hypothesis the shipped diagnostics do not cover. The EA prints it
itself: the MI measure is "marginal (one feature at a time) and per-bar",
so a floor reading "cannot rule out one that only exists in combination or
across time". The instrument for across-time structure is the sequence
model, and until bfc1da9 it read its window backwards - so the hypothesis
has never been tested, not once.
Asks the DATA rather than retraining the LSTM. A topology comparison in
MT5 confounds "can order be exploited" with learning rate, init, capacity,
class weighting, batch-norm state and a single OOS slice, and costs hours
per run on a CPU-only box.
Three configurations, identical model / CV / null:
A entry bar only joint across FEATURES, one bar
B window, chronological joint across features AND time
C window, lag order shuffled INDEPENDENTLY PER SAMPLE
B-A: does history help at all. B-C: does temporal POSITION carry
information - the actual hypothesis.
C is the design. A single global shuffle tests nothing, because any model
relearns a permuted column layout; per-sample shuffling kills the
lag<->column correspondence while preserving the exact multiset of values
in the window. It is the time-order analogue of the shuffled-label null.
VALIDATED WITH BOTH CONTROLS, because an untested research script is worse
than none:
planted order-only label (monotone up/down):
A -4.13pp B +85.68pp C -4.13pp B-C +89.81pp
C collapses to exactly A - an order-destroyed window carries
precisely as much as one bar. Correct.
planted order-free label (window sum):
A +22.23pp B +65.62pp C +65.44pp B-C +0.18pp
history helps (+43pp over one bar) and the test attributes none of
it to order. Correct.
Metric is directional precision against max(P(Buy),P(Sell)) - the same
metric and zero-skill benchmark the EA's deployability gate uses, so a
result reads directly against an era line.
Two p-values, because the analytic one lies: barrier labels are massively
autocorrelated, so the binomial p is optimistic and is reported only for
comparability. The BLOCK-permutation p (block = horizon, Phipson & Smyth)
is the one to believe - same correction test_volume.py already applies.
Purged, embargoed walk-forward CV (H bars each side), because without the
embargo the barrier horizon leaks across the split - the mistake that
scored +0.53 R on an "honest" holdout once already.
normal_upper() mirrors NormalUpperTail() in Training.mqh so the two agree
to 6dp.
Needs <SYM>_<TF>_rates.csv from a WARRIOR_EXPORT_FEATURES build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Implemented sqx_audit.py to audit StrategyQuant X trade lists, focusing on performance metrics and cost analysis.
- Created sqx_portfolio.py to evaluate portfolio performance based on uncorrelated components and their impact on risk and return.
- Developed swing.py to analyze cost ratios across different holding periods and assess swing trading structures.
- Introduced test_management.py to investigate the effectiveness of exit rules on random entries and their impact on expectancy.
The MI sample builder used `MathAbs(labelBarOffset)` as a padding, causing rows from offset and non-offset builds to be paired with a double shift. This broke the positive control, failed the 5× gate, and voided all reported mutual‑information figures. Replace with the fixed `MiShiftPad` constant to ensure builds enumerate the same set of bars and row-k alignment is preserved.
Add `BatchOptionsTotal()` to `CNeuronBatchNormOCL` and split the packed BN weight array in the learning report into separate norms for the outgoing dense matrix, gamma, beta, running statistics, and Adam moment buffers. This turns an ambiguous single‑norm reading into precise diagnostics that distinguish weight divergence from scaling issues.
Three results today were invalidated by the same family of error: a price level
used as an entry while the outcome was measured from somewhere else. The defence
is not vigilance - it is an interface where the mistake cannot be expressed.
bidask.py M1 bars carrying SEPARATE BID and ASK OHLC, built from the tick
stream in one parallel pass. Every existing bar file stores MID, which is fine
for measuring returns and useless for simulating orders, because no order ever
executes at the mid. Mid bars force the spread to be bolted on afterwards as an
average - the approximation that let today's artifacts through. With both sides
carried, the spread is whatever it actually was, including the overnight and
news blowouts an average hides. Fails loudly if ask < bid.
fills.py owns the ENTIRE trade lifecycle; no test may open a position any other
way. Longs enter at the ask and exit at the bid, shorts the reverse. A buy stop
triggers on the ask and fills at the trigger price OR the bar's open if the bar
gapped past it, which is where real slippage comes from. Limits fill on the
opposite side and a gap is capped in their favour.
The invariant that was violated: THE OUTCOME CLOCK STARTS AT THE FILL BAR. The
fill index IS the start index - they are the same variable and cannot diverge.
That is what went wrong before, and it is now unrepresentable rather than
merely discouraged.
Same-bar ambiguity is REPORTED, not assumed away: every result carries the
fraction of trades decided by a bar containing both barriers, alongside fill
rate and unresolved rate. A resolution-limited result now says so itself.
M1 rather than raw ticks is a deliberate, stated bound: 513M ticks per symbol is
~8 GB packed and four symbols will not fit in memory, while M1 keeps it at ~500 MB
and cuts the residual ambiguity 60x versus H1. It is an approximation with a
visible error bar, not an exact simulator.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Applies the same five-trace context score from c998d65 to a completely
different entry - the LPS retest instead of the shakeout - as a check that the
dose-response was not specific to one setup.
shakeout trigger slope +0.0464 R/trace t +2.16 8/8 cells positive
LPS trigger slope +0.0425 R/trace t +3.18 7/8 cells positive
Two unrelated triggers, 31,000 non-overlapping trades, same magnitude to within
10%, same sign in 15 of 16 cells. This is the strongest confirmed result in the
whole programme: reading past structure to qualify current structure has real,
measurable predictive content. Villahermosa is describing something that exists.
AND IT IS STILL NOT ENOUGH. The buckets:
1 agree -0.287 3 agree -0.151
2 agree -0.206 4 agree -0.157
Context is a MODIFIER worth ~+0.045 R per agreeing trace. Every trigger it can
modify starts between -0.15 and -0.33, and only five traces exist, so full
confluence still lands short of break-even. Reaching zero would need ~7.
One cell goes positive - EURUSD H1 at 4 agreeing traces, +0.14 on n=133 - and it
is 1 of 40, on the smallest bucket, in the instrument with the lowest costs. That
is the exact shape of the three artifacts already retracted today, so it is
recorded as noise unless a fill-correct simulator says otherwise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Earlier tests fired on the shakeout alone, which is not the method. Book 2 2.3
treats it as the third of four cumulative traces and reads the structure's own
history first. This scores all of them, oriented to the shakeout's direction:
1 Phase A test location (upper vs lower half of the structure)
2 Phase B test location
2b STRUCTURAL FAILURE - after the Phase B test, did price fail to reach the
opposite extreme
4 effort/result on the shakeout bar (close position + volume vs range average)
7.1 higher-timeframe context - is the larger move in the shakeout's favour
Conditioning on agreement shrinks the sample and multiplies the ways to slice
it, so the test is NOT 'find the combination that works'. It is the one
pre-specified prediction the books make and mining does not: expR must rise
MONOTONICALLY with the number of agreeing traces. One slope, no threshold to
tune, no best cell to pick.
POOLED (16,234 non-overlapping trades):
0 traces -0.332 3 traces -0.114
1 trace -0.214 4 traces -0.117
2 traces -0.184
slope +0.0464 R per agreeing trace, t +2.16
per-symbol slopes POSITIVE IN ALL 8 CELLS (p ~ 0.004 on sign alone)
So the context logic is real and measurable - it is not folklore. But the base
trade is in too deep a hole for it to matter: full confluence still returns
-0.117, and reaching break-even would need ~7 agreeing traces when only 5 exist.
The useful reading is that context is a MODIFIER worth about +0.05 R per trace,
which is only interesting when bolted to a trigger whose base expectancy is
already near zero. The shakeout's is not, because its structural target sits
4-6R away and is rarely reached.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the programme on both books. Entries are MARKET ORDERS at a bar's
open throughout, so the fill-timing artifact that invalidated the last round is
designed out rather than remembered. Benchmark is analytic: entry, stop and
target fixed at entry means a driftless market gives expR = 0 exactly.
1. SPRING / UPTHRUST (book 1 ch.18, the event 'all Wyckoff operators wait for').
Pierce of a COMPRESSION-QUALIFIED range edge, close back inside, stop beyond
the shakeout extreme, target the far side of the range.
30 cells across 4 symbols x M15/H1/H4. Reward-to-risk averages 4-6:1, so the
break-even win rate is only 15-20%, and it still loses nearly everywhere:
M15 all four symbols -0.15 to -0.24 with 0/4 folds positive. Best cell is
EURUSD H1 climactic-volume +0.302 at t +2.27, which over 30 cells is inside
the family-wise band.
The books' volume requirement was applied - climactic (>1.5x range average)
and quiet (<0.8x) shakeouts scored separately. Neither rescues it.
2. LPS / LPSY, the test-after-breakout, and book 2's A/B (5.7.1, 5.8.3): it
claims the retest should be awaited at the VOLUME PROFILE level, not the
price edge. Same breakout, same stop, same 2R target, only the location
differs:
retest at typical expR
A price edge -0.041 .. -0.315
B value-area edge -0.128 .. -0.413
C range VPOC -0.129 .. -0.506
24/24 cells negative, and A > B > C in ALL EIGHT symbol/timeframe
combinations. That monotone ordering is not noise, and it inverts the book's
recommendation. Mechanism is adverse selection: the VPOC sits deep inside the
old range, so a retest that reaches it is disproportionately a breakout that
has already failed. The deeper the level you wait at, the more your fills are
selected against you.
Practical consequence: the volume profile is real (levels beat distance-matched
placebos at z +3 to +7.8) but using it to LOCATE ENTRIES makes this trade
worse, not better.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The +0.097 R EURUSD result in c9b489e is wrong. So is the +0.108 R pooled edge
in c2dd9eb and the selection model in 821f16d, which used the same labels.
THE BUG. The mirror test entered at the retail trigger price e2 - a STOP order
level - but started the outcome race at the OPEN of the fill bar. Price at that
open is on the far side of e2 by construction; that is why the order is a stop
order. So the race began before price had reached the entry, which handed the
fade a free run toward its target and pushed its stop further away than it
really was. Retail's side carries the same bias with the sign reversed, so the
DIFFERENCE - which is exactly how 'edge' was computed - was inflated twice over.
Found by generalising the trigger to swing-extreme breakouts, per the user's
suggestion. That version returned +0.7 R at t +110, which is not a result, and
it has the identical structure: enter at a level, measure from the bar open.
WITH AN HONEST INTRABAR FILL (first M5 bar that actually trades at the entry),
every EURUSD cell inverts:
bar open honest fill
H1 pin +0.0953 -0.0096
H1 pin +0.0559 -0.0573
H1 inside +0.0612 -0.0395
M15 pin +0.0407 -0.0135
The 4/4 walk-forward held because the bias was present in every fold. A
walk-forward validates against regime change, not against a broken fill model.
The tell was there and I walked past it: sweep_entry() was the ONE test that
modelled the fill properly, and it was the ONE test that came out negative. When
one arm of a suite disagrees with the rest, check what it does differently before
believing the majority.
So the standing conclusion returns to what it was: retail setups are close to a
coin flip that pays the spread, and there is nothing in them to harvest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reframes what the net is asked. Direction is the one thing the data says is
barely predictable; but the fade of retail pin/inside setups has a MEASURED
gross edge of ~0.139 R and a cost of spread/stop that varies 5x across
instruments, sessions and vol regimes. So the answerable question is selection:
spend the edge only where it survives.
label realised R of the fade (what actually pays)
features 28, all knowable at entry, incl. the volume-profile levels
split CHRONOLOGICAL train/val/test; threshold picked on val, frozen for test
TEST BLOCK 2022-08 .. 2026-07, 63,901 candidate trades, keep top 10%:
mean R +0.1005 vs -0.0616 for taking everything
by year +0.140 +0.075 +0.098 +0.105 +0.105 -> 5/5 positive
with the COST FEATURE REMOVED: +0.0800, still 5/5 positive
That last control matters: the model is not merely learning 'skip wide
spreads'. Something in the setup geometry, session and level structure carries
signal beyond the cost.
HONEST SIGNIFICANCE. The naive t of +8.06 is not believable - with an 8-day
horizon these trades overlap heavily and thousands share one price path. On a
strictly non-overlapping subset (163 independent trades) it is +0.123 R at
t +1.61; without the cost feature, +0.257 at t +3.45. The overlap filter applies
the H1 horizon to M15 trades too, so 163 is a conservative floor and the true
independent count is higher. Suggestive, not settled.
THE LEAK THIS RUN NEARLY SHIPPED. First version scored +0.53 R on the held-out
block, t +54. Bar-derived features were read at the FILL bar i2, but the order
fills intrabar and the outcome race starts at the first M5 bar inside i2 - so
i2's close, tick count and realised volatility are not knowable at entry. The
model was seeing how the bar it entered on turned out. Second lookahead of this
hunt (the first was worth +0.15 R in the sweep test).
RULE: a clean chronological split does NOT protect against lookahead. The split
was honest and the features were not. Any feature indexed at the entry bar must
be re-derived from the bar before it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit pooled four instruments with very different spread-to-stop
ratios and concluded the edge never beats the cost. That was too broad. Per
cell (48 cells, M5 path, k=1), 8 clear - and they are not scattered:
EURUSD H1 pin spread/stop 0.042 edge +0.160 cost 0.063 -> +0.097 R
EURUSD H1 pin 0.045 +0.127 0.067 +0.060
EURUSD H1 inside 0.047 +0.123 0.060 +0.063
EURUSD H1 inside 0.050 +0.102 0.062 +0.040
EURUSD M15 pin 0.074 +0.141 0.100 +0.042
EURUSD M15 pin 0.072 +0.140 0.099 +0.041
Every clearing cell is on the tightest-spread instrument. XAUUSD carries the
same gross edge (+0.09 to +0.135) and never clears, because its cost is 3x.
That is the mechanism predicting where the effect should survive and being
right - the opposite of the stop-run case, which inverted.
WALK-FORWARD, 4 chronological folds: 6 of 8 hold at >=3/4. EURUSD H1 pin short
side is +0.116 / +0.061 / +0.143 / +0.067 across 23 years, 4/4.
WIDENING THE STOP still says what it said: EURUSD H1 pin goes +0.078 (m=1) ->
+0.028 -> +0.009 -> +0.017 -> -0.003 (m=5). The gross edge collapses ~15x while
the stop widens 5x, so this is NOT drift - it is reversion inside roughly one
setup-risk of a stop order filled at a local extreme. It is only tradeable at
the tight stop, which is exactly where cost bites hardest.
WHAT IS NOT MODELLED, and it decides this: commission and stop slippage. Gross
edge is ~0.139 R = ~2.4 pips on a 17.3-pip stop, against 0.75 pips of spread.
That leaves ~1.6 pips of headroom for commission plus slippage before it is
gone. A demo forward test measuring both is the next step, not more history.
Also fixes a LOOKAHEAD found in the sweep-entry test: the protective stop was
anchored to the low of the very bar that filled the limit order, which is not
known until that bar closes. It was worth ~+0.15 R - larger than any real effect
here - and it inflated the placebo equally, which is how it was caught. With it
removed, buying at retail stop levels is no better than buying at an arbitrary
level the same distance away: the 'stops are a farmable magnet' claim fails its
own control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tests the user's thesis directly: if price is unpredictable, trade against the
people predicting it badly. Implements the three mechanical setups from 'How To
Day Trade Forex For Profit' ch.5 with their DOCUMENTED stop rules, so retail
stops are located exactly rather than by proxy.
THE MIRROR TEST. Retail's trade and its exact mirror, priced under identical
rules. Both sides pay the same spread and suffer the same same-bar tie
convention, so those cancel in the difference and double in the sum:
edge = (mirror - retail)/2 cost = -(mirror + retail)/2
pin EDGE +0.108 R COST +0.143 R
inside EDGE +0.068 R COST +0.140 R
engulf EDGE -0.001 R COST +0.095 R
So pin-bar and inside-bar setups really are anti-predictive - the first
confirmed directional edge in this project. Engulfing is a pure coin flip whose
loss is entirely the spread, i.e. money already gone to the broker.
Stable across three conventions: H1 bars pessimistic ties, M5 path pessimistic,
M5 path optimistic. Re-walking the barriers on M5 CUT the cost (0.195 -> 0.143)
and RAISED the edge (0.078 -> 0.108), so the coarse-bar convention was masking
the effect, not manufacturing it.
THEN THE TEST THAT KILLS IT. Cost in R is spread/stop-distance, so widening the
stop divides it. If the edge is directional drift it survives. Fade expR by stop
multiple (pin, k=1, 122k trades):
m=1.0 cost 0.146 expR -0.045 implied edge +0.101
m=1.5 cost 0.097 expR -0.067 +0.030
m=2.0 cost 0.073 expR -0.065 +0.008
m=3.0 cost 0.049 expR -0.051 -0.002
m=5.0 cost 0.029 expR -0.040 -0.011
The edge decays exactly as fast as the cost, then inverts. It was never drift:
it is reversion against a stop order filled AT a local extreme, and it lives
within one bar-range of the entry - the same short-horizon reversal the tick-flow
work already measured, meeting the same fate.
Also in this commit, the volume-profile claims from Wyckoff 2.0:
MAGNET all 10 tests positive vs a distance-matched placebo, z +3.0 to +7.8,
family-wise bar 2.79 - but the effect is +0.15 to +0.59pp on a ~74%
base rate, i.e. ~0.01 R.
REACTION naked VPOC and VPOC reject +0.71 to +0.86pp (z to +5.65); value-area
edges null or negative; HVN/LVN marginal.
80% RULE dead. 27.8% traversal against a 29.0% martingale benchmark. Acceptance
nearly DOUBLES the raw rate (14.1% -> 27.8%) and the benchmark doubles
with it - every bit of the apparent improvement is geometry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 1:1 range projection is the target rule both books recommend (book 1 ch.8
discards point-and-figure counting as too subjective and keeps the vertical
projection). Tested as a complete trade on 4 symbols x M15/H1: enter on the
range breakout, stop at the far side of the range, target k x risk.
A driftless market gives P(win) = 1/(1+k) and expR = 0 at EVERY k, so the
benchmark here is analytic - no permutation null needed. Result: expR sits on
that benchmark at every k on every symbol. Target placement does not move
expectancy, which is what a martingale already said.
But the law itself is measurable, and it is not 1:1. Regressing
log(MFE) on log(range height) with log(ATR) as a FREE regressor (a shared ATR
denominator correlates the errors and biases the exponent towards the
hypothesis, so it cannot be used to argue against it):
b = 0.10 .. 0.92, centred ~0.6
b = 1 rejected in 5 of 8 at >2sd, never significantly above 1
b = 0 rejected in 7 of 8
So a bigger cause does produce a bigger effect - sub-proportionally. The 1:1
projection systematically over-reaches after a large consolidation and
under-reaches after a small one. Median travel in risk-multiples falls
monotonically across height quartiles in 8 of 8 runs.
Also adds the volume-profile machinery the second book is built on and which
nothing in the EA has: tick-level volume-at-price on a fixed absolute grid,
per-session VPOC/value area by the standard Market Profile walk, naked VPOCs,
and HVN/LVN from a rolling causal composite.
Two biases are left in deliberately, both against the hypothesis: a bar
spanning stop and target books the loss, and spread is charged on entry and
both barriers. Unresolved trades are marked to market at the horizon rather
than discarded - discarding them deletes slow winners and manufactures a
false deficit at large k.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The statistics had cleared a family-wise bar (z to +6.56) and a split-half. Both
necessary, neither sufficient: thorough enough mining passes both. What mining
cannot do is obey a mechanism it was never fitted to - so the decisive test is
whether the effect appears WHERE THE THEORY SAYS IT MUST.
Osler's stop-clustering predicts the edge concentrates where the stop reservoir is
deepest and fresh forced orders arrive: London open and the London/NY overlap.
Measured, by session in real UTC (broker is UTC+2):
EURUSD Sydney/Tokyo +0.197* London open -0.063 London -0.156
USDJPY Rollover +0.233 London open -0.223
XAUUSD Sydney/Tokyo +0.070 London open -0.345 LDN/NY -0.185
SP500 Rollover +0.206 NY afternoon -0.098
Exactly inverted. The liquid sessions where stops actually cluster are the worst
on every instrument; what remains lives in Sydney/Tokyo and rollover - the
THINNEST hours, where fewest stops sit. That is not the mechanism, and thin hours
are also where spreads are widest and fills worst, so even the surviving fragment
points away from tradeability rather than toward it.
Walk-forward by quarter agrees, and shows what the two-way split was hiding:
EURUSD 2/4 positive (Q1 -0.029, Q2 +0.062, Q3 -0.010, Q4 +0.089)
USDJPY 1/4 XAUUSD 0/4 SP500 2/4
No instrument reaches 3/4. The split-half HOLDS was Q2+Q4 carrying Q1+Q3.
Verdict: not an edge. Recording it as disconfirmed rather than leaving an
encouraging half-result in the log, because the next person to read this - me -
would otherwise build on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Osler's currency-order-flow work is the published mechanism: stop-loss orders
cluster just beyond recent swing extremes and cascade price; take-profits cluster
and reverse it. So the claim is not "a pattern repeats" but "there is a reservoir
of forced orders at a location computable in advance" - which is falsifiable in a
way chart patterns are not.
Tested as a COMPLETE trade rather than a signal with barriers bolted on: entry,
stop and target all come from one structure. Price takes out the N-bar extreme
MARGINALLY (<= over*ATR), closes back inside, enter the opposite way next open,
stop just beyond the sweep extreme (where the liquidity actually was), target a
multiple of that risk.
19 of 24 configs clear a family-wise max-statistic bar on EURUSD/USDJPY H1, all
in the predicted direction, z to +6.56. R=1 configs are negative and R=2/R=3 turn
positive, which is coherent: the edge is directional and a tight stop pays the
spread as a large fraction of risk, so it needs a big R to clear.
SPLIT-HALF then kills most of it, as it should:
EURUSD N=50 ov=0.5 R=3 +0.013 / +0.042 HOLDS
EURUSD N=20 ov=0.5 R=3 +0.014 / +0.034 HOLDS
every R=2 config one half negative
USDJPY one half negative
Surviving configs are STRONGER in the second half, the opposite of a mined
artifact decaying out of sample. But N=20 and N=50 overlap heavily and are not
independent, so this is one instrument and one R - a lead, not a system.
dukas.py: direct Dukascopy datafeed client. SQX mirrors through its own CDN
(CdnCache/CdnDownloadJob) so there is nothing reusable there. Dukascopy publishes
the raw feed - bi5, raw LZMA, 20-byte big-endian records, ZERO-BASED MONTH in the
URL (fails silently into the wrong month otherwise). Cached, resumable, bounded
concurrency.
Two corrections it forced, per the user: SQX conforms Dukascopy data to the5ers'
broker profile AND timestamps. Measured empirically, broker time = UTC+2 (EET),
clean minimum. So (1) previously reported "hours" are BROKER time - gold's hour 1
is 23:00 UTC, the daily rollover and COMEX Globex reopen, a real mechanism; and
(2) Dukascopy's raw 0.2-pip ECN spread must NOT be used for cost - the5ers' ~0.47
is what is actually paid, so the existing cost analysis was right and Dukascopy
would have made every result look falsely tradeable. Its value is the bid/ask
VOLUMES, which SQX lacks entirely - true signed flow instead of the event-count
proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything tested before this asked ONE question - can recent price or flow
predict the next bar's direction - and answered no four ways. These are different
families, each with a published prior rather than a hunch.
1 TIME-SERIES MOMENTUM (Moskowitz/Ooi/Pedersen). 34 configurations across 4
symbols x D1/H4 x 6 lookbacks. Nothing. The one rule that looks strong -
XAUUSD H4 250-bar, p=0.0038, t+3.30, +10.32%/yr - returns essentially exactly
buy-and-hold's +10.36%. It is not timing gold, it is being long gold. Hence the
vs-B&H column: on a drifting asset a rule that is merely long most of the time
looks skilful and is not.
2 SEASONALITY. The first thing in this project to survive a properly controlled
test: 5 of 8 clear a family-wise max-statistic bar, two at p=0.0002. Split-half
kills two of them (USDJPY dow-6 n=116 and SP500 hour-0 n=533 are thin
off-session buckets). Two HOLD with near-identical halves:
XAUUSD hour 1 +2.29 bp (t+6.44) / +2.43 bp (t+5.53)
EURUSD hour 13 -1.47 bp (t-6.80) / -0.64 bp (t-3.56)
Gold's hour 1 alone carries more than half the +4.22 bp/day drift.
And it is still not tradeable. Widening the window to amortise the 4.92 bp
round trip: the best of 144 windows (hour 1, 8h) nets +0.14 bp/day, t +0.23,
and splits +1.29 / -1.00 - the sign flips between halves. Every other window is
negative. Real, stable, well measured, and about 2x too small to cross its own
spread. Same shape as the flow result.
3 OVERNIGHT/INTRADAY - folded into the hour analysis above.
4 VOLATILITY-MANAGED DRIFT (Moreira/Muir) - the one needing no directional edge.
Does NOT reproduce here: flat on gold (-0.01), and it HURTS SP500 (0.71 -> 0.41
D1, 0.77 -> 0.54 H4). Honest negative against a strong prior.
What survives all of it is drift, which is large and significant while every
timing rule is noise: XAUUSD +10.24%/yr (t 2.88), SP500 +12.25%/yr (t 2.84),
against USDJPY +1.16%/yr (t 0.59) and EURUSD ~0.
resample.py gains D1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
resample.py composes M5 bars into M15/H1/H4 exactly - every column this pipeline
produces is composable (sums sum, maxes max, OHLC nests, means re-weight by tick
count), so this costs seconds instead of another 37-minute decode per timeframe.
Asserts tick conservation and extreme preservation on every output.
Motivation: ATR grows ~sqrt(time) while the spread does not, so spread/ATR should
fall with timeframe and make a small edge affordable. It does, monotonically, and
the measurement is clean (EURUSD, 1:1 barriers):
M5 spread 0.099 ATR random wins 36.8% cost 13.2pp
M15 0.057 39.7% 10.3pp
H1 0.029 43.6% 6.4pp
H4 0.015 47.6% 2.4pp
But the signal decays at the same rate. Rows clearing the family-wise bar:
M5 many, z to -10.1
M15 many, z to -5.8
H1 2 of 9, one POSITIVE and one negative - the shape of noise, not signal
H4 none
So the effect lives where the cost is fatal and is gone where the cost is
affordable. They never cross.
Also added --cheap=Q, which trades only the lowest-Q quantile of spread/ATR. This
is the one honest use of an unsigned feature: it cannot point a direction but it
can decline to trade, and both terms are known before entry. It does cut cost
(EURUSD M15 10.3 -> 7.6pp, USDJPY 12.7 -> 6.4pp) and the effect does not survive
there either - nothing clears the bar.
test_flow.py gained --tf= and keeps the z-score window at ~1 day on every
timeframe rather than a fixed 288 bars.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1.93 BILLION ticks -> 5.5M M5 bars (EURUSD/USDJPY/XAUUSD 2003-2026, SP500
2011-2026). Sequential non-overlapping trades, triple barriers, per-bar spread,
direction-permutation null with a family-wise max-statistic bar.
Order flow is genuinely ANTI-predictive at M5 - price mildly reverses the prior
bar's flow. Same sign on all four instruments, clearing the family-wise bar on
three:
USDJPY z -10.10 -1.73pp vs null
EURUSD z -7.95 -1.41pp
XAUUSD z -5.34 -0.68pp
SP500 z -2.60 -0.87pp (does not clear; half the sample)
Agrees with the -0.0151 next-bar correlation (vs +0.4961 same-bar, which is the
contemporaneous Cont/Kukanov/Stoikov effect and is not edge).
And the cost dwarfs it. Random entry at 1 ATR barriers after spread:
EURUSD spread 0.099 ATR -> wins 36.8% (13.2pp below the costless 50%)
USDJPY 0.154 34.9% (15.1pp)
SP500 0.292 26.5% (23.5pp)
XAUUSD 0.450 20.0% (30.0pp)
Cost rises monotonically with spread/ATR, which is an internal consistency check
on the apparatus. A ~1pp effect against 13-30pp of cost is 10-100x short.
Reversing does not rescue it - expR_rev is negative in every row of every
geometry. Widening the barriers does not either: at 4-8 ATR nothing clears the
bar (max |z| 2.51 vs 2.97). The effect lives exactly where the spread is fatal
and vanishes where the spread would be affordable, which is what a
seconds-to-minutes phenomenon predicts.
Adds --narrow/--wide geometry sets so both regimes are reproducible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first run reported -23pp edges at -75 sigma, which is not a market effect -
it is the tell this project has been burned by before (a lookahead, or here a
wrong reference, inflates whatever sign it lands on). The give-away was in the
output itself: a family-wise 5% bar of |z| > 71.67 where a centred null over 16
tests should sit near 2.5. Random entry was losing almost as badly as the signal.
Cause: z and "edge pp" were measured against be = sl/(sl+tp), the break-even of a
COSTLESS coin. These barriers charge the spread and book a loss when a single bar
spans both levels, so random entry at 1 ATR on M5 wins ~36.8%, not 50%. The table
was reporting the fixed cost of trading as if it were signal.
Now every row shows the empirical null win rate, the gap against it, and z
standardised by the null's own spread. Family-wise bar drops to 2.95 and the
result becomes legible: order flow is genuinely ANTI-predictive at M5, about 1pp
below random at z -5 to -8, clearing the bar in 12 of 16 tests and reproducing
across three geometries and two independent signal families. It agrees with the
-0.0151 next-bar correlation.
It is also untradeable, which the table now says out loud: the spread is 0.099 ATR
and costs 13pp of win rate against a 1pp effect. Reversing does not rescue it -
expR_rev is reported per row and stays negative everywhere. Added a footer stating
that beating the null is necessary but NOT sufficient; only exp R > 0 makes money.
Also:
- permutation null was allocating a single (nperm x nT) array, ~2 GB at these
trade counts. Now batched.
- null permutes the OBSERVED directions instead of flipping a fair coin, so a
directionally skewed rule on a trending instrument cannot pass on drift alone.
- timeouts reported separately rather than silently booked as stop-outs.
- calibration falls back to a midpoint sample when the tick history predates the
MT5 reference series (XAUUSD ticks start 2003-05-05, its H1 export 2004-06-11,
so the head sample overlapped by nothing). 2M ticks, because the sample must
span >=50 reference HOURS - 200k ticks of modern gold is nine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SQX decoder is a per-record Python loop and cannot be vectorised - record
LENGTH depends on the config nibbles, so record k+1's offset is unknowable
without parsing record k. It therefore saturated exactly one core: 10% CPU on a
12-core box, ~3h for the four files.
But the format is randomly seekable. Every BLOCK_LENGTH records SQX restates all
four fields as absolute int64s, so byte ranges beginning at block headers decode
with no shared history. split_offsets() cuts a file on those boundaries and
decode_iter() gained start/stop. EURUSD: 55 min -> 9.4 min, 94% CPU.
find_block() will not trust a bare MAGIC match: 0x00..0x0e is a byte run that
delta payloads produce by coincidence, so a candidate is accepted only when the
next header downstream carries the next sequential block index.
Verified equal, not assumed equal: the same 315MB span decoded serially and in 6
chunks gives identical tick counts (31,056,000), identical bar counts (206,316)
and identical OHLC. The only divergence is the documented seam artifact - the
first tick of a chunk has no predecessor so its delta counts as zero, bounded at
workers-1 ticks in 513M (~2e-8).
Two fixes this shook out:
- The feed is not perfectly time-ordered. EURUSD carries 2 backward steps in
513,494,303 ticks, both under an hour, both in 2003-2006. Bucketing is by
absolute timestamp so every tick still lands in its true bar; the symptom is a
bucket emitted twice out of order. finalise() now stable-sorts before the
duplicate merge. The ordering assert is kept but keyed to MAGNITUDE, since a
real chunking bug displaces a large fraction of rows and feed noise displaces a
handful - only one of those is safe to continue past.
- Chunk workers return undivided sums; means are divided once globally. Dividing
per chunk would weight a straddling bar's mean-of-means wrong.
test_flow.py: charge the PER-BAR spread instead of a single median across
2003-2026 - FX spreads narrowed by roughly an order of magnitude over that span,
so one median charges modern cost to the 2000s and vice versa. Timeouts are now
reported separately rather than silently booked as stop-outs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ready ahead of the bar build so the analysis runs the moment EURUSD lands.
Tests the two SIGNED features the tick pipeline can produce - tick-rule imbalance and
event-count OFI - as entry triggers at 1:1/12, 1:2/24 and 2:3/48 bar geometries. Everything
else the pipeline computes is unsigned and cannot point a trade however well it measures.
Same discipline as every other test here: signal on bar i, entry at the OPEN of i+1,
barriers scanned forward only, sequential NON-OVERLAPPING trades so each is independent
(skipping that is what produced a fake +2.66pp at 2.9 sigma earlier in this project),
break-even == chance by the gambler's-ruin identity, spread charged inside the barrier, and
a sign-flip null taking the max over the whole family for the family-wise bar.
The prior is written into the docstring before any result exists: OFI is established as a
CONTEMPORANEOUS explainer whose predictive power decays within seconds, and it reproduced
that here at +0.56 against the same-bar return. So the expectation is that it explains the
bar it is measured in and says nothing about the next. What is actually being tested is the
gap between that literature - equities, sub-second, size-weighted book data - and this
setting: retail FX CFD feed, 5-minute bars, event counts without sizes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces bidmoves/askmoves with bid_up, bid_dn, ask_up, ask_dn. A bid ticking UP and an ask
ticking DOWN both mean buy-side pressure, and a counter that only records "the bid changed"
cannot tell them apart - it throws away the direction, which is the only part that could
ever point a trade. This is order-flow imbalance in the Cont/Kukanov/Stoikov sense, in its
event-count form; the feed carries no sizes so it cannot be size-weighted.
Caught before the 3-hour build rather than after, which was the point of smoke-testing on a
bounded sample first.
Verified against the previous column set on the same 3M ticks: 21,971 bars, ticks/bar 76,
up 38, dn 38, spread 0.000126, rvol 4.800e-07, gaps 3.95/31.4 - all identical - and
bid_up+bid_dn reproduces the old bidmoves count of 74 exactly, as it must.
The orientation check that matters: OFI correlates +0.56 with the SAME-bar return. That is
the contemporaneous signature the literature reports, and it is also the cheapest guard
against the failure mode that would otherwise pass silently - a sign flip would read -0.56
and every downstream test would then be fitting the negative of the intended feature.
Expectations set in the docstring rather than discovered later: OFI is well established as
a contemporaneous EXPLAINER of price change and its predictive power decays within seconds.
At M5 with multi-hour horizons the prior should be that it explains the bar it is measured
in, not the next one. Measuring it anyway is the point - but a +0.56 contemporaneous
correlation is not evidence of an edge and must not be reported as one.
Merge/mean bookkeeping is now index-driven off COLUMNS instead of positional, so adding a
feature cannot silently mis-merge a bar that straddles a batch boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sqx.decode_iter() turns the decoder into a generator, and ticks_to_bars.py reduces a
symbol to bars in one bounded-memory pass. Necessary rather than tidy: EURUSD is ~458M
ticks, which is ~15 GB held as arrays, so nothing downstream can take the raw stream.
23 years collapses to ~2.4M M5 bars that every test can load instantly.
Aggregation is vectorised with reduceat rather than looping per tick. The only real
complexity is that a bar can straddle a batch boundary, so the last partial bar of each
batch is carried and merged into the first of the next; per-tick deltas are likewise seeded
from the previous batch's final tick, so the first tick of a batch is not silently treated
as having no predecessor. Verified against the per-tick implementation it replaces: 21,971
bars either way, and every reported median identical to the digit (ticks/bar 76, up 38, dn
38, bidmoves 74, askmoves 74, spread 0.000126/0.000250, rvol 4.800e-07, gaps 3.95/31.4).
Throughput 132k ticks/s, at which point the decoder itself is the bottleneck and the
aggregation costs ~12%.
Features are chosen by what the feed can honestly support. It carries (time, bid, ask,
volume) and no trade direction - SQX's record has one volume field and MT5's
TICK_FLAG_BUY/SELL are empty on FX - so true signed order flow does not exist here and is
not synthesised under a flattering name. What is available:
tick rule up/down mid-price changes; the standard Lee-Ready fallback
quote asymmetry bid updates vs ask updates - which side is being repriced harder
arrival rate inter-tick gaps, mean and max; urgency rather than size
realised variance sum of squared mid returns, a far better volatility estimate than
the bar range and only obtainable from ticks
spread mean and max within the bar
Of these only the tick rule and quote asymmetry can point a direction; the rest are
unsigned, like every feature that has measured above noise in this project so far.
Bars are stamped by the OPEN of their interval and built only from ticks inside it, so no
bar's features depend on a tick after it closes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads SQX tick format 4.2 in pure Python. Derived from SQX's own writer, disassembled out
of internal/libs/SQDataLib.jar (TickDataWriter, NewDataFormat{,Writter}) with the javap
bundled in the install - so this follows the format as specified rather than as guessed.
Why, when Scripts/ExportTicks.mq5 pulls the same four fields from MT5: DEPTH. The broker's
MT5 tick history covers a few years; this file starts 2011-09-19. Sample size has been the
binding constraint on every question in this project - the H1/128-bar setup yields ~300
independent trades in 18 years, enough to resolve only a +6pp edge - so 15 years of ticks
is worth a decoder.
Format: four writeUTF strings, ten zero bytes, one more writeUTF, then records of
(time, ASK, BID, volume) - ask before bid, and the writer swaps them when bid>ask so ask is
always the larger. Every BLOCK_LENGTH=1000 records: MAGIC (15 bytes 0x00..0x0e) + int32
block index + config + four raw int64s. In between, deltas against the previous record.
Config is two bytes = four nibbles laid out high-first, nibble = (logicType << 2) |
dataType, where dataType 0..3 selects a 1/2/4/8-byte payload by magnitude and logicType
supplies the sign (MINUS/PLUS carry unsigned magnitudes; ASIS is a plain signed read).
The scale is the one thing NOT in the file. SQX keeps `decimals` in external metadata, and
1216010000 is equally plausible at 10^3, 10^5 or 10^6 - nothing in the bytes distinguishes
them. Guessing would be precisely the silent, plausible-looking error this project keeps
getting caught by: a 100x price scale error crashes nothing, it just quietly rescales every
ATR-normalised feature downstream. So calibrate_decimals() matches against a known
reference series instead. Against the MT5 SP500 H1 export the answer is not marginal:
decimals=5 median rel.err 9.001630
decimals=6 median rel.err 0.004078 <-
decimals=7 median rel.err 0.899984
Verified on 4M ticks: strictly monotonic timestamps, zero negative spreads, price range
1118.03-2048.38 over 2011-09 to 2014-11 (correct for SP500), median spread 0.43 (matches
this broker's H1 record). The 0.41% residual is the expected artefact of comparing a tick
ask against the nearest H1 bar close.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
