- Implemented sqx_audit.py to audit StrategyQuant X trade lists, focusing on performance metrics and cost analysis.
- Created sqx_portfolio.py to evaluate portfolio performance based on uncorrelated components and their impact on risk and return.
- Developed swing.py to analyze cost ratios across different holding periods and assess swing trading structures.
- Introduced test_management.py to investigate the effectiveness of exit rules on random entries and their impact on expectancy.
The MI sample builder used `MathAbs(labelBarOffset)` as a padding, causing rows from offset and non-offset builds to be paired with a double shift. This broke the positive control, failed the 5× gate, and voided all reported mutual‑information figures. Replace with the fixed `MiShiftPad` constant to ensure builds enumerate the same set of bars and row-k alignment is preserved.
Add `BatchOptionsTotal()` to `CNeuronBatchNormOCL` and split the packed BN weight array in the learning report into separate norms for the outgoing dense matrix, gamma, beta, running statistics, and Adam moment buffers. This turns an ambiguous single‑norm reading into precise diagnostics that distinguish weight divergence from scaling issues.
Three results today were invalidated by the same family of error: a price level
used as an entry while the outcome was measured from somewhere else. The defence
is not vigilance - it is an interface where the mistake cannot be expressed.
bidask.py M1 bars carrying SEPARATE BID and ASK OHLC, built from the tick
stream in one parallel pass. Every existing bar file stores MID, which is fine
for measuring returns and useless for simulating orders, because no order ever
executes at the mid. Mid bars force the spread to be bolted on afterwards as an
average - the approximation that let today's artifacts through. With both sides
carried, the spread is whatever it actually was, including the overnight and
news blowouts an average hides. Fails loudly if ask < bid.
fills.py owns the ENTIRE trade lifecycle; no test may open a position any other
way. Longs enter at the ask and exit at the bid, shorts the reverse. A buy stop
triggers on the ask and fills at the trigger price OR the bar's open if the bar
gapped past it, which is where real slippage comes from. Limits fill on the
opposite side and a gap is capped in their favour.
The invariant that was violated: THE OUTCOME CLOCK STARTS AT THE FILL BAR. The
fill index IS the start index - they are the same variable and cannot diverge.
That is what went wrong before, and it is now unrepresentable rather than
merely discouraged.
Same-bar ambiguity is REPORTED, not assumed away: every result carries the
fraction of trades decided by a bar containing both barriers, alongside fill
rate and unresolved rate. A resolution-limited result now says so itself.
M1 rather than raw ticks is a deliberate, stated bound: 513M ticks per symbol is
~8 GB packed and four symbols will not fit in memory, while M1 keeps it at ~500 MB
and cuts the residual ambiguity 60x versus H1. It is an approximation with a
visible error bar, not an exact simulator.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Applies the same five-trace context score from c998d65 to a completely
different entry - the LPS retest instead of the shakeout - as a check that the
dose-response was not specific to one setup.
shakeout trigger slope +0.0464 R/trace t +2.16 8/8 cells positive
LPS trigger slope +0.0425 R/trace t +3.18 7/8 cells positive
Two unrelated triggers, 31,000 non-overlapping trades, same magnitude to within
10%, same sign in 15 of 16 cells. This is the strongest confirmed result in the
whole programme: reading past structure to qualify current structure has real,
measurable predictive content. Villahermosa is describing something that exists.
AND IT IS STILL NOT ENOUGH. The buckets:
1 agree -0.287 3 agree -0.151
2 agree -0.206 4 agree -0.157
Context is a MODIFIER worth ~+0.045 R per agreeing trace. Every trigger it can
modify starts between -0.15 and -0.33, and only five traces exist, so full
confluence still lands short of break-even. Reaching zero would need ~7.
One cell goes positive - EURUSD H1 at 4 agreeing traces, +0.14 on n=133 - and it
is 1 of 40, on the smallest bucket, in the instrument with the lowest costs. That
is the exact shape of the three artifacts already retracted today, so it is
recorded as noise unless a fill-correct simulator says otherwise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Earlier tests fired on the shakeout alone, which is not the method. Book 2 2.3
treats it as the third of four cumulative traces and reads the structure's own
history first. This scores all of them, oriented to the shakeout's direction:
1 Phase A test location (upper vs lower half of the structure)
2 Phase B test location
2b STRUCTURAL FAILURE - after the Phase B test, did price fail to reach the
opposite extreme
4 effort/result on the shakeout bar (close position + volume vs range average)
7.1 higher-timeframe context - is the larger move in the shakeout's favour
Conditioning on agreement shrinks the sample and multiplies the ways to slice
it, so the test is NOT 'find the combination that works'. It is the one
pre-specified prediction the books make and mining does not: expR must rise
MONOTONICALLY with the number of agreeing traces. One slope, no threshold to
tune, no best cell to pick.
POOLED (16,234 non-overlapping trades):
0 traces -0.332 3 traces -0.114
1 trace -0.214 4 traces -0.117
2 traces -0.184
slope +0.0464 R per agreeing trace, t +2.16
per-symbol slopes POSITIVE IN ALL 8 CELLS (p ~ 0.004 on sign alone)
So the context logic is real and measurable - it is not folklore. But the base
trade is in too deep a hole for it to matter: full confluence still returns
-0.117, and reaching break-even would need ~7 agreeing traces when only 5 exist.
The useful reading is that context is a MODIFIER worth about +0.05 R per trace,
which is only interesting when bolted to a trigger whose base expectancy is
already near zero. The shakeout's is not, because its structural target sits
4-6R away and is rarely reached.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the programme on both books. Entries are MARKET ORDERS at a bar's
open throughout, so the fill-timing artifact that invalidated the last round is
designed out rather than remembered. Benchmark is analytic: entry, stop and
target fixed at entry means a driftless market gives expR = 0 exactly.
1. SPRING / UPTHRUST (book 1 ch.18, the event 'all Wyckoff operators wait for').
Pierce of a COMPRESSION-QUALIFIED range edge, close back inside, stop beyond
the shakeout extreme, target the far side of the range.
30 cells across 4 symbols x M15/H1/H4. Reward-to-risk averages 4-6:1, so the
break-even win rate is only 15-20%, and it still loses nearly everywhere:
M15 all four symbols -0.15 to -0.24 with 0/4 folds positive. Best cell is
EURUSD H1 climactic-volume +0.302 at t +2.27, which over 30 cells is inside
the family-wise band.
The books' volume requirement was applied - climactic (>1.5x range average)
and quiet (<0.8x) shakeouts scored separately. Neither rescues it.
2. LPS / LPSY, the test-after-breakout, and book 2's A/B (5.7.1, 5.8.3): it
claims the retest should be awaited at the VOLUME PROFILE level, not the
price edge. Same breakout, same stop, same 2R target, only the location
differs:
retest at typical expR
A price edge -0.041 .. -0.315
B value-area edge -0.128 .. -0.413
C range VPOC -0.129 .. -0.506
24/24 cells negative, and A > B > C in ALL EIGHT symbol/timeframe
combinations. That monotone ordering is not noise, and it inverts the book's
recommendation. Mechanism is adverse selection: the VPOC sits deep inside the
old range, so a retest that reaches it is disproportionately a breakout that
has already failed. The deeper the level you wait at, the more your fills are
selected against you.
Practical consequence: the volume profile is real (levels beat distance-matched
placebos at z +3 to +7.8) but using it to LOCATE ENTRIES makes this trade
worse, not better.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The +0.097 R EURUSD result in c9b489e is wrong. So is the +0.108 R pooled edge
in c2dd9eb and the selection model in 821f16d, which used the same labels.
THE BUG. The mirror test entered at the retail trigger price e2 - a STOP order
level - but started the outcome race at the OPEN of the fill bar. Price at that
open is on the far side of e2 by construction; that is why the order is a stop
order. So the race began before price had reached the entry, which handed the
fade a free run toward its target and pushed its stop further away than it
really was. Retail's side carries the same bias with the sign reversed, so the
DIFFERENCE - which is exactly how 'edge' was computed - was inflated twice over.
Found by generalising the trigger to swing-extreme breakouts, per the user's
suggestion. That version returned +0.7 R at t +110, which is not a result, and
it has the identical structure: enter at a level, measure from the bar open.
WITH AN HONEST INTRABAR FILL (first M5 bar that actually trades at the entry),
every EURUSD cell inverts:
bar open honest fill
H1 pin +0.0953 -0.0096
H1 pin +0.0559 -0.0573
H1 inside +0.0612 -0.0395
M15 pin +0.0407 -0.0135
The 4/4 walk-forward held because the bias was present in every fold. A
walk-forward validates against regime change, not against a broken fill model.
The tell was there and I walked past it: sweep_entry() was the ONE test that
modelled the fill properly, and it was the ONE test that came out negative. When
one arm of a suite disagrees with the rest, check what it does differently before
believing the majority.
So the standing conclusion returns to what it was: retail setups are close to a
coin flip that pays the spread, and there is nothing in them to harvest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reframes what the net is asked. Direction is the one thing the data says is
barely predictable; but the fade of retail pin/inside setups has a MEASURED
gross edge of ~0.139 R and a cost of spread/stop that varies 5x across
instruments, sessions and vol regimes. So the answerable question is selection:
spend the edge only where it survives.
label realised R of the fade (what actually pays)
features 28, all knowable at entry, incl. the volume-profile levels
split CHRONOLOGICAL train/val/test; threshold picked on val, frozen for test
TEST BLOCK 2022-08 .. 2026-07, 63,901 candidate trades, keep top 10%:
mean R +0.1005 vs -0.0616 for taking everything
by year +0.140 +0.075 +0.098 +0.105 +0.105 -> 5/5 positive
with the COST FEATURE REMOVED: +0.0800, still 5/5 positive
That last control matters: the model is not merely learning 'skip wide
spreads'. Something in the setup geometry, session and level structure carries
signal beyond the cost.
HONEST SIGNIFICANCE. The naive t of +8.06 is not believable - with an 8-day
horizon these trades overlap heavily and thousands share one price path. On a
strictly non-overlapping subset (163 independent trades) it is +0.123 R at
t +1.61; without the cost feature, +0.257 at t +3.45. The overlap filter applies
the H1 horizon to M15 trades too, so 163 is a conservative floor and the true
independent count is higher. Suggestive, not settled.
THE LEAK THIS RUN NEARLY SHIPPED. First version scored +0.53 R on the held-out
block, t +54. Bar-derived features were read at the FILL bar i2, but the order
fills intrabar and the outcome race starts at the first M5 bar inside i2 - so
i2's close, tick count and realised volatility are not knowable at entry. The
model was seeing how the bar it entered on turned out. Second lookahead of this
hunt (the first was worth +0.15 R in the sweep test).
RULE: a clean chronological split does NOT protect against lookahead. The split
was honest and the features were not. Any feature indexed at the entry bar must
be re-derived from the bar before it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit pooled four instruments with very different spread-to-stop
ratios and concluded the edge never beats the cost. That was too broad. Per
cell (48 cells, M5 path, k=1), 8 clear - and they are not scattered:
EURUSD H1 pin spread/stop 0.042 edge +0.160 cost 0.063 -> +0.097 R
EURUSD H1 pin 0.045 +0.127 0.067 +0.060
EURUSD H1 inside 0.047 +0.123 0.060 +0.063
EURUSD H1 inside 0.050 +0.102 0.062 +0.040
EURUSD M15 pin 0.074 +0.141 0.100 +0.042
EURUSD M15 pin 0.072 +0.140 0.099 +0.041
Every clearing cell is on the tightest-spread instrument. XAUUSD carries the
same gross edge (+0.09 to +0.135) and never clears, because its cost is 3x.
That is the mechanism predicting where the effect should survive and being
right - the opposite of the stop-run case, which inverted.
WALK-FORWARD, 4 chronological folds: 6 of 8 hold at >=3/4. EURUSD H1 pin short
side is +0.116 / +0.061 / +0.143 / +0.067 across 23 years, 4/4.
WIDENING THE STOP still says what it said: EURUSD H1 pin goes +0.078 (m=1) ->
+0.028 -> +0.009 -> +0.017 -> -0.003 (m=5). The gross edge collapses ~15x while
the stop widens 5x, so this is NOT drift - it is reversion inside roughly one
setup-risk of a stop order filled at a local extreme. It is only tradeable at
the tight stop, which is exactly where cost bites hardest.
WHAT IS NOT MODELLED, and it decides this: commission and stop slippage. Gross
edge is ~0.139 R = ~2.4 pips on a 17.3-pip stop, against 0.75 pips of spread.
That leaves ~1.6 pips of headroom for commission plus slippage before it is
gone. A demo forward test measuring both is the next step, not more history.
Also fixes a LOOKAHEAD found in the sweep-entry test: the protective stop was
anchored to the low of the very bar that filled the limit order, which is not
known until that bar closes. It was worth ~+0.15 R - larger than any real effect
here - and it inflated the placebo equally, which is how it was caught. With it
removed, buying at retail stop levels is no better than buying at an arbitrary
level the same distance away: the 'stops are a farmable magnet' claim fails its
own control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tests the user's thesis directly: if price is unpredictable, trade against the
people predicting it badly. Implements the three mechanical setups from 'How To
Day Trade Forex For Profit' ch.5 with their DOCUMENTED stop rules, so retail
stops are located exactly rather than by proxy.
THE MIRROR TEST. Retail's trade and its exact mirror, priced under identical
rules. Both sides pay the same spread and suffer the same same-bar tie
convention, so those cancel in the difference and double in the sum:
edge = (mirror - retail)/2 cost = -(mirror + retail)/2
pin EDGE +0.108 R COST +0.143 R
inside EDGE +0.068 R COST +0.140 R
engulf EDGE -0.001 R COST +0.095 R
So pin-bar and inside-bar setups really are anti-predictive - the first
confirmed directional edge in this project. Engulfing is a pure coin flip whose
loss is entirely the spread, i.e. money already gone to the broker.
Stable across three conventions: H1 bars pessimistic ties, M5 path pessimistic,
M5 path optimistic. Re-walking the barriers on M5 CUT the cost (0.195 -> 0.143)
and RAISED the edge (0.078 -> 0.108), so the coarse-bar convention was masking
the effect, not manufacturing it.
THEN THE TEST THAT KILLS IT. Cost in R is spread/stop-distance, so widening the
stop divides it. If the edge is directional drift it survives. Fade expR by stop
multiple (pin, k=1, 122k trades):
m=1.0 cost 0.146 expR -0.045 implied edge +0.101
m=1.5 cost 0.097 expR -0.067 +0.030
m=2.0 cost 0.073 expR -0.065 +0.008
m=3.0 cost 0.049 expR -0.051 -0.002
m=5.0 cost 0.029 expR -0.040 -0.011
The edge decays exactly as fast as the cost, then inverts. It was never drift:
it is reversion against a stop order filled AT a local extreme, and it lives
within one bar-range of the entry - the same short-horizon reversal the tick-flow
work already measured, meeting the same fate.
Also in this commit, the volume-profile claims from Wyckoff 2.0:
MAGNET all 10 tests positive vs a distance-matched placebo, z +3.0 to +7.8,
family-wise bar 2.79 - but the effect is +0.15 to +0.59pp on a ~74%
base rate, i.e. ~0.01 R.
REACTION naked VPOC and VPOC reject +0.71 to +0.86pp (z to +5.65); value-area
edges null or negative; HVN/LVN marginal.
80% RULE dead. 27.8% traversal against a 29.0% martingale benchmark. Acceptance
nearly DOUBLES the raw rate (14.1% -> 27.8%) and the benchmark doubles
with it - every bit of the apparent improvement is geometry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 1:1 range projection is the target rule both books recommend (book 1 ch.8
discards point-and-figure counting as too subjective and keeps the vertical
projection). Tested as a complete trade on 4 symbols x M15/H1: enter on the
range breakout, stop at the far side of the range, target k x risk.
A driftless market gives P(win) = 1/(1+k) and expR = 0 at EVERY k, so the
benchmark here is analytic - no permutation null needed. Result: expR sits on
that benchmark at every k on every symbol. Target placement does not move
expectancy, which is what a martingale already said.
But the law itself is measurable, and it is not 1:1. Regressing
log(MFE) on log(range height) with log(ATR) as a FREE regressor (a shared ATR
denominator correlates the errors and biases the exponent towards the
hypothesis, so it cannot be used to argue against it):
b = 0.10 .. 0.92, centred ~0.6
b = 1 rejected in 5 of 8 at >2sd, never significantly above 1
b = 0 rejected in 7 of 8
So a bigger cause does produce a bigger effect - sub-proportionally. The 1:1
projection systematically over-reaches after a large consolidation and
under-reaches after a small one. Median travel in risk-multiples falls
monotonically across height quartiles in 8 of 8 runs.
Also adds the volume-profile machinery the second book is built on and which
nothing in the EA has: tick-level volume-at-price on a fixed absolute grid,
per-session VPOC/value area by the standard Market Profile walk, naked VPOCs,
and HVN/LVN from a rolling causal composite.
Two biases are left in deliberately, both against the hypothesis: a bar
spanning stop and target books the loss, and spread is charged on entry and
both barriers. Unresolved trades are marked to market at the horizon rather
than discarded - discarding them deletes slow winners and manufactures a
false deficit at large k.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The statistics had cleared a family-wise bar (z to +6.56) and a split-half. Both
necessary, neither sufficient: thorough enough mining passes both. What mining
cannot do is obey a mechanism it was never fitted to - so the decisive test is
whether the effect appears WHERE THE THEORY SAYS IT MUST.
Osler's stop-clustering predicts the edge concentrates where the stop reservoir is
deepest and fresh forced orders arrive: London open and the London/NY overlap.
Measured, by session in real UTC (broker is UTC+2):
EURUSD Sydney/Tokyo +0.197* London open -0.063 London -0.156
USDJPY Rollover +0.233 London open -0.223
XAUUSD Sydney/Tokyo +0.070 London open -0.345 LDN/NY -0.185
SP500 Rollover +0.206 NY afternoon -0.098
Exactly inverted. The liquid sessions where stops actually cluster are the worst
on every instrument; what remains lives in Sydney/Tokyo and rollover - the
THINNEST hours, where fewest stops sit. That is not the mechanism, and thin hours
are also where spreads are widest and fills worst, so even the surviving fragment
points away from tradeability rather than toward it.
Walk-forward by quarter agrees, and shows what the two-way split was hiding:
EURUSD 2/4 positive (Q1 -0.029, Q2 +0.062, Q3 -0.010, Q4 +0.089)
USDJPY 1/4 XAUUSD 0/4 SP500 2/4
No instrument reaches 3/4. The split-half HOLDS was Q2+Q4 carrying Q1+Q3.
Verdict: not an edge. Recording it as disconfirmed rather than leaving an
encouraging half-result in the log, because the next person to read this - me -
would otherwise build on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Osler's currency-order-flow work is the published mechanism: stop-loss orders
cluster just beyond recent swing extremes and cascade price; take-profits cluster
and reverse it. So the claim is not "a pattern repeats" but "there is a reservoir
of forced orders at a location computable in advance" - which is falsifiable in a
way chart patterns are not.
Tested as a COMPLETE trade rather than a signal with barriers bolted on: entry,
stop and target all come from one structure. Price takes out the N-bar extreme
MARGINALLY (<= over*ATR), closes back inside, enter the opposite way next open,
stop just beyond the sweep extreme (where the liquidity actually was), target a
multiple of that risk.
19 of 24 configs clear a family-wise max-statistic bar on EURUSD/USDJPY H1, all
in the predicted direction, z to +6.56. R=1 configs are negative and R=2/R=3 turn
positive, which is coherent: the edge is directional and a tight stop pays the
spread as a large fraction of risk, so it needs a big R to clear.
SPLIT-HALF then kills most of it, as it should:
EURUSD N=50 ov=0.5 R=3 +0.013 / +0.042 HOLDS
EURUSD N=20 ov=0.5 R=3 +0.014 / +0.034 HOLDS
every R=2 config one half negative
USDJPY one half negative
Surviving configs are STRONGER in the second half, the opposite of a mined
artifact decaying out of sample. But N=20 and N=50 overlap heavily and are not
independent, so this is one instrument and one R - a lead, not a system.
dukas.py: direct Dukascopy datafeed client. SQX mirrors through its own CDN
(CdnCache/CdnDownloadJob) so there is nothing reusable there. Dukascopy publishes
the raw feed - bi5, raw LZMA, 20-byte big-endian records, ZERO-BASED MONTH in the
URL (fails silently into the wrong month otherwise). Cached, resumable, bounded
concurrency.
Two corrections it forced, per the user: SQX conforms Dukascopy data to the5ers'
broker profile AND timestamps. Measured empirically, broker time = UTC+2 (EET),
clean minimum. So (1) previously reported "hours" are BROKER time - gold's hour 1
is 23:00 UTC, the daily rollover and COMEX Globex reopen, a real mechanism; and
(2) Dukascopy's raw 0.2-pip ECN spread must NOT be used for cost - the5ers' ~0.47
is what is actually paid, so the existing cost analysis was right and Dukascopy
would have made every result look falsely tradeable. Its value is the bid/ask
VOLUMES, which SQX lacks entirely - true signed flow instead of the event-count
proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything tested before this asked ONE question - can recent price or flow
predict the next bar's direction - and answered no four ways. These are different
families, each with a published prior rather than a hunch.
1 TIME-SERIES MOMENTUM (Moskowitz/Ooi/Pedersen). 34 configurations across 4
symbols x D1/H4 x 6 lookbacks. Nothing. The one rule that looks strong -
XAUUSD H4 250-bar, p=0.0038, t+3.30, +10.32%/yr - returns essentially exactly
buy-and-hold's +10.36%. It is not timing gold, it is being long gold. Hence the
vs-B&H column: on a drifting asset a rule that is merely long most of the time
looks skilful and is not.
2 SEASONALITY. The first thing in this project to survive a properly controlled
test: 5 of 8 clear a family-wise max-statistic bar, two at p=0.0002. Split-half
kills two of them (USDJPY dow-6 n=116 and SP500 hour-0 n=533 are thin
off-session buckets). Two HOLD with near-identical halves:
XAUUSD hour 1 +2.29 bp (t+6.44) / +2.43 bp (t+5.53)
EURUSD hour 13 -1.47 bp (t-6.80) / -0.64 bp (t-3.56)
Gold's hour 1 alone carries more than half the +4.22 bp/day drift.
And it is still not tradeable. Widening the window to amortise the 4.92 bp
round trip: the best of 144 windows (hour 1, 8h) nets +0.14 bp/day, t +0.23,
and splits +1.29 / -1.00 - the sign flips between halves. Every other window is
negative. Real, stable, well measured, and about 2x too small to cross its own
spread. Same shape as the flow result.
3 OVERNIGHT/INTRADAY - folded into the hour analysis above.
4 VOLATILITY-MANAGED DRIFT (Moreira/Muir) - the one needing no directional edge.
Does NOT reproduce here: flat on gold (-0.01), and it HURTS SP500 (0.71 -> 0.41
D1, 0.77 -> 0.54 H4). Honest negative against a strong prior.
What survives all of it is drift, which is large and significant while every
timing rule is noise: XAUUSD +10.24%/yr (t 2.88), SP500 +12.25%/yr (t 2.84),
against USDJPY +1.16%/yr (t 0.59) and EURUSD ~0.
resample.py gains D1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
resample.py composes M5 bars into M15/H1/H4 exactly - every column this pipeline
produces is composable (sums sum, maxes max, OHLC nests, means re-weight by tick
count), so this costs seconds instead of another 37-minute decode per timeframe.
Asserts tick conservation and extreme preservation on every output.
Motivation: ATR grows ~sqrt(time) while the spread does not, so spread/ATR should
fall with timeframe and make a small edge affordable. It does, monotonically, and
the measurement is clean (EURUSD, 1:1 barriers):
M5 spread 0.099 ATR random wins 36.8% cost 13.2pp
M15 0.057 39.7% 10.3pp
H1 0.029 43.6% 6.4pp
H4 0.015 47.6% 2.4pp
But the signal decays at the same rate. Rows clearing the family-wise bar:
M5 many, z to -10.1
M15 many, z to -5.8
H1 2 of 9, one POSITIVE and one negative - the shape of noise, not signal
H4 none
So the effect lives where the cost is fatal and is gone where the cost is
affordable. They never cross.
Also added --cheap=Q, which trades only the lowest-Q quantile of spread/ATR. This
is the one honest use of an unsigned feature: it cannot point a direction but it
can decline to trade, and both terms are known before entry. It does cut cost
(EURUSD M15 10.3 -> 7.6pp, USDJPY 12.7 -> 6.4pp) and the effect does not survive
there either - nothing clears the bar.
test_flow.py gained --tf= and keeps the z-score window at ~1 day on every
timeframe rather than a fixed 288 bars.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1.93 BILLION ticks -> 5.5M M5 bars (EURUSD/USDJPY/XAUUSD 2003-2026, SP500
2011-2026). Sequential non-overlapping trades, triple barriers, per-bar spread,
direction-permutation null with a family-wise max-statistic bar.
Order flow is genuinely ANTI-predictive at M5 - price mildly reverses the prior
bar's flow. Same sign on all four instruments, clearing the family-wise bar on
three:
USDJPY z -10.10 -1.73pp vs null
EURUSD z -7.95 -1.41pp
XAUUSD z -5.34 -0.68pp
SP500 z -2.60 -0.87pp (does not clear; half the sample)
Agrees with the -0.0151 next-bar correlation (vs +0.4961 same-bar, which is the
contemporaneous Cont/Kukanov/Stoikov effect and is not edge).
And the cost dwarfs it. Random entry at 1 ATR barriers after spread:
EURUSD spread 0.099 ATR -> wins 36.8% (13.2pp below the costless 50%)
USDJPY 0.154 34.9% (15.1pp)
SP500 0.292 26.5% (23.5pp)
XAUUSD 0.450 20.0% (30.0pp)
Cost rises monotonically with spread/ATR, which is an internal consistency check
on the apparatus. A ~1pp effect against 13-30pp of cost is 10-100x short.
Reversing does not rescue it - expR_rev is negative in every row of every
geometry. Widening the barriers does not either: at 4-8 ATR nothing clears the
bar (max |z| 2.51 vs 2.97). The effect lives exactly where the spread is fatal
and vanishes where the spread would be affordable, which is what a
seconds-to-minutes phenomenon predicts.
Adds --narrow/--wide geometry sets so both regimes are reproducible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first run reported -23pp edges at -75 sigma, which is not a market effect -
it is the tell this project has been burned by before (a lookahead, or here a
wrong reference, inflates whatever sign it lands on). The give-away was in the
output itself: a family-wise 5% bar of |z| > 71.67 where a centred null over 16
tests should sit near 2.5. Random entry was losing almost as badly as the signal.
Cause: z and "edge pp" were measured against be = sl/(sl+tp), the break-even of a
COSTLESS coin. These barriers charge the spread and book a loss when a single bar
spans both levels, so random entry at 1 ATR on M5 wins ~36.8%, not 50%. The table
was reporting the fixed cost of trading as if it were signal.
Now every row shows the empirical null win rate, the gap against it, and z
standardised by the null's own spread. Family-wise bar drops to 2.95 and the
result becomes legible: order flow is genuinely ANTI-predictive at M5, about 1pp
below random at z -5 to -8, clearing the bar in 12 of 16 tests and reproducing
across three geometries and two independent signal families. It agrees with the
-0.0151 next-bar correlation.
It is also untradeable, which the table now says out loud: the spread is 0.099 ATR
and costs 13pp of win rate against a 1pp effect. Reversing does not rescue it -
expR_rev is reported per row and stays negative everywhere. Added a footer stating
that beating the null is necessary but NOT sufficient; only exp R > 0 makes money.
Also:
- permutation null was allocating a single (nperm x nT) array, ~2 GB at these
trade counts. Now batched.
- null permutes the OBSERVED directions instead of flipping a fair coin, so a
directionally skewed rule on a trending instrument cannot pass on drift alone.
- timeouts reported separately rather than silently booked as stop-outs.
- calibration falls back to a midpoint sample when the tick history predates the
MT5 reference series (XAUUSD ticks start 2003-05-05, its H1 export 2004-06-11,
so the head sample overlapped by nothing). 2M ticks, because the sample must
span >=50 reference HOURS - 200k ticks of modern gold is nine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SQX decoder is a per-record Python loop and cannot be vectorised - record
LENGTH depends on the config nibbles, so record k+1's offset is unknowable
without parsing record k. It therefore saturated exactly one core: 10% CPU on a
12-core box, ~3h for the four files.
But the format is randomly seekable. Every BLOCK_LENGTH records SQX restates all
four fields as absolute int64s, so byte ranges beginning at block headers decode
with no shared history. split_offsets() cuts a file on those boundaries and
decode_iter() gained start/stop. EURUSD: 55 min -> 9.4 min, 94% CPU.
find_block() will not trust a bare MAGIC match: 0x00..0x0e is a byte run that
delta payloads produce by coincidence, so a candidate is accepted only when the
next header downstream carries the next sequential block index.
Verified equal, not assumed equal: the same 315MB span decoded serially and in 6
chunks gives identical tick counts (31,056,000), identical bar counts (206,316)
and identical OHLC. The only divergence is the documented seam artifact - the
first tick of a chunk has no predecessor so its delta counts as zero, bounded at
workers-1 ticks in 513M (~2e-8).
Two fixes this shook out:
- The feed is not perfectly time-ordered. EURUSD carries 2 backward steps in
513,494,303 ticks, both under an hour, both in 2003-2006. Bucketing is by
absolute timestamp so every tick still lands in its true bar; the symptom is a
bucket emitted twice out of order. finalise() now stable-sorts before the
duplicate merge. The ordering assert is kept but keyed to MAGNITUDE, since a
real chunking bug displaces a large fraction of rows and feed noise displaces a
handful - only one of those is safe to continue past.
- Chunk workers return undivided sums; means are divided once globally. Dividing
per chunk would weight a straddling bar's mean-of-means wrong.
test_flow.py: charge the PER-BAR spread instead of a single median across
2003-2026 - FX spreads narrowed by roughly an order of magnitude over that span,
so one median charges modern cost to the 2000s and vice versa. Timeouts are now
reported separately rather than silently booked as stop-outs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ready ahead of the bar build so the analysis runs the moment EURUSD lands.
Tests the two SIGNED features the tick pipeline can produce - tick-rule imbalance and
event-count OFI - as entry triggers at 1:1/12, 1:2/24 and 2:3/48 bar geometries. Everything
else the pipeline computes is unsigned and cannot point a trade however well it measures.
Same discipline as every other test here: signal on bar i, entry at the OPEN of i+1,
barriers scanned forward only, sequential NON-OVERLAPPING trades so each is independent
(skipping that is what produced a fake +2.66pp at 2.9 sigma earlier in this project),
break-even == chance by the gambler's-ruin identity, spread charged inside the barrier, and
a sign-flip null taking the max over the whole family for the family-wise bar.
The prior is written into the docstring before any result exists: OFI is established as a
CONTEMPORANEOUS explainer whose predictive power decays within seconds, and it reproduced
that here at +0.56 against the same-bar return. So the expectation is that it explains the
bar it is measured in and says nothing about the next. What is actually being tested is the
gap between that literature - equities, sub-second, size-weighted book data - and this
setting: retail FX CFD feed, 5-minute bars, event counts without sizes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces bidmoves/askmoves with bid_up, bid_dn, ask_up, ask_dn. A bid ticking UP and an ask
ticking DOWN both mean buy-side pressure, and a counter that only records "the bid changed"
cannot tell them apart - it throws away the direction, which is the only part that could
ever point a trade. This is order-flow imbalance in the Cont/Kukanov/Stoikov sense, in its
event-count form; the feed carries no sizes so it cannot be size-weighted.
Caught before the 3-hour build rather than after, which was the point of smoke-testing on a
bounded sample first.
Verified against the previous column set on the same 3M ticks: 21,971 bars, ticks/bar 76,
up 38, dn 38, spread 0.000126, rvol 4.800e-07, gaps 3.95/31.4 - all identical - and
bid_up+bid_dn reproduces the old bidmoves count of 74 exactly, as it must.
The orientation check that matters: OFI correlates +0.56 with the SAME-bar return. That is
the contemporaneous signature the literature reports, and it is also the cheapest guard
against the failure mode that would otherwise pass silently - a sign flip would read -0.56
and every downstream test would then be fitting the negative of the intended feature.
Expectations set in the docstring rather than discovered later: OFI is well established as
a contemporaneous EXPLAINER of price change and its predictive power decays within seconds.
At M5 with multi-hour horizons the prior should be that it explains the bar it is measured
in, not the next one. Measuring it anyway is the point - but a +0.56 contemporaneous
correlation is not evidence of an edge and must not be reported as one.
Merge/mean bookkeeping is now index-driven off COLUMNS instead of positional, so adding a
feature cannot silently mis-merge a bar that straddles a batch boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sqx.decode_iter() turns the decoder into a generator, and ticks_to_bars.py reduces a
symbol to bars in one bounded-memory pass. Necessary rather than tidy: EURUSD is ~458M
ticks, which is ~15 GB held as arrays, so nothing downstream can take the raw stream.
23 years collapses to ~2.4M M5 bars that every test can load instantly.
Aggregation is vectorised with reduceat rather than looping per tick. The only real
complexity is that a bar can straddle a batch boundary, so the last partial bar of each
batch is carried and merged into the first of the next; per-tick deltas are likewise seeded
from the previous batch's final tick, so the first tick of a batch is not silently treated
as having no predecessor. Verified against the per-tick implementation it replaces: 21,971
bars either way, and every reported median identical to the digit (ticks/bar 76, up 38, dn
38, bidmoves 74, askmoves 74, spread 0.000126/0.000250, rvol 4.800e-07, gaps 3.95/31.4).
Throughput 132k ticks/s, at which point the decoder itself is the bottleneck and the
aggregation costs ~12%.
Features are chosen by what the feed can honestly support. It carries (time, bid, ask,
volume) and no trade direction - SQX's record has one volume field and MT5's
TICK_FLAG_BUY/SELL are empty on FX - so true signed order flow does not exist here and is
not synthesised under a flattering name. What is available:
tick rule up/down mid-price changes; the standard Lee-Ready fallback
quote asymmetry bid updates vs ask updates - which side is being repriced harder
arrival rate inter-tick gaps, mean and max; urgency rather than size
realised variance sum of squared mid returns, a far better volatility estimate than
the bar range and only obtainable from ticks
spread mean and max within the bar
Of these only the tick rule and quote asymmetry can point a direction; the rest are
unsigned, like every feature that has measured above noise in this project so far.
Bars are stamped by the OPEN of their interval and built only from ticks inside it, so no
bar's features depend on a tick after it closes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads SQX tick format 4.2 in pure Python. Derived from SQX's own writer, disassembled out
of internal/libs/SQDataLib.jar (TickDataWriter, NewDataFormat{,Writter}) with the javap
bundled in the install - so this follows the format as specified rather than as guessed.
Why, when Scripts/ExportTicks.mq5 pulls the same four fields from MT5: DEPTH. The broker's
MT5 tick history covers a few years; this file starts 2011-09-19. Sample size has been the
binding constraint on every question in this project - the H1/128-bar setup yields ~300
independent trades in 18 years, enough to resolve only a +6pp edge - so 15 years of ticks
is worth a decoder.
Format: four writeUTF strings, ten zero bytes, one more writeUTF, then records of
(time, ASK, BID, volume) - ask before bid, and the writer swaps them when bid>ask so ask is
always the larger. Every BLOCK_LENGTH=1000 records: MAGIC (15 bytes 0x00..0x0e) + int32
block index + config + four raw int64s. In between, deltas against the previous record.
Config is two bytes = four nibbles laid out high-first, nibble = (logicType << 2) |
dataType, where dataType 0..3 selects a 1/2/4/8-byte payload by magnitude and logicType
supplies the sign (MINUS/PLUS carry unsigned magnitudes; ASIS is a plain signed read).
The scale is the one thing NOT in the file. SQX keeps `decimals` in external metadata, and
1216010000 is equally plausible at 10^3, 10^5 or 10^6 - nothing in the bytes distinguishes
them. Guessing would be precisely the silent, plausible-looking error this project keeps
getting caught by: a 100x price scale error crashes nothing, it just quietly rescales every
ATR-normalised feature downstream. So calibrate_decimals() matches against a known
reference series instead. Against the MT5 SP500 H1 export the answer is not marginal:
decimals=5 median rel.err 9.001630
decimals=6 median rel.err 0.004078 <-
decimals=7 median rel.err 0.899984
Verified on 4M ticks: strictly monotonic timestamps, zero negative spreads, price range
1118.03-2048.38 over 2011-09 to 2014-11 (correct for SP500), median spread 0.43 (matches
this broker's H1 record). The 0.41% residual is the expected artefact of comparing a tick
ask against the nearest H1 bar close.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Spread is the one microstructure channel that survived the API audit - FX-available, and
genuinely historical in the tester ("During testing, the spread is not modeled but is taken
from historical data"), unlike swap (no history), signed tick flow (empty on FX) or depth
of market (absent on retail FX, never replayed).
test_spread.py measures four spread features against the triple-barrier label with the same
block-permutation null as test_volume.py. spr/atr - cost relative to the volatility a trade
must overcome - is the strongest reading anywhere in this project so far: significant on 5
of 8 instrument/geometry cells and 2-4x the magnitude of any volume feature.
Which immediately looked too good, because the label is computed WITH the spread charged
inside the barriers. A bar with high spread/ATR has its barriers shifted more adversely and
is mechanically likelier to resolve as a loss - so the feature would partly predict its own
cost model, which is not tradeable information.
Tested directly by relabelling at zero cost and re-measuring the identical feature:
EURUSD 2:3 +0.000655 -> +0.000485 (p 0.030 -> 0.066, loses significance)
EURUSD 1:2 +0.000626 -> +0.000399 (p 0.003 -> 0.017)
USDJPY 2:3 +0.000418 -> +0.000164 (never significant either way)
USDJPY 1:2 +0.000876 -> +0.000532 (p 0.003 -> 0.003)
XAUUSD 2:3 +0.000769 -> +0.000744 (p 0.027 -> 0.027)
XAUUSD 1:2 +0.000876 -> +0.000698 (p 0.003 -> 0.003)
So roughly 20-40% of it WAS the tautology, and the majority is not. What remains is a
volatility-regime reading: spread is near-fixed while ATR is not, so spr/atr is high
exactly when realised volatility is running below its own ATR estimate - which genuinely
predicts whether ATR-scaled barriers get reached at all.
Note what that does and does not buy. Like volume, spread is UNSIGNED: it informs Neutral
vs directional, never Buy vs Sell. It is the best-measured feature in this project and it
still cannot pick a side.
No EA changes in this commit - measurement only, and the MQL5 side already has an
uncompiled backlog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The block fed exactly one number: (v[i] - v[i-1]) / v[i-1]. That is the first difference,
and it cannot express three things that matter - the LEVEL relative to a baseline (two
dead bars and two frantic bars both read ~0 change), and the two volume-vs-range
interactions, where heavy participation that went NOWHERE (absorption) and heavy
participation that travelled (continuation) mean opposite things and currently collapse
onto the same value.
research/test_volume.py measures each candidate's mutual information with the triple-
barrier label across 3 instruments x 2 geometries, against a BLOCK-permutation null -
blocks sized to the barrier horizon, because adjacent labels share almost their entire
outcome window and a free shuffle yields a null so tight that everything looks
significant. Finite-sample MI bias (~7/n here) is reported alongside rather than
subtracted, since the permutation null already absorbs it.
Result: volLevel beats the shipped change ratio outright on 4 of 6 cells (EURUSD 2:3
+0.000118 excess at p=0.006, USDJPY 1:2 +0.000284 at p=0.002); absorption is the single
strongest reading anywhere in the sweep at EURUSD 1:2 (+0.000404, p=0.002) though it is
null on XAUUSD; vol x range clears on 4 of 6. The shipped change ratio is itself
significant on 5 of 6, so it stays.
Kept OUT: a session-relative z-score against the same hour-of-day's own recent history.
It was the weakest candidate - null on both EURUSD cells - and it is the only one needing
per-hour rolling bookkeeping in MQL5. Not worth the state for a reading that did not
survive its own null on the primary instrument.
Magnitudes, stated plainly because they are the point: the excess MI is ~2e-4 nats against
a label entropy near 1.05. That is under a tenth of one percent of the label's
uncertainty. It is real, it repeats across instruments, and it is nowhere near an edge -
this is worth having because it costs one 50-bar loop, not because it changes the answer.
Prior work stands: the whole single-series feature family measured at the noise floor.
m_neuronsCount is already in the fingerprint, so the width change re-keys existing caches
by itself, which is correct - the input vector genuinely changed shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Transcribes all 26 classic vote models (MA 4, RSI 4, MACD 6, Ichimoku 12) from
Signals/*.mqh into vectorised Python, with their shipped constructor weights, then tests
them as entry triggers on 178k-bar FX histories.
Pre-registered by construction: the rules were written long before this test and nothing
about them is fitted here, so there is no in-sample/out-of-sample split to draw and the
whole history is usable. Break-even == chance by the gambler's-ruin identity, so "beats a
coin" and "makes money" are one question. Sequential non-overlapping trades only; a
sign-flip null over the whole pattern family gives the family-wise bar.
The result that matters is a negative one, and it took two lookahead fixes to see:
- _price_extremum reproduced the standard library's CENTRED MinValue(pos-2,5) window,
which reads up to 2 bars newer than the extremum it describes.
- turning_points marked a turn AT bar i, which is only knowable once bar i+1 closes.
Together those two bars of leakage WERE the entire apparent edge. MACD_p4 on EURUSD 1:2
read +5.05pp at +4.05 sigma before, -0.02pp at -0.02 sigma after; USDJPY 1:2 went +5.24pp
-> +0.02pp. RSI_p2's large NEGATIVE went the same way (-10.33pp -> -1.34pp), which is the
tell: a leak inflates whatever sign it lands on.
With both closed, across 4 instruments x 3 geometries: no pattern, no vote threshold, no
quorum and no event+confirmation rule separates from chance. One cell in ~180 tests stars
(SP500 2:6 vote>=30) and it is non-monotone in the threshold either side of the hit.
test_exits.py answers the trade-management half with the control that makes it mean
something: hold entries fixed, vary only the exit, and run every rule again on RANDOM
entries at the same bars. Breakeven-at-1R, chandelier trails, partials and time stops all
move E[R] - and move it by the same amount on random entries. No rule beats its own
control (max +0.99 sigma over 32 comparisons). Management reshapes the win-rate/payoff
split; it does not manufacture expectancy from a directionless entry.
Residual E[R] across every cell is -0.01 to -0.08 R, which is approximately the spread.
Incidental, both worth fixing in the EA:
- CSignalMA pattern 1 is unsatisfiable at the shipped EMA default. For an EMA,
MA[i]-MA[i-1] and c[i]-MA[i] are both positive multiples of (c[i]-MA[i-1]), so
"close below the MA while the MA rises" cannot occur. Dead code (weight 10).
- Ichimoku pattern 11 (Sanyaku, weight 100, the method's top signal) fires on 27% of
bars because it is a conjunction of three standing STATES with no event term, so it
dominates the averaged vote while carrying no trigger information.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keeps the Python that turned a per-hypothesis cost of minutes into
seconds, so the next person (or the next me) can re-run any of this
without MetaTrader in the loop.
kit.py vectorised triple-barrier labeller + scale-free feature set.
The labeller is a faithful port - stop tested before target
within a bar, so a bar spanning both scores as the loss -
and reproduces the EA's own distribution to 0.05pp
(24.93/22.01/53.06 vs 24.9/22.0/53.1) at 12x the speed.
wf.py purged, embargoed walk-forward gradient boosting.
sim.py sequential NON-OVERLAPPING trade simulation.
sweep.py the cell x geometry sweep.
detail.py full threshold profile for one configuration.
WHAT IT FOUND, and the order matters because the first answer was wrong:
Naive walk-forward looked like an edge - precision rising monotonically
with model confidence, 24.4/24.6/25.3/26.2/27.1%, topping out at +2.66pp
and 2.9 sigma. All of it pseudo-replication: a 128-bar barrier means
adjacent bars share almost their whole outcome window, so one trade was
being counted up to 128 times. Counting each trade ONCE (sim.py) the
ordering collapses to 25.9/31.1/26.8/28.1/25.8 and nothing is
significant. Same error family as the MI null that assumed independence.
That exposed a structural problem bigger than the result: at a 128-bar
horizon, 18 years of SP500 H1 yields at most ~300 independent trades,
which can only resolve a +6pp edge at 2 sigma. Real edges are 1-3pp. The
shipped configuration is not merely unproven - it is statistically
UNFALSIFIABLE on the available history.
The cost/power screen then showed SP500 is among the worst cells
available: 54k bars and spread/ATR 0.34, against EURUSD/USDJPY at 178k
bars and 0.05. Weeks of training went into the hardest instrument on the
list, 3x less data and 7x the relative cost.
Final sweep - 4 instruments x 3 geometries, purged walk-forward,
independent trades: nothing clears +2 sigma. The one survivor (EURUSD
2:3 h48, +2.76pp at +1.57 sigma) dissolves under its full threshold
profile: non-monotone across thresholds, and its per-fold win rate decays
monotonically through time (52.9 -> 45.9 -> 41.7 -> 35.2 -> 26.2).
Conclusion: with price/volume-derived technical features there is no
tradeable entry-direction edge on these instruments - now tested with a
model class that finds interactions, on 3x the data, with honest
statistics.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AnimateDread