BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The daily (4%) and total (8%) rules bound how FAST an account can lose. Nothing
noticed WHETHER it was losing. A negative-expectancy signal traded at 1% inside
that envelope breaches no rule and still arrives at zero - it just takes longer,
with every limit green the whole way down. That is the realistic way this EA
destroys an account, and no existing guard could see it.
THE ARITHMETIC THIS ENFORCES. Expected value per trade is p*TP - (1-p)*SL - cost.
With no directional edge p equals SL/(SL+TP), which is also the break-even rate,
so the payoff terms cancel exactly and EV = -cost. Expected P&L is -(trades) x
cost: strictly negative, proportional to activity. Measured here: directional
precision 23-24% against a 25% break-even, flat across every confidence tier,
with 58 points of spread on SP500. Sizing, stop placement and trailing move
variance around that mean; none of them changes its sign.
So every closed position now reports its result in R (net profit over money
actually at risk) and the running mean is tested against zero. Above the
configured minimum sample, if mean + sigma*SE < 0, new entries stop.
- SIGNIFICANTLY below, not merely below. A run of losers is ordinary variance
even for a profitable system; halting on the raw mean would be the same
act-on-noise error the MI gates exist to prevent. Using the standard error
means a wide spread simply demands more trades before the rule can fire.
- NET of swap and commission (ResolveClose already sums all three). Deliberate
and load-bearing: when the edge is zero, cost IS the expectancy, so a gross
version would measure a strategy nobody can trade.
- Reported in R so symbols, lot sizes and balances share one scale and one
mean. Trades without a stop are not scored rather than assigned a guessed R.
- LATCHED across restarts, like the daily halt and for the same reason: a
latch a reattach clears is not a latch. Clearing it means deleting the risk
state file, deliberately, after looking at why.
State is appended to the risk file length-guarded, so files written before this
still load and start their sample at zero rather than misreading.
Defaults 40 trades / 2 sigma; ExpectancyMinTrades = 0 disables it.
This does not make the strategy profitable and is not meant to. It stops paying
tuition on one the results say is losing, and does it on measurement rather than
on a drawdown limit finally being reached.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DIRECTION IS NOT THERE, and this run is what establishes it. Three symbols:
raw ASYMMETRY clears on all three (p=0.0199 / 0.0050 / 0.0050)
norm ASYMMETRY collapses on all three (p=0.3433 / 0.5075 / 0.2736),
USDCAD landing BELOW its own null
RANGE control strengthens to 3-5x its null everywhere
Divide sigma out and the apparent directional signal vanishes entirely. What
cleared was volatility leaking through an unnormalised difference. Note this
would have passed any replication test: three instruments at p=0.005 is exactly
the evidence one would accept before committing to a rebuild, and the confound
reproduces perfectly. Replication was never going to catch it - only the
normalisation could.
Two defects of mine, both surfaced by the same run.
1. THE GEOMETRY DERIVATION WAS DIVERGING, NOT CONVERGING. It produced a
14.57*ATR stop and a 29.14*ATR target that only 5.7% of bars ever reach.
Excursions were measured over the barrier horizon; the horizon scales with
the target; the target is a quantile of the excursions - so target ->
horizon -> excursions -> target ran away, and "settled" only because the
horizon ladder caps at 384 bars. A saturated runaway, which the iteration
guard could not catch because it watches for OSCILLATION.
Fixed at the root: excursions now accumulate only over m_swingMedianBars -
the UNSCALED median ZigZag leg, a property of the instrument that owes
nothing to the barrier. The barrier walk still runs the full horizon,
because that is how long the trade is held; only the MEASUREMENT used to
size the barrier is confined to a geometry-independent window.
(The Min_Risk_Reward_Ratio warning fired correctly and is what flagged it -
the diagnostic worked while the derivation behind it did not.)
2. THE CONFOUND VERDICT WAS UNREACHABLE. `sizeCleared && !asymCleared` was
tested first and is true whenever size clears - i.e. always - so the branch
that NAMES the volatility confound never printed; all three symbols showed
the generic size-not-direction message instead. Verdict chain rewritten with
the specific case first, and the dangling elses my first patch introduced
removed.
FORCES A FULL RETRAIN (the excursion window changes every derived barrier).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three;
raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500
(p=0.1045). That looked like the first directional signal this project has
found. It probably is not, and the test as built could not tell.
(up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its
null on every instrument - and the directional part is symmetric noise eps, then
up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A
pure volatility predictor scores positive MI against a 3-bin (up-dn) while
carrying no directional information at all. Crucially that confound REPLICATES,
so reproducing on two instruments is not evidence against it - and the effect
sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries
~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a
leaked fraction of the volatility signal, not an independent one.
So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only
target a directional claim may rest on. The verdict now separates the cases and
NAMES the confound when raw clears while normalised does not, instead of
reporting the raw line as a finding.
Two bugs of mine in the same block, both caught by output rather than review:
- The derived-geometry line had a MISORDERED argument list: it printed
"stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the
multiple and the multiple as the quantile. Real values were 2.61 stop /
8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen.
- THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25
"so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the
stop - hit three times in four. The printed reachability said exactly that
("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate.
This is the entire reason reachability is measured and printed rather than
assumed.
Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop
still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And
bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the
end of the sorted array.
The geometry from the previous run is NOT usable and the asymmetry result is
unresolved, not established. Both are decided by the next run.
FORCES A FULL RETRAIN (the stop quantile changes every label).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in
3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever
chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of
eleven guesses is not deriving anything.
WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It
ranks pairings by how predictable their OUTCOME is - a question about direction.
The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing
absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE
scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x.
Hence the scan failing its own gate on every run, and its "winner" wandering
2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is
strongly measurable, so derive the geometry from that instead.
stop = q25 of measured ADVERSE travel (ordinary noise does not reach it)
target = q50 of measured FAVOURABLE travel (reached ~half the time, by
construction, inside the horizon)
Continuous, in ATR units, superseding the enum multiples. Reachability ("target
on X% of bars, stop on Y%") and the implied break-even are printed so the choice
is auditable rather than trusted.
FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the
horizon with the target (first-passage time grows with the band) and the
excursions are measured OVER the horizon, so target -> horizon -> excursions ->
target is a real loop - deriving once sizes the target from travel measured
under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at
3 passes, and says so if it does not settle.
Does NOT create expectancy, and the log says as much: chance precision equals
break-even at every geometry (m/(m+k) on both sides). It buys a target the
market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio
forces a target the market rarely reaches, it WARNS rather than overriding -
the ratio is the user's risk policy, so the honest move is to state its cost.
That is the collision that once rejected 100% of setups.
Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg
files written earlier today still load (their length guard finds no doubles) and
a model that carries them was trained on them and never re-derives.
Also fixes a message from e5ceed6 that claimed "this model resumed from disk"
unconditionally - it printed above a "seeding era 0" line on a brand-new model,
because the branch fires whenever the cache is not built, which is equally true
before a fresh model's first prebuild. A diagnostic that misreports its own
trigger is worse than one that says nothing: it gets quoted back as evidence.
FORCES A FULL RETRAIN (labels change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three enums left the Inputs tab. They were three things a user had to pick and,
in the tester, three more axes for a genetic optimization to overfit.
Entry_Multiplier is pinned to MARKET. Its pending modes place the entry at a
LEVEL while the rest of the pipeline measures from the bar open - the exact
mismatch that manufactured the +0.097 R "retail fade" result later retracted as
a fill artifact. This codebase's fill model cannot honestly simulate a pending
entry, so it is no longer offered.
SL_Mode/TP_Mode become a STARTING pair. ReportBarrierGeometryScan now ADOPTS its
winner instead of printing "set SL_Mode/TP_Mode to X and retrain":
- only when it clears the family-wise gate from 04ee2e1 (beat the null of the
MAXIMUM, not merely the incumbent). This is why that gate had to land first:
without it, removing the inputs would hand a noise-picked geometry direct
control over the training target with no human in the loop - strictly worse
than the input it replaced. On SP500 H1 today it does NOT clear (p=0.1463),
so 2:6 is what you get - now chosen by measurement rather than assumed.
- only at m_eraCount == 0. Relabelling a partly-trained net moves the target
out from under weights already fitted to the old one.
THE GEOMETRY LEFT THE WEIGHTS-FILENAME HASH, because it is now measured. Same
rule that moved the horizon and the derived topology values out: a filename
keyed on a measured quantity changes the moment the measurement does - a few
more bars shift which pairing wins - and the EA then looks for a file that does
not exist, starts from era 0 and orphans a trained model silently. It is PINNED
IN THE .cfg instead: appended at the end (the only backward-safe change),
length-guarded like the 2026-07-30 derived pair, and ADOPTED on load rather than
compared, so a trained model keeps the barriers it actually learned and never
re-measures.
Two traps closed while wiring it, neither of which announces itself:
- m_barrierHorizonResolved latches the horizon ONCE PER PROCESS. Adopting 2:8
(wants ~192 bars) after it settled for 2:6 (128) would label the new target
against the old ceiling - the truncation fixed in 168422f, where every model
learned "target within 128 bars" while the EA holds to SL/TP. It lands in
Neutral, not in the timeout counter watching for it. Unlatched on adoption,
along with the label cache the old barriers filled.
- the .cfg adopt runs at init, before the horizon latches and before any label
is computed, so a resumed model has its pinned pair in place first. Verified,
not assumed.
FORCES A FULL RETRAIN: the fingerprint change orphans every existing .nnw.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates
returned exactly 0.00359 nats): the tune loop re-inits the indicators and then
scores, with no RefreshData() between.
ReInitADIndicators() does its part - Create() builds a NEW handle carrying the
new parameters, and the feature cache is flagged stale so features really are
recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and
only Refresh() copies data out of a handle into those. So every candidate was
scored on values still held from the PREVIOUS handle. My earlier guess in the
diagnostic ("suspect the feature cache") was wrong: the cache invalidation works.
Two things land together, because neither is safe alone:
1. RefreshData() after the re-init, so a candidate is scored on its own features.
2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and
the maximum of N draws from a null beats its incumbent almost every time - so
"it beat the incumbent" installs noise. This selector is the highest-stakes of
the three found in this audit because it ACTS: it overwrites the user's
configured indicator settings and forces BuildFreshTopology(), so the network
then trains on whatever the noise picked. Fixing (1) without (2) would have
made a dormant bug actively harmful.
The gate draws the winner's own permutation null once, then corrects the p-value
for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather
than the max-of-N resample used by the geometry scan because each candidate here
has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only
the one null. Exact under independence, mildly anti-conservative under positive
dependence - stated in the comment rather than hidden. A rejected winner restores
the configured settings, which best[] cannot do since the descent mutates it.
Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate()
calculates asynchronously, so if the spread is STILL zero the handles simply are
not done and the tuner needs to yield between candidates rather than score them
back to back - a state machine like the label prebuild. That distinction is now
readable from the log instead of requiring another guess.
No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A comment above the diagnostic branch says it "runs even when the sweep does
not: on a resumed model ... tying it to that gate meant the only way to see the
answer on a running model was to delete the model."
It does not. Moving the diagnostic out of the tuner's gate left it behind
m_labelCachePrebuilt, which has the same effect: the eager label pre-scan runs
only on a FRESH start, because a net loaded from disk labels lazily per bar. So
on a resumed model the flag is false forever and the whole MI block - headline,
positive control, alignment scan, lag profile, geometry scan, winner test, and
the auto-tune line - silently never runs.
Measured on SP500 H1 2026-08-07: attached at era 271, still nothing by era 314,
zero MI lines in the day's log, and the only "label cache pre-built" entry
predates the attach. It also explains the shape of every capture on 08-05/06:
each one came directly after a weights reset. The situation the comment was
written to eliminate is exactly the situation that persisted.
So drive the pre-scan when it is the only thing missing. Safe on a trained net:
its one fresh-net side effect, pushing the output-layer bias toward the dominant
class, is already gated on m_eraCount == 0, and the advance gate in Train() sits
ABOVE if(!m_trainRunActive), so the era loop keeps its state - training pauses
for the scan (~1s at 38k bars) and continues from where it was, not from 0.
Announced only on a start that actually armed, since StartLabelCachePrebuild()
returns unarmed when history is not ready and is retried per bar event.
NOT sampled from the lazily-filled cache instead: BuildMiSample skips bars with
no cached label, so that would score whichever subset training happened to have
visited - a biased subsample presented as a measurement, which is the failure
this diagnostic exists to catch.
Also corrects a claim in 0d58923's comment. It argued four consecutive "no
improvement" runs were ~1-in-100,000 evidence the indicator tuner is inert, by
multiplying 5.6% across four runs. They are not independent trials: the MI
scorer is deterministic and all four covered nearly the same bars, so an
incumbent that is the maximum on this data is the maximum on every run. One
~1-in-18 observation with three correlated repeats, ~5.6% - unremarkable. The
same independence assumption that made the uncorrected lag profile star four
lags. The candidate-spread line stands: it settles inert-vs-live directly.
No input, topology or label change: no retrain. Training in flight stays valid.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Auditing the other best-of-N scans after cccf94f turned up a third instance of
the same pattern, and this one is worse than the two already fixed: the geometry
scan and the lag profile PRINT a row, whereas TuneIndicatorsByFilter INSTALLS
its winner (Unflatten + ReInitADIndicators) and the caller then calls
BuildFreshTopology(), so an unguarded maximum changes the feature vector the
network trains on.
It has no null of any kind. But before adding one, the logs say something a
noise-driven best-of-N cannot: 2026-08-05/06, four consecutive runs, 17
candidates each, every one "no improvement" with start and best identical to
4dp. The maximum of 17 draws from a noise distribution beats its incumbent
about 94% of the time, so 4/4 is on the order of 1 in 100,000.
Two readings fit and they want opposite responses:
- INERT: trial scores come back identical to the incumbent because the
parameter change never reaches the scored features (suspect the feature
cache surviving ReInitADIndicators), so `sc > bestScore` can never fire.
That is a dead code path, and gating it would be decorating a corpse.
- LIVE and correctly finding nothing: then it needs the family-wise gate.
The current log line cannot separate them, so add the number that can: the span
of the candidate scores, with an explicit ZERO SPREAD callout naming the likely
cause. Also widened the MI figures from 4dp to 5dp - at this scale 4dp rounds
the entire effect away.
No gate yet, deliberately: measure which failure this is, then fix that one.
Read-only diagnostic. No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3271f1e tested each of ~21 lags against its OWN null at alpha 0.05 and starred
whatever cleared. That is about one false positive per run before any signal
exists, and because neighbouring lags share nearly their entire feature window
the false positives arrive in CLUSTERS that read like a hump.
It did exactly that on SP500 H1, twice in one afternoon on identical data:
13:55 nothing clears at any lag headline MI p=0.4478
16:22 k6/k10/k12/k16 starred, headline MI p=0.8756, observed
"information survives to lag 16" BELOW its own null mean
Same 31 features, same 2009 samples, same 287 blocks, cross-asset absent in
both - so this was not two different measurements. Non-replication on identical
data is the signature of an uncorrected multiple comparison, and acting on the
second run would have pinned the lookback to 17 off noise.
Galling detail: 04ee2e1 had just added exactly this correction to the
barrier-geometry scan one function below. The rigorous bar went on the report
with 6 candidates and the naive one stayed on the report with 21.
So the lag profile now uses the same construction as the geometry winner test:
one draw from every lag, keep the largest, repeat; a lag clears only by beating
that distribution. Draws centred leave-one-out to match how the observed excess
is centred. Independence across lags overstates the spread of the maximum
(neighbours share their window), so it errs toward rejecting.
Also: the positive branch now says to re-run before acting, because one run of
this report has demonstrably not been a result; and MI_LAG_MAX_PROFILE caps the
retained-draw matrix rather than trusting a derived m_historyBars.
Read-only diagnostic. No input, topology or label change: no retrain, and a
training run already in flight stays valid.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain".
That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two
numbers, with no test that either is distinguishable from zero.
bestExcess is a MAXIMUM over the eligible candidates. The maximum of several
draws from a null sits well above any single draw from it, so a max-shaped
statistic tested against a single-candidate null crowns a winner on noise
almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the
lag profile committed in 3271f1e measures the pure-noise swing on this exact
data at +/-0.0004, peaking at +0.00042 with nothing clearing its own null at
any lag. The advisory was one ratio away from talking us into relabelling and
retraining all four topologies to chase that.
So build the null OF THE MAXIMUM: retain every candidate's permutation draws,
take one draw from each candidate, keep the largest, repeat. The winner must
beat that distribution.
- draws centred LEAVE-ONE-OUT, so a draw is centred by a mean excluding it -
exactly how the observed score is centred. Centring a draw by a mean that
contains it shrinks it toward zero and would deflate the null.
- only ELIGIBLE candidates enrol: the family the max was taken over is the
family to correct for, and a clamped or sub-minRR pairing can never win.
rrOK hoisted above the draws for this.
- draws per candidate 20 -> MI_GEOMETRY_PERMUTATIONS (40): they now have to
resolve an upper tail, which is where 20 draws are thinnest.
- MI_GEOMETRY_ALPHA 0.05, stricter than the lag profile's: a wrong lookback
costs input width, a wrong geometry costs a full retrain from era 0.
Independence across candidates overstates the spread of the max (the real
candidates share features and overlapping label windows), so the gate errs
toward rejecting - the safe direction when passing costs a retrain.
Read-only diagnostic. No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BuildMiSample samples features from ONE bar. So every "MI is at the noise
floor" result this codebase has produced - including yesterday's p=0.18 on
SP500 H1 - described the ENTRY BAR's 31 features only, while the network is
fed 20 bars of them. If information lived at lag 7 and not lag 0, the report
would have said "no signal" while the model could still learn. The diagnostic
we have been making decisions on had a blind spot exactly the width of the
input vector.
Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as
the existing labelBarOffset and is not interchangeable with it. Shifting the
LABEL changes which trade is predicted, so at any non-zero offset the
features sit inside the labelled window and the score is lookahead - that is
precisely what the alignment scan measures and correctly reports (4.7x more
knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label
pinned to the entry bar, so every row stays causal.
ReportFeatureLagProfile() then scores k = 0..historyBars against the same
block-permutation null and reports the deepest lag that clears it - the
lookback the data supports, versus the 20 that was picked by hand and never
measured. The null is redrawn PER LAG: finite-sample MI bias moves with the
realised class counts and bin occupancy, and different rows survive the
validity checks at each lag, so one shared floor would be right for lag 0 and
wrong everywhere else. Draw count is reduced accordingly (40, not 200) since
cost is draws x historyBars; this figure decides a lookback, never a trade.
MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that
makes two builds comparable row by row.
Read-only - no input, topology or label change, so no retrain. Both builds
0/0. Build tag lag-profile-v1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Implemented sqx_audit.py to audit StrategyQuant X trade lists, focusing on performance metrics and cost analysis.
- Created sqx_portfolio.py to evaluate portfolio performance based on uncorrelated components and their impact on risk and return.
- Developed swing.py to analyze cost ratios across different holding periods and assess swing trading structures.
- Introduced test_management.py to investigate the effectiveness of exit rules on random entries and their impact on expectancy.
The MI sample builder used `MathAbs(labelBarOffset)` as a padding, causing rows from offset and non-offset builds to be paired with a double shift. This broke the positive control, failed the 5× gate, and voided all reported mutual‑information figures. Replace with the fixed `MiShiftPad` constant to ensure builds enumerate the same set of bars and row-k alignment is preserved.
Add `BatchOptionsTotal()` to `CNeuronBatchNormOCL` and split the packed BN weight array in the learning report into separate norms for the outgoing dense matrix, gamma, beta, running statistics, and Adam moment buffers. This turns an ambiguous single‑norm reading into precise diagnostics that distinguish weight divergence from scaling issues.
- BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name
and add buffer index/element count to all error prints for easier debugging.
- NetPersistence: distinguish missing file from transient lock by probing
FileIsExist before logging, eliminating false "sharing violation" warnings
when no saved model exists on first run.
Three results today were invalidated by the same family of error: a price level
used as an entry while the outcome was measured from somewhere else. The defence
is not vigilance - it is an interface where the mistake cannot be expressed.
bidask.py M1 bars carrying SEPARATE BID and ASK OHLC, built from the tick
stream in one parallel pass. Every existing bar file stores MID, which is fine
for measuring returns and useless for simulating orders, because no order ever
executes at the mid. Mid bars force the spread to be bolted on afterwards as an
average - the approximation that let today's artifacts through. With both sides
carried, the spread is whatever it actually was, including the overnight and
news blowouts an average hides. Fails loudly if ask < bid.
fills.py owns the ENTIRE trade lifecycle; no test may open a position any other
way. Longs enter at the ask and exit at the bid, shorts the reverse. A buy stop
triggers on the ask and fills at the trigger price OR the bar's open if the bar
gapped past it, which is where real slippage comes from. Limits fill on the
opposite side and a gap is capped in their favour.
The invariant that was violated: THE OUTCOME CLOCK STARTS AT THE FILL BAR. The
fill index IS the start index - they are the same variable and cannot diverge.
That is what went wrong before, and it is now unrepresentable rather than
merely discouraged.
Same-bar ambiguity is REPORTED, not assumed away: every result carries the
fraction of trades decided by a bar containing both barriers, alongside fill
rate and unresolved rate. A resolution-limited result now says so itself.
M1 rather than raw ticks is a deliberate, stated bound: 513M ticks per symbol is
~8 GB packed and four symbols will not fit in memory, while M1 keeps it at ~500 MB
and cuts the residual ambiguity 60x versus H1. It is an approximation with a
visible error bar, not an exact simulator.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Applies the same five-trace context score from c998d65 to a completely
different entry - the LPS retest instead of the shakeout - as a check that the
dose-response was not specific to one setup.
shakeout trigger slope +0.0464 R/trace t +2.16 8/8 cells positive
LPS trigger slope +0.0425 R/trace t +3.18 7/8 cells positive
Two unrelated triggers, 31,000 non-overlapping trades, same magnitude to within
10%, same sign in 15 of 16 cells. This is the strongest confirmed result in the
whole programme: reading past structure to qualify current structure has real,
measurable predictive content. Villahermosa is describing something that exists.
AND IT IS STILL NOT ENOUGH. The buckets:
1 agree -0.287 3 agree -0.151
2 agree -0.206 4 agree -0.157
Context is a MODIFIER worth ~+0.045 R per agreeing trace. Every trigger it can
modify starts between -0.15 and -0.33, and only five traces exist, so full
confluence still lands short of break-even. Reaching zero would need ~7.
One cell goes positive - EURUSD H1 at 4 agreeing traces, +0.14 on n=133 - and it
is 1 of 40, on the smallest bucket, in the instrument with the lowest costs. That
is the exact shape of the three artifacts already retracted today, so it is
recorded as noise unless a fill-correct simulator says otherwise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Earlier tests fired on the shakeout alone, which is not the method. Book 2 2.3
treats it as the third of four cumulative traces and reads the structure's own
history first. This scores all of them, oriented to the shakeout's direction:
1 Phase A test location (upper vs lower half of the structure)
2 Phase B test location
2b STRUCTURAL FAILURE - after the Phase B test, did price fail to reach the
opposite extreme
4 effort/result on the shakeout bar (close position + volume vs range average)
7.1 higher-timeframe context - is the larger move in the shakeout's favour
Conditioning on agreement shrinks the sample and multiplies the ways to slice
it, so the test is NOT 'find the combination that works'. It is the one
pre-specified prediction the books make and mining does not: expR must rise
MONOTONICALLY with the number of agreeing traces. One slope, no threshold to
tune, no best cell to pick.
POOLED (16,234 non-overlapping trades):
0 traces -0.332 3 traces -0.114
1 trace -0.214 4 traces -0.117
2 traces -0.184
slope +0.0464 R per agreeing trace, t +2.16
per-symbol slopes POSITIVE IN ALL 8 CELLS (p ~ 0.004 on sign alone)
So the context logic is real and measurable - it is not folklore. But the base
trade is in too deep a hole for it to matter: full confluence still returns
-0.117, and reaching break-even would need ~7 agreeing traces when only 5 exist.
The useful reading is that context is a MODIFIER worth about +0.05 R per trace,
which is only interesting when bolted to a trigger whose base expectancy is
already near zero. The shakeout's is not, because its structural target sits
4-6R away and is rarely reached.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the programme on both books. Entries are MARKET ORDERS at a bar's
open throughout, so the fill-timing artifact that invalidated the last round is
designed out rather than remembered. Benchmark is analytic: entry, stop and
target fixed at entry means a driftless market gives expR = 0 exactly.
1. SPRING / UPTHRUST (book 1 ch.18, the event 'all Wyckoff operators wait for').
Pierce of a COMPRESSION-QUALIFIED range edge, close back inside, stop beyond
the shakeout extreme, target the far side of the range.
30 cells across 4 symbols x M15/H1/H4. Reward-to-risk averages 4-6:1, so the
break-even win rate is only 15-20%, and it still loses nearly everywhere:
M15 all four symbols -0.15 to -0.24 with 0/4 folds positive. Best cell is
EURUSD H1 climactic-volume +0.302 at t +2.27, which over 30 cells is inside
the family-wise band.
The books' volume requirement was applied - climactic (>1.5x range average)
and quiet (<0.8x) shakeouts scored separately. Neither rescues it.
2. LPS / LPSY, the test-after-breakout, and book 2's A/B (5.7.1, 5.8.3): it
claims the retest should be awaited at the VOLUME PROFILE level, not the
price edge. Same breakout, same stop, same 2R target, only the location
differs:
retest at typical expR
A price edge -0.041 .. -0.315
B value-area edge -0.128 .. -0.413
C range VPOC -0.129 .. -0.506
24/24 cells negative, and A > B > C in ALL EIGHT symbol/timeframe
combinations. That monotone ordering is not noise, and it inverts the book's
recommendation. Mechanism is adverse selection: the VPOC sits deep inside the
old range, so a retest that reaches it is disproportionately a breakout that
has already failed. The deeper the level you wait at, the more your fills are
selected against you.
Practical consequence: the volume profile is real (levels beat distance-matched
placebos at z +3 to +7.8) but using it to LOCATE ENTRIES makes this trade
worse, not better.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The +0.097 R EURUSD result in c9b489e is wrong. So is the +0.108 R pooled edge
in c2dd9eb and the selection model in 821f16d, which used the same labels.
THE BUG. The mirror test entered at the retail trigger price e2 - a STOP order
level - but started the outcome race at the OPEN of the fill bar. Price at that
open is on the far side of e2 by construction; that is why the order is a stop
order. So the race began before price had reached the entry, which handed the
fade a free run toward its target and pushed its stop further away than it
really was. Retail's side carries the same bias with the sign reversed, so the
DIFFERENCE - which is exactly how 'edge' was computed - was inflated twice over.
Found by generalising the trigger to swing-extreme breakouts, per the user's
suggestion. That version returned +0.7 R at t +110, which is not a result, and
it has the identical structure: enter at a level, measure from the bar open.
WITH AN HONEST INTRABAR FILL (first M5 bar that actually trades at the entry),
every EURUSD cell inverts:
bar open honest fill
H1 pin +0.0953 -0.0096
H1 pin +0.0559 -0.0573
H1 inside +0.0612 -0.0395
M15 pin +0.0407 -0.0135
The 4/4 walk-forward held because the bias was present in every fold. A
walk-forward validates against regime change, not against a broken fill model.
The tell was there and I walked past it: sweep_entry() was the ONE test that
modelled the fill properly, and it was the ONE test that came out negative. When
one arm of a suite disagrees with the rest, check what it does differently before
believing the majority.
So the standing conclusion returns to what it was: retail setups are close to a
coin flip that pays the spread, and there is nothing in them to harvest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reframes what the net is asked. Direction is the one thing the data says is
barely predictable; but the fade of retail pin/inside setups has a MEASURED
gross edge of ~0.139 R and a cost of spread/stop that varies 5x across
instruments, sessions and vol regimes. So the answerable question is selection:
spend the edge only where it survives.
label realised R of the fade (what actually pays)
features 28, all knowable at entry, incl. the volume-profile levels
split CHRONOLOGICAL train/val/test; threshold picked on val, frozen for test
TEST BLOCK 2022-08 .. 2026-07, 63,901 candidate trades, keep top 10%:
mean R +0.1005 vs -0.0616 for taking everything
by year +0.140 +0.075 +0.098 +0.105 +0.105 -> 5/5 positive
with the COST FEATURE REMOVED: +0.0800, still 5/5 positive
That last control matters: the model is not merely learning 'skip wide
spreads'. Something in the setup geometry, session and level structure carries
signal beyond the cost.
HONEST SIGNIFICANCE. The naive t of +8.06 is not believable - with an 8-day
horizon these trades overlap heavily and thousands share one price path. On a
strictly non-overlapping subset (163 independent trades) it is +0.123 R at
t +1.61; without the cost feature, +0.257 at t +3.45. The overlap filter applies
the H1 horizon to M15 trades too, so 163 is a conservative floor and the true
independent count is higher. Suggestive, not settled.
THE LEAK THIS RUN NEARLY SHIPPED. First version scored +0.53 R on the held-out
block, t +54. Bar-derived features were read at the FILL bar i2, but the order
fills intrabar and the outcome race starts at the first M5 bar inside i2 - so
i2's close, tick count and realised volatility are not knowable at entry. The
model was seeing how the bar it entered on turned out. Second lookahead of this
hunt (the first was worth +0.15 R in the sweep test).
RULE: a clean chronological split does NOT protect against lookahead. The split
was honest and the features were not. Any feature indexed at the entry bar must
be re-derived from the bar before it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit pooled four instruments with very different spread-to-stop
ratios and concluded the edge never beats the cost. That was too broad. Per
cell (48 cells, M5 path, k=1), 8 clear - and they are not scattered:
EURUSD H1 pin spread/stop 0.042 edge +0.160 cost 0.063 -> +0.097 R
EURUSD H1 pin 0.045 +0.127 0.067 +0.060
EURUSD H1 inside 0.047 +0.123 0.060 +0.063
EURUSD H1 inside 0.050 +0.102 0.062 +0.040
EURUSD M15 pin 0.074 +0.141 0.100 +0.042
EURUSD M15 pin 0.072 +0.140 0.099 +0.041
Every clearing cell is on the tightest-spread instrument. XAUUSD carries the
same gross edge (+0.09 to +0.135) and never clears, because its cost is 3x.
That is the mechanism predicting where the effect should survive and being
right - the opposite of the stop-run case, which inverted.
WALK-FORWARD, 4 chronological folds: 6 of 8 hold at >=3/4. EURUSD H1 pin short
side is +0.116 / +0.061 / +0.143 / +0.067 across 23 years, 4/4.
WIDENING THE STOP still says what it said: EURUSD H1 pin goes +0.078 (m=1) ->
+0.028 -> +0.009 -> +0.017 -> -0.003 (m=5). The gross edge collapses ~15x while
the stop widens 5x, so this is NOT drift - it is reversion inside roughly one
setup-risk of a stop order filled at a local extreme. It is only tradeable at
the tight stop, which is exactly where cost bites hardest.
WHAT IS NOT MODELLED, and it decides this: commission and stop slippage. Gross
edge is ~0.139 R = ~2.4 pips on a 17.3-pip stop, against 0.75 pips of spread.
That leaves ~1.6 pips of headroom for commission plus slippage before it is
gone. A demo forward test measuring both is the next step, not more history.
Also fixes a LOOKAHEAD found in the sweep-entry test: the protective stop was
anchored to the low of the very bar that filled the limit order, which is not
known until that bar closes. It was worth ~+0.15 R - larger than any real effect
here - and it inflated the placebo equally, which is how it was caught. With it
removed, buying at retail stop levels is no better than buying at an arbitrary
level the same distance away: the 'stops are a farmable magnet' claim fails its
own control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
40 MB of third-party copyrighted books sit in references/ so the research
scripts can read them. Untracked is not the same as safe: one broad
`git add -A` puts them in history permanently and on a public remote.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tests the user's thesis directly: if price is unpredictable, trade against the
people predicting it badly. Implements the three mechanical setups from 'How To
Day Trade Forex For Profit' ch.5 with their DOCUMENTED stop rules, so retail
stops are located exactly rather than by proxy.
THE MIRROR TEST. Retail's trade and its exact mirror, priced under identical
rules. Both sides pay the same spread and suffer the same same-bar tie
convention, so those cancel in the difference and double in the sum:
edge = (mirror - retail)/2 cost = -(mirror + retail)/2
pin EDGE +0.108 R COST +0.143 R
inside EDGE +0.068 R COST +0.140 R
engulf EDGE -0.001 R COST +0.095 R
So pin-bar and inside-bar setups really are anti-predictive - the first
confirmed directional edge in this project. Engulfing is a pure coin flip whose
loss is entirely the spread, i.e. money already gone to the broker.
Stable across three conventions: H1 bars pessimistic ties, M5 path pessimistic,
M5 path optimistic. Re-walking the barriers on M5 CUT the cost (0.195 -> 0.143)
and RAISED the edge (0.078 -> 0.108), so the coarse-bar convention was masking
the effect, not manufacturing it.
THEN THE TEST THAT KILLS IT. Cost in R is spread/stop-distance, so widening the
stop divides it. If the edge is directional drift it survives. Fade expR by stop
multiple (pin, k=1, 122k trades):
m=1.0 cost 0.146 expR -0.045 implied edge +0.101
m=1.5 cost 0.097 expR -0.067 +0.030
m=2.0 cost 0.073 expR -0.065 +0.008
m=3.0 cost 0.049 expR -0.051 -0.002
m=5.0 cost 0.029 expR -0.040 -0.011
The edge decays exactly as fast as the cost, then inverts. It was never drift:
it is reversion against a stop order filled AT a local extreme, and it lives
within one bar-range of the entry - the same short-horizon reversal the tick-flow
work already measured, meeting the same fate.
Also in this commit, the volume-profile claims from Wyckoff 2.0:
MAGNET all 10 tests positive vs a distance-matched placebo, z +3.0 to +7.8,
family-wise bar 2.79 - but the effect is +0.15 to +0.59pp on a ~74%
base rate, i.e. ~0.01 R.
REACTION naked VPOC and VPOC reject +0.71 to +0.86pp (z to +5.65); value-area
edges null or negative; HVN/LVN marginal.
80% RULE dead. 27.8% traversal against a 29.0% martingale benchmark. Acceptance
nearly DOUBLES the raw rate (14.1% -> 27.8%) and the benchmark doubles
with it - every bit of the apparent improvement is geometry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 1:1 range projection is the target rule both books recommend (book 1 ch.8
discards point-and-figure counting as too subjective and keeps the vertical
projection). Tested as a complete trade on 4 symbols x M15/H1: enter on the
range breakout, stop at the far side of the range, target k x risk.
A driftless market gives P(win) = 1/(1+k) and expR = 0 at EVERY k, so the
benchmark here is analytic - no permutation null needed. Result: expR sits on
that benchmark at every k on every symbol. Target placement does not move
expectancy, which is what a martingale already said.
But the law itself is measurable, and it is not 1:1. Regressing
log(MFE) on log(range height) with log(ATR) as a FREE regressor (a shared ATR
denominator correlates the errors and biases the exponent towards the
hypothesis, so it cannot be used to argue against it):
b = 0.10 .. 0.92, centred ~0.6
b = 1 rejected in 5 of 8 at >2sd, never significantly above 1
b = 0 rejected in 7 of 8
So a bigger cause does produce a bigger effect - sub-proportionally. The 1:1
projection systematically over-reaches after a large consolidation and
under-reaches after a small one. Median travel in risk-multiples falls
monotonically across height quartiles in 8 of 8 runs.
Also adds the volume-profile machinery the second book is built on and which
nothing in the EA has: tick-level volume-at-price on a fixed absolute grid,
per-session VPOC/value area by the standard Market Profile walk, naked VPOCs,
and HVN/LVN from a rolling causal composite.
Two biases are left in deliberately, both against the hypothesis: a bar
spanning stop and target books the loss, and spread is charged on entry and
both barriers. Unresolved trades are marked to market at the horizon rather
than discarded - discarding them deletes slow winners and manufactures a
false deficit at large k.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The statistics had cleared a family-wise bar (z to +6.56) and a split-half. Both
necessary, neither sufficient: thorough enough mining passes both. What mining
cannot do is obey a mechanism it was never fitted to - so the decisive test is
whether the effect appears WHERE THE THEORY SAYS IT MUST.
Osler's stop-clustering predicts the edge concentrates where the stop reservoir is
deepest and fresh forced orders arrive: London open and the London/NY overlap.
Measured, by session in real UTC (broker is UTC+2):
EURUSD Sydney/Tokyo +0.197* London open -0.063 London -0.156
USDJPY Rollover +0.233 London open -0.223
XAUUSD Sydney/Tokyo +0.070 London open -0.345 LDN/NY -0.185
SP500 Rollover +0.206 NY afternoon -0.098
Exactly inverted. The liquid sessions where stops actually cluster are the worst
on every instrument; what remains lives in Sydney/Tokyo and rollover - the
THINNEST hours, where fewest stops sit. That is not the mechanism, and thin hours
are also where spreads are widest and fills worst, so even the surviving fragment
points away from tradeability rather than toward it.
Walk-forward by quarter agrees, and shows what the two-way split was hiding:
EURUSD 2/4 positive (Q1 -0.029, Q2 +0.062, Q3 -0.010, Q4 +0.089)
USDJPY 1/4 XAUUSD 0/4 SP500 2/4
No instrument reaches 3/4. The split-half HOLDS was Q2+Q4 carrying Q1+Q3.
Verdict: not an edge. Recording it as disconfirmed rather than leaving an
encouraging half-result in the log, because the next person to read this - me -
would otherwise build on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Osler's currency-order-flow work is the published mechanism: stop-loss orders
cluster just beyond recent swing extremes and cascade price; take-profits cluster
and reverse it. So the claim is not "a pattern repeats" but "there is a reservoir
of forced orders at a location computable in advance" - which is falsifiable in a
way chart patterns are not.
Tested as a COMPLETE trade rather than a signal with barriers bolted on: entry,
stop and target all come from one structure. Price takes out the N-bar extreme
MARGINALLY (<= over*ATR), closes back inside, enter the opposite way next open,
stop just beyond the sweep extreme (where the liquidity actually was), target a
multiple of that risk.
19 of 24 configs clear a family-wise max-statistic bar on EURUSD/USDJPY H1, all
in the predicted direction, z to +6.56. R=1 configs are negative and R=2/R=3 turn
positive, which is coherent: the edge is directional and a tight stop pays the
spread as a large fraction of risk, so it needs a big R to clear.
SPLIT-HALF then kills most of it, as it should:
EURUSD N=50 ov=0.5 R=3 +0.013 / +0.042 HOLDS
EURUSD N=20 ov=0.5 R=3 +0.014 / +0.034 HOLDS
every R=2 config one half negative
USDJPY one half negative
Surviving configs are STRONGER in the second half, the opposite of a mined
artifact decaying out of sample. But N=20 and N=50 overlap heavily and are not
independent, so this is one instrument and one R - a lead, not a system.
dukas.py: direct Dukascopy datafeed client. SQX mirrors through its own CDN
(CdnCache/CdnDownloadJob) so there is nothing reusable there. Dukascopy publishes
the raw feed - bi5, raw LZMA, 20-byte big-endian records, ZERO-BASED MONTH in the
URL (fails silently into the wrong month otherwise). Cached, resumable, bounded
concurrency.
Two corrections it forced, per the user: SQX conforms Dukascopy data to the5ers'
broker profile AND timestamps. Measured empirically, broker time = UTC+2 (EET),
clean minimum. So (1) previously reported "hours" are BROKER time - gold's hour 1
is 23:00 UTC, the daily rollover and COMEX Globex reopen, a real mechanism; and
(2) Dukascopy's raw 0.2-pip ECN spread must NOT be used for cost - the5ers' ~0.47
is what is actually paid, so the existing cost analysis was right and Dukascopy
would have made every result look falsely tradeable. Its value is the bid/ask
VOLUMES, which SQX lacks entirely - true signed flow instead of the event-count
proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything tested before this asked ONE question - can recent price or flow
predict the next bar's direction - and answered no four ways. These are different
families, each with a published prior rather than a hunch.
1 TIME-SERIES MOMENTUM (Moskowitz/Ooi/Pedersen). 34 configurations across 4
symbols x D1/H4 x 6 lookbacks. Nothing. The one rule that looks strong -
XAUUSD H4 250-bar, p=0.0038, t+3.30, +10.32%/yr - returns essentially exactly
buy-and-hold's +10.36%. It is not timing gold, it is being long gold. Hence the
vs-B&H column: on a drifting asset a rule that is merely long most of the time
looks skilful and is not.
2 SEASONALITY. The first thing in this project to survive a properly controlled
test: 5 of 8 clear a family-wise max-statistic bar, two at p=0.0002. Split-half
kills two of them (USDJPY dow-6 n=116 and SP500 hour-0 n=533 are thin
off-session buckets). Two HOLD with near-identical halves:
XAUUSD hour 1 +2.29 bp (t+6.44) / +2.43 bp (t+5.53)
EURUSD hour 13 -1.47 bp (t-6.80) / -0.64 bp (t-3.56)
Gold's hour 1 alone carries more than half the +4.22 bp/day drift.
And it is still not tradeable. Widening the window to amortise the 4.92 bp
round trip: the best of 144 windows (hour 1, 8h) nets +0.14 bp/day, t +0.23,
and splits +1.29 / -1.00 - the sign flips between halves. Every other window is
negative. Real, stable, well measured, and about 2x too small to cross its own
spread. Same shape as the flow result.
3 OVERNIGHT/INTRADAY - folded into the hour analysis above.
4 VOLATILITY-MANAGED DRIFT (Moreira/Muir) - the one needing no directional edge.
Does NOT reproduce here: flat on gold (-0.01), and it HURTS SP500 (0.71 -> 0.41
D1, 0.77 -> 0.54 H4). Honest negative against a strong prior.
What survives all of it is drift, which is large and significant while every
timing rule is noise: XAUUSD +10.24%/yr (t 2.88), SP500 +12.25%/yr (t 2.84),
against USDJPY +1.16%/yr (t 0.59) and EURUSD ~0.
resample.py gains D1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First live run exposed both problems at once. It logged "+83 release(s)
recorded", and every one of those rows shared a single observed_time up to 30
hours after its event_time: they were the startup backfill, not release-time
observations. Their actual figures are whatever the terminal holds NOW - the
post-revision values this recorder exists to avoid - and the very first batch
proved that is not hypothetical: a Retail Sales row came back previous 3.5 /
revised_prev 3.4, and a Core CPI row already carried revision=1.
Backfill is still worth keeping (a fine snapshot of the revised series, and it
carries the event metadata) but must never be silently mixed with release-time
observations. Every row now records lag_sec and a capture class, so the
distinction cannot be lost by whoever loads the CSV later:
LIVE observed within InpLiveLagSeconds (default 600s) of release
BACKFILL seen long after the fact - MUST NOT be used for surprise research
The log now reports the split per poll and says so explicitly when a poll is
entirely backfill.
Second and worse, in LoadSeen: the FILE_CSV field walk was off by one and keyed
the seen-set on event_id instead of value_id. event_id identifies the event TYPE,
not the release, so after any restart every future release of every event already
in the file would have been skipped - permanently, and silently, exactly for the
recurring high-importance events (NFP, CPI) that matter most. Now reads whole
lines and indexes a split array by a NAME-CHECKED column position, which cannot
drift when the schema changes. Refuses to guess if value_id is absent.
Schema change is handled by rotating any file with a non-matching header to
<name>.<timestamp>.old rather than appending, since mixing layouts mis-parses
every old row.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The calendar is the only non-price source MQL5 carries with real content, and it
is the one this project cannot research from history:
MqlCalendarValue.actual_value returns the CURRENT figure, i.e. after every later
revision. Reading 2019's NFP today returns a number nobody could have known in
2019, so any surprise = actual - forecast feature built from history carries
lookahead - and the flattering kind, since it makes a model look most prescient
exactly on the events that were revised most.
The only sound fix is to write down what the terminal reported at the moment of
release and never touch that row again. Write-once is the entire contract here: a
row is appended the first time a value_id is seen carrying an actual figure, and
is never rewritten, because re-recording on a later poll would silently import
the revision this file exists to avoid. The seen-set is rebuilt from the file on
init so a restart cannot duplicate or re-import either.
Deliberately standalone - no includes from the EA tree, and not wired into
Warrior_EA. It may run for months on a spare chart, and coupling it to the
trading system would mean a refactor there can stop the recorder; a gap in a
write-once series cannot be backfilled by definition. It also keeps a
data-collection task from adding any failure mode to a system about to trade a
prop account.
Records actual/forecast/previous/revised_previous, revision number, impact,
importance, units, plus observation time and the quote at observation. Every
FileOpen carries FILE_SHARE_READ|FILE_SHARE_WRITE per the rule this codebase
learned the hard way (exclusive opens fail 5004 and look like "no data").
Compiles 0 errors, 0 warnings. Value starts at zero and accrues with time, which
is the argument for starting it now rather than when it is wanted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
resample.py composes M5 bars into M15/H1/H4 exactly - every column this pipeline
produces is composable (sums sum, maxes max, OHLC nests, means re-weight by tick
count), so this costs seconds instead of another 37-minute decode per timeframe.
Asserts tick conservation and extreme preservation on every output.
Motivation: ATR grows ~sqrt(time) while the spread does not, so spread/ATR should
fall with timeframe and make a small edge affordable. It does, monotonically, and
the measurement is clean (EURUSD, 1:1 barriers):
M5 spread 0.099 ATR random wins 36.8% cost 13.2pp
M15 0.057 39.7% 10.3pp
H1 0.029 43.6% 6.4pp
H4 0.015 47.6% 2.4pp
But the signal decays at the same rate. Rows clearing the family-wise bar:
M5 many, z to -10.1
M15 many, z to -5.8
H1 2 of 9, one POSITIVE and one negative - the shape of noise, not signal
H4 none
So the effect lives where the cost is fatal and is gone where the cost is
affordable. They never cross.
Also added --cheap=Q, which trades only the lowest-Q quantile of spread/ATR. This
is the one honest use of an unsigned feature: it cannot point a direction but it
can decline to trade, and both terms are known before entry. It does cut cost
(EURUSD M15 10.3 -> 7.6pp, USDJPY 12.7 -> 6.4pp) and the effect does not survive
there either - nothing clears the bar.
test_flow.py gained --tf= and keeps the z-score window at ~1 day on every
timeframe rather than a fixed 288 bars.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1.93 BILLION ticks -> 5.5M M5 bars (EURUSD/USDJPY/XAUUSD 2003-2026, SP500
2011-2026). Sequential non-overlapping trades, triple barriers, per-bar spread,
direction-permutation null with a family-wise max-statistic bar.
Order flow is genuinely ANTI-predictive at M5 - price mildly reverses the prior
bar's flow. Same sign on all four instruments, clearing the family-wise bar on
three:
USDJPY z -10.10 -1.73pp vs null
EURUSD z -7.95 -1.41pp
XAUUSD z -5.34 -0.68pp
SP500 z -2.60 -0.87pp (does not clear; half the sample)
Agrees with the -0.0151 next-bar correlation (vs +0.4961 same-bar, which is the
contemporaneous Cont/Kukanov/Stoikov effect and is not edge).
And the cost dwarfs it. Random entry at 1 ATR barriers after spread:
EURUSD spread 0.099 ATR -> wins 36.8% (13.2pp below the costless 50%)
USDJPY 0.154 34.9% (15.1pp)
SP500 0.292 26.5% (23.5pp)
XAUUSD 0.450 20.0% (30.0pp)
Cost rises monotonically with spread/ATR, which is an internal consistency check
on the apparatus. A ~1pp effect against 13-30pp of cost is 10-100x short.
Reversing does not rescue it - expR_rev is negative in every row of every
geometry. Widening the barriers does not either: at 4-8 ATR nothing clears the
bar (max |z| 2.51 vs 2.97). The effect lives exactly where the spread is fatal
and vanishes where the spread would be affordable, which is what a
seconds-to-minutes phenomenon predicts.
Adds --narrow/--wide geometry sets so both regimes are reproducible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first run reported -23pp edges at -75 sigma, which is not a market effect -
it is the tell this project has been burned by before (a lookahead, or here a
wrong reference, inflates whatever sign it lands on). The give-away was in the
output itself: a family-wise 5% bar of |z| > 71.67 where a centred null over 16
tests should sit near 2.5. Random entry was losing almost as badly as the signal.
Cause: z and "edge pp" were measured against be = sl/(sl+tp), the break-even of a
COSTLESS coin. These barriers charge the spread and book a loss when a single bar
spans both levels, so random entry at 1 ATR on M5 wins ~36.8%, not 50%. The table
was reporting the fixed cost of trading as if it were signal.
Now every row shows the empirical null win rate, the gap against it, and z
standardised by the null's own spread. Family-wise bar drops to 2.95 and the
result becomes legible: order flow is genuinely ANTI-predictive at M5, about 1pp
below random at z -5 to -8, clearing the bar in 12 of 16 tests and reproducing
across three geometries and two independent signal families. It agrees with the
-0.0151 next-bar correlation.
It is also untradeable, which the table now says out loud: the spread is 0.099 ATR
and costs 13pp of win rate against a 1pp effect. Reversing does not rescue it -
expR_rev is reported per row and stays negative everywhere. Added a footer stating
that beating the null is necessary but NOT sufficient; only exp R > 0 makes money.
Also:
- permutation null was allocating a single (nperm x nT) array, ~2 GB at these
trade counts. Now batched.
- null permutes the OBSERVED directions instead of flipping a fair coin, so a
directionally skewed rule on a trending instrument cannot pass on drift alone.
- timeouts reported separately rather than silently booked as stop-outs.
- calibration falls back to a midpoint sample when the tick history predates the
MT5 reference series (XAUUSD ticks start 2003-05-05, its H1 export 2004-06-11,
so the head sample overlapped by nothing). 2M ticks, because the sample must
span >=50 reference HOURS - 200k ticks of modern gold is nine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SQX decoder is a per-record Python loop and cannot be vectorised - record
LENGTH depends on the config nibbles, so record k+1's offset is unknowable
without parsing record k. It therefore saturated exactly one core: 10% CPU on a
12-core box, ~3h for the four files.
But the format is randomly seekable. Every BLOCK_LENGTH records SQX restates all
four fields as absolute int64s, so byte ranges beginning at block headers decode
with no shared history. split_offsets() cuts a file on those boundaries and
decode_iter() gained start/stop. EURUSD: 55 min -> 9.4 min, 94% CPU.
find_block() will not trust a bare MAGIC match: 0x00..0x0e is a byte run that
delta payloads produce by coincidence, so a candidate is accepted only when the
next header downstream carries the next sequential block index.
Verified equal, not assumed equal: the same 315MB span decoded serially and in 6
chunks gives identical tick counts (31,056,000), identical bar counts (206,316)
and identical OHLC. The only divergence is the documented seam artifact - the
first tick of a chunk has no predecessor so its delta counts as zero, bounded at
workers-1 ticks in 513M (~2e-8).
Two fixes this shook out:
- The feed is not perfectly time-ordered. EURUSD carries 2 backward steps in
513,494,303 ticks, both under an hour, both in 2003-2006. Bucketing is by
absolute timestamp so every tick still lands in its true bar; the symptom is a
bucket emitted twice out of order. finalise() now stable-sorts before the
duplicate merge. The ordering assert is kept but keyed to MAGNITUDE, since a
real chunking bug displaces a large fraction of rows and feed noise displaces a
handful - only one of those is safe to continue past.
- Chunk workers return undivided sums; means are divided once globally. Dividing
per chunk would weight a straddling bar's mean-of-means wrong.
test_flow.py: charge the PER-BAR spread instead of a single median across
2003-2026 - FX spreads narrowed by roughly an order of magnitude over that span,
so one median charges modern cost to the 2000s and vice versa. Timeouts are now
reported separately rather than silently booked as stop-outs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ready ahead of the bar build so the analysis runs the moment EURUSD lands.
Tests the two SIGNED features the tick pipeline can produce - tick-rule imbalance and
event-count OFI - as entry triggers at 1:1/12, 1:2/24 and 2:3/48 bar geometries. Everything
else the pipeline computes is unsigned and cannot point a trade however well it measures.
Same discipline as every other test here: signal on bar i, entry at the OPEN of i+1,
barriers scanned forward only, sequential NON-OVERLAPPING trades so each is independent
(skipping that is what produced a fake +2.66pp at 2.9 sigma earlier in this project),
break-even == chance by the gambler's-ruin identity, spread charged inside the barrier, and
a sign-flip null taking the max over the whole family for the family-wise bar.
The prior is written into the docstring before any result exists: OFI is established as a
CONTEMPORANEOUS explainer whose predictive power decays within seconds, and it reproduced
that here at +0.56 against the same-bar return. So the expectation is that it explains the
bar it is measured in and says nothing about the next. What is actually being tested is the
gap between that literature - equities, sub-second, size-weighted book data - and this
setting: retail FX CFD feed, 5-minute bars, event counts without sizes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces bidmoves/askmoves with bid_up, bid_dn, ask_up, ask_dn. A bid ticking UP and an ask
ticking DOWN both mean buy-side pressure, and a counter that only records "the bid changed"
cannot tell them apart - it throws away the direction, which is the only part that could
ever point a trade. This is order-flow imbalance in the Cont/Kukanov/Stoikov sense, in its
event-count form; the feed carries no sizes so it cannot be size-weighted.
Caught before the 3-hour build rather than after, which was the point of smoke-testing on a
bounded sample first.
Verified against the previous column set on the same 3M ticks: 21,971 bars, ticks/bar 76,
up 38, dn 38, spread 0.000126, rvol 4.800e-07, gaps 3.95/31.4 - all identical - and
bid_up+bid_dn reproduces the old bidmoves count of 74 exactly, as it must.
The orientation check that matters: OFI correlates +0.56 with the SAME-bar return. That is
the contemporaneous signature the literature reports, and it is also the cheapest guard
against the failure mode that would otherwise pass silently - a sign flip would read -0.56
and every downstream test would then be fitting the negative of the intended feature.
Expectations set in the docstring rather than discovered later: OFI is well established as
a contemporaneous EXPLAINER of price change and its predictive power decays within seconds.
At M5 with multi-hour horizons the prior should be that it explains the bar it is measured
in, not the next one. Measuring it anyway is the point - but a +0.56 contemporaneous
correlation is not evidence of an edge and must not be reported as one.
Merge/mean bookkeeping is now index-driven off COLUMNS instead of positional, so adding a
feature cannot silently mis-merge a bar that straddles a batch boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sqx.decode_iter() turns the decoder into a generator, and ticks_to_bars.py reduces a
symbol to bars in one bounded-memory pass. Necessary rather than tidy: EURUSD is ~458M
ticks, which is ~15 GB held as arrays, so nothing downstream can take the raw stream.
23 years collapses to ~2.4M M5 bars that every test can load instantly.
Aggregation is vectorised with reduceat rather than looping per tick. The only real
complexity is that a bar can straddle a batch boundary, so the last partial bar of each
batch is carried and merged into the first of the next; per-tick deltas are likewise seeded
from the previous batch's final tick, so the first tick of a batch is not silently treated
as having no predecessor. Verified against the per-tick implementation it replaces: 21,971
bars either way, and every reported median identical to the digit (ticks/bar 76, up 38, dn
38, bidmoves 74, askmoves 74, spread 0.000126/0.000250, rvol 4.800e-07, gaps 3.95/31.4).
Throughput 132k ticks/s, at which point the decoder itself is the bottleneck and the
aggregation costs ~12%.
Features are chosen by what the feed can honestly support. It carries (time, bid, ask,
volume) and no trade direction - SQX's record has one volume field and MT5's
TICK_FLAG_BUY/SELL are empty on FX - so true signed order flow does not exist here and is
not synthesised under a flattering name. What is available:
tick rule up/down mid-price changes; the standard Lee-Ready fallback
quote asymmetry bid updates vs ask updates - which side is being repriced harder
arrival rate inter-tick gaps, mean and max; urgency rather than size
realised variance sum of squared mid returns, a far better volatility estimate than
the bar range and only obtainable from ticks
spread mean and max within the bar
Of these only the tick rule and quote asymmetry can point a direction; the rest are
unsigned, like every feature that has measured above noise in this project so far.
Bars are stamped by the OPEN of their interval and built only from ticks inside it, so no
bar's features depend on a tick after it closes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A 2.1 GB tick .dat was committed in c7e9777 by a broad 'git add -A' and removed again in
6bb3386 - but a delete does not remove the blob from history, so the 2096 MB object is
still reachable and still gets pushed. That is what made syncing hang.
Ignoring the paths only prevents a recurrence; clearing the existing blob needs a history
rewrite, which is the user's call since it rewrites pushed commits.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads SQX tick format 4.2 in pure Python. Derived from SQX's own writer, disassembled out
of internal/libs/SQDataLib.jar (TickDataWriter, NewDataFormat{,Writter}) with the javap
bundled in the install - so this follows the format as specified rather than as guessed.
Why, when Scripts/ExportTicks.mq5 pulls the same four fields from MT5: DEPTH. The broker's
MT5 tick history covers a few years; this file starts 2011-09-19. Sample size has been the
binding constraint on every question in this project - the H1/128-bar setup yields ~300
independent trades in 18 years, enough to resolve only a +6pp edge - so 15 years of ticks
is worth a decoder.
Format: four writeUTF strings, ten zero bytes, one more writeUTF, then records of
(time, ASK, BID, volume) - ask before bid, and the writer swaps them when bid>ask so ask is
always the larger. Every BLOCK_LENGTH=1000 records: MAGIC (15 bytes 0x00..0x0e) + int32
block index + config + four raw int64s. In between, deltas against the previous record.
Config is two bytes = four nibbles laid out high-first, nibble = (logicType << 2) |
dataType, where dataType 0..3 selects a 1/2/4/8-byte payload by magnitude and logicType
supplies the sign (MINUS/PLUS carry unsigned magnitudes; ASIS is a plain signed read).
The scale is the one thing NOT in the file. SQX keeps `decimals` in external metadata, and
1216010000 is equally plausible at 10^3, 10^5 or 10^6 - nothing in the bytes distinguishes
them. Guessing would be precisely the silent, plausible-looking error this project keeps
getting caught by: a 100x price scale error crashes nothing, it just quietly rescales every
ATR-normalised feature downstream. So calibrate_decimals() matches against a known
reference series instead. Against the MT5 SP500 H1 export the answer is not marginal:
decimals=5 median rel.err 9.001630
decimals=6 median rel.err 0.004078 <-
decimals=7 median rel.err 0.899984
Verified on 4M ticks: strictly monotonic timestamps, zero negative spreads, price range
1118.03-2048.38 over 2011-09 to 2014-11 (correct for SP500), median spread 0.43 (matches
this broker's H1 record). The 0.41% residual is the expected artefact of comparing a tick
ask against the nearest H1 bar close.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dumps CopyTicksRange output to CSV in the COMMON files folder, where research/ already
reads its rate exports from. Chunked by hour range because a single unbounded request over
years is both slow and liable to ERR_HISTORY_SMALL_BUFFER; boundaries are half-open on
purpose since CopyTicksRange is inclusive at both ends and adjacent chunks would otherwise
duplicate any tick landing exactly on a split.
Keeps MqlTick.flags RAW rather than decoding to a direction. On FX/CFD only
TICK_FLAG_BID/ASK are ever set - TICK_FLAG_BUY/SELL and volume/volume_real are empty for
Forex - so signed trade direction does not exist in this feed and has to be synthesised
offline from quote dynamics. Exporting a decoded 'side' column would be inventing data.
Written as the reliable alternative to decoding StrategyQuant's .dat: that format's base
record parses cleanly (32 bytes, ms timestamp + bid + ask + one volume, prices x1e6,
verified against a known SP500 level) but the delta stream is a custom bit-aligned
dictionary scheme, and it carries only ONE volume field - so it offers nothing MT5 does not
already provide.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature,
default on). Spread is the one microstructure channel that is both FX-available and
genuinely historical in the Strategy Tester - "during testing, the spread is not modeled
but is taken from historical data" - so unlike swap, signed tick flow or depth of market it
is something a backtest can honestly validate.
What it encodes, stated precisely because the raw measurement overstates it.
research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5
of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges
the spread inside its own barriers, so a wide-spread bar is mechanically likelier to
resolve as a loss and the feature would partly be predicting its own cost model. Relabelling
at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology
and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime
reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when
realised volatility is below its own ATR estimate, which genuinely predicts whether
ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side.
Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated
in the spread series. Both cached on length alone:
if(m_crossAsset.Bars() >= bars) return true;
MQL5 series indices are relative to NOW, so one new closed candle shifts every index by
one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer
the newest, and every cross-asset value is read one bar out of step with the price features
sitting beside it in the same vector - silently, with no error and no shape change. This is
the same class of defect as the dtStudied watermark behind the zero-direction backtests.
Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the
label/feature bar caches already use.
And a performance fix that fell out of it: with correct invalidation the panel rebuilds on
every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one
full multi-symbol resample per simulated bar at training depth. Inference only reads bars
0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The
cache check is >=, so a deeper panel left from training still satisfies it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Spread is the one microstructure channel that survived the API audit - FX-available, and
genuinely historical in the tester ("During testing, the spread is not modeled but is taken
from historical data"), unlike swap (no history), signed tick flow (empty on FX) or depth
of market (absent on retail FX, never replayed).
test_spread.py measures four spread features against the triple-barrier label with the same
block-permutation null as test_volume.py. spr/atr - cost relative to the volatility a trade
must overcome - is the strongest reading anywhere in this project so far: significant on 5
of 8 instrument/geometry cells and 2-4x the magnitude of any volume feature.
Which immediately looked too good, because the label is computed WITH the spread charged
inside the barriers. A bar with high spread/ATR has its barriers shifted more adversely and
is mechanically likelier to resolve as a loss - so the feature would partly predict its own
cost model, which is not tradeable information.
Tested directly by relabelling at zero cost and re-measuring the identical feature:
EURUSD 2:3 +0.000655 -> +0.000485 (p 0.030 -> 0.066, loses significance)
EURUSD 1:2 +0.000626 -> +0.000399 (p 0.003 -> 0.017)
USDJPY 2:3 +0.000418 -> +0.000164 (never significant either way)
USDJPY 1:2 +0.000876 -> +0.000532 (p 0.003 -> 0.003)
XAUUSD 2:3 +0.000769 -> +0.000744 (p 0.027 -> 0.027)
XAUUSD 1:2 +0.000876 -> +0.000698 (p 0.003 -> 0.003)
So roughly 20-40% of it WAS the tautology, and the majority is not. What remains is a
volatility-regime reading: spread is near-fixed while ATR is not, so spr/atr is high
exactly when realised volatility is running below its own ATR estimate - which genuinely
predicts whether ATR-scaled barriers get reached at all.
Note what that does and does not buy. Like volume, spread is UNSIGNED: it informs Neutral
vs directional, never Buy vs Sell. It is the best-measured feature in this project and it
still cannot pick a side.
No EA changes in this commit - measurement only, and the MQL5 side already has an
uncompiled backlog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The block fed exactly one number: (v[i] - v[i-1]) / v[i-1]. That is the first difference,
and it cannot express three things that matter - the LEVEL relative to a baseline (two
dead bars and two frantic bars both read ~0 change), and the two volume-vs-range
interactions, where heavy participation that went NOWHERE (absorption) and heavy
participation that travelled (continuation) mean opposite things and currently collapse
onto the same value.
research/test_volume.py measures each candidate's mutual information with the triple-
barrier label across 3 instruments x 2 geometries, against a BLOCK-permutation null -
blocks sized to the barrier horizon, because adjacent labels share almost their entire
outcome window and a free shuffle yields a null so tight that everything looks
significant. Finite-sample MI bias (~7/n here) is reported alongside rather than
subtracted, since the permutation null already absorbs it.
Result: volLevel beats the shipped change ratio outright on 4 of 6 cells (EURUSD 2:3
+0.000118 excess at p=0.006, USDJPY 1:2 +0.000284 at p=0.002); absorption is the single
strongest reading anywhere in the sweep at EURUSD 1:2 (+0.000404, p=0.002) though it is
null on XAUUSD; vol x range clears on 4 of 6. The shipped change ratio is itself
significant on 5 of 6, so it stays.
Kept OUT: a session-relative z-score against the same hour-of-day's own recent history.
It was the weakest candidate - null on both EURUSD cells - and it is the only one needing
per-hour rolling bookkeeping in MQL5. Not worth the state for a reading that did not
survive its own null on the primary instrument.
Magnitudes, stated plainly because they are the point: the excess MI is ~2e-4 nats against
a label entropy near 1.05. That is under a tenth of one percent of the label's
uncertainty. It is real, it repeats across instruments, and it is nowhere near an edge -
this is worth having because it costs one 50-bar loop, not because it changes the answer.
Prior work stands: the whole single-series feature family measured at the noise floor.
m_neuronsCount is already in the fingerprint, so the width change re-keys existing caches
by itself, which is correct - the input vector genuinely changed shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every feature the network sees today is a function of the traded symbol's own OHLCV:
returns, ranges, oscillators, cloud distances, swing structure. Measured end to end that
whole family sits at the noise floor (research/test_classic.py, and the mutual-information
verdict before it). EURUSD moving is a statement about EUR and about USD, and which one
moved is invisible from EURUSD alone - but plainly visible if you also look at EURJPY,
GBPUSD and the rest.
System\CrossAsset.mqh builds a currency-strength panel from the FX pairs in Market Watch:
per bar, each currency's index is the average log return across every available pair
containing it, signed so "up" always means that currency strengthened. Six features - base
and quote strength at 1 and 20 bars, the DIVERGENCE between the pair and what its two
currencies separately did, and the cross-sectional dispersion of currency moves as a
regime term. The divergence is the thesis: it is the one value here that cannot be derived
from the traded series at all, being defined only relative to the rest of the market.
Built ONCE per training run against the traded symbol's bar grid, not per bar - a per-bar
cross-symbol lookup would be pairs x 178k iBarShift calls.
Correctness work, all of it driven by what MT5 actually guarantees rather than by what the
API surface suggests:
- Alignment is by TIMESTAMP, never by index. Bars do not open together across symbols, and
in the tester each symbol gets its own generated tick sequence, so index k on GBPUSD and
index k on USDJPY are not the same instant. Each traded bar takes the last reference bar
at or BEFORE its timestamp - never after, which would be lookahead - and anything more
than one bar period stale is treated as absent rather than carried forward across a
holiday gap.
- SeriesReady() gates every pair on SymbolSelect + SymbolIsSynchronized + the PER-TIMEFRAME
SERIES_SYNCHRONIZED. The symbol-wide and per-timeframe flags can disagree because the
terminal builds series on separate threads, so checking only the first is not enough.
Non-blocking by design: an unready pair is skipped and picked up on a later build.
- Failure is never fatal. Fewer than two usable pairs logs why and every Features() call
0-fills, so a missing reference symbol costs the context block rather than the whole run.
Fingerprint: the flag goes in, the DISCOVERED REFERENCE SET does not. Which pairs exist in
Market Watch is a measured property of the terminal, exactly like the bar count the
existing comment warns about - keying the weights filename on it would orphan a trained
model the moment the user adds a symbol, silently, because a missing cache reads as a
normal first run.
Defaults ON, which re-keys existing databases on first run. That is intended: the input
vector genuinely changed shape.
Deliberately NOT built, having checked what the platform actually provides:
- swap/carry. SYMBOL_SWAP_LONG/SHORT have no history - "last values will be used for the
whole test period" - so a backtest over 2020-2026 applies 2026 carry to 2020 bars.
- signed order flow. TICK_FLAG_BUY/SELL and volume_real are empty on Forex; any feature
assuming trade direction would silently be all zeros.
- depth of market. Unavailable on retail FX symbols and never replayed in the tester.
- calendar actual-vs-forecast surprise. MqlCalendarValue.actual_value is the FINAL,
post-revision figure and the calendar keeps no as-of-release snapshot, so a surprise
feature for a 2019 bar is built from a number nobody had in 2019. Needs a live recorder,
not a historical read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Transcribes all 26 classic vote models (MA 4, RSI 4, MACD 6, Ichimoku 12) from
Signals/*.mqh into vectorised Python, with their shipped constructor weights, then tests
them as entry triggers on 178k-bar FX histories.
Pre-registered by construction: the rules were written long before this test and nothing
about them is fitted here, so there is no in-sample/out-of-sample split to draw and the
whole history is usable. Break-even == chance by the gambler's-ruin identity, so "beats a
coin" and "makes money" are one question. Sequential non-overlapping trades only; a
sign-flip null over the whole pattern family gives the family-wise bar.
The result that matters is a negative one, and it took two lookahead fixes to see:
- _price_extremum reproduced the standard library's CENTRED MinValue(pos-2,5) window,
which reads up to 2 bars newer than the extremum it describes.
- turning_points marked a turn AT bar i, which is only knowable once bar i+1 closes.
Together those two bars of leakage WERE the entire apparent edge. MACD_p4 on EURUSD 1:2
read +5.05pp at +4.05 sigma before, -0.02pp at -0.02 sigma after; USDJPY 1:2 went +5.24pp
-> +0.02pp. RSI_p2's large NEGATIVE went the same way (-10.33pp -> -1.34pp), which is the
tell: a leak inflates whatever sign it lands on.
With both closed, across 4 instruments x 3 geometries: no pattern, no vote threshold, no
quorum and no event+confirmation rule separates from chance. One cell in ~180 tests stars
(SP500 2:6 vote>=30) and it is non-monotone in the threshold either side of the hit.
test_exits.py answers the trade-management half with the control that makes it mean
something: hold entries fixed, vary only the exit, and run every rule again on RANDOM
entries at the same bars. Breakeven-at-1R, chandelier trails, partials and time stops all
move E[R] - and move it by the same amount on random entries. No rule beats its own
control (max +0.99 sigma over 32 comparisons). Management reshapes the win-rate/payoff
split; it does not manufacture expectancy from a directionless entry.
Residual E[R] across every cell is -0.01 to -0.08 R, which is approximately the spread.
Incidental, both worth fixing in the EA:
- CSignalMA pattern 1 is unsatisfiable at the shipped EMA default. For an EMA,
MA[i]-MA[i-1] and c[i]-MA[i] are both positive multiples of (c[i]-MA[i-1]), so
"close below the MA while the MA rises" cannot occur. Dead code (weight 10).
- Ichimoku pattern 11 (Sanyaku, weight 100, the method's top signal) fires on 27% of
bars because it is a conjunction of three standing STATES with no event term, so it
dominates the averaged vote while carrying no trigger information.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keeps the Python that turned a per-hypothesis cost of minutes into
seconds, so the next person (or the next me) can re-run any of this
without MetaTrader in the loop.
kit.py vectorised triple-barrier labeller + scale-free feature set.
The labeller is a faithful port - stop tested before target
within a bar, so a bar spanning both scores as the loss -
and reproduces the EA's own distribution to 0.05pp
(24.93/22.01/53.06 vs 24.9/22.0/53.1) at 12x the speed.
wf.py purged, embargoed walk-forward gradient boosting.
sim.py sequential NON-OVERLAPPING trade simulation.
sweep.py the cell x geometry sweep.
detail.py full threshold profile for one configuration.
WHAT IT FOUND, and the order matters because the first answer was wrong:
Naive walk-forward looked like an edge - precision rising monotonically
with model confidence, 24.4/24.6/25.3/26.2/27.1%, topping out at +2.66pp
and 2.9 sigma. All of it pseudo-replication: a 128-bar barrier means
adjacent bars share almost their whole outcome window, so one trade was
being counted up to 128 times. Counting each trade ONCE (sim.py) the
ordering collapses to 25.9/31.1/26.8/28.1/25.8 and nothing is
significant. Same error family as the MI null that assumed independence.
That exposed a structural problem bigger than the result: at a 128-bar
horizon, 18 years of SP500 H1 yields at most ~300 independent trades,
which can only resolve a +6pp edge at 2 sigma. Real edges are 1-3pp. The
shipped configuration is not merely unproven - it is statistically
UNFALSIFIABLE on the available history.
The cost/power screen then showed SP500 is among the worst cells
available: 54k bars and spread/ATR 0.34, against EURUSD/USDJPY at 178k
bars and 0.05. Weeks of training went into the hardest instrument on the
list, 3x less data and 7x the relative cost.
Final sweep - 4 instruments x 3 geometries, purged walk-forward,
independent trades: nothing clears +2 sigma. The one survivor (EURUSD
2:3 h48, +2.76pp at +1.57 sigma) dissolves under its full threshold
profile: non-monotone across thresholds, and its per-fold win rate decays
monotonically through time (52.9 -> 45.9 -> 41.7 -> 35.2 -> 26.2).
Conclusion: with price/volume-derived technical features there is no
tradeable entry-direction edge on these instruments - now tested with a
model class that finds interactions, on 3x the data, with honest
statistics.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AnimateDread