forked from animatedread/Warrior_EA
298 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
AnimateDread
|
bfc1da9de1 |
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
da54639996 |
feat: expectancy stop - halt when the measured result says the strategy loses
The daily (4%) and total (8%) rules bound how FAST an account can lose. Nothing
noticed WHETHER it was losing. A negative-expectancy signal traded at 1% inside
that envelope breaches no rule and still arrives at zero - it just takes longer,
with every limit green the whole way down. That is the realistic way this EA
destroys an account, and no existing guard could see it.
THE ARITHMETIC THIS ENFORCES. Expected value per trade is p*TP - (1-p)*SL - cost.
With no directional edge p equals SL/(SL+TP), which is also the break-even rate,
so the payoff terms cancel exactly and EV = -cost. Expected P&L is -(trades) x
cost: strictly negative, proportional to activity. Measured here: directional
precision 23-24% against a 25% break-even, flat across every confidence tier,
with 58 points of spread on SP500. Sizing, stop placement and trailing move
variance around that mean; none of them changes its sign.
So every closed position now reports its result in R (net profit over money
actually at risk) and the running mean is tested against zero. Above the
configured minimum sample, if mean + sigma*SE < 0, new entries stop.
- SIGNIFICANTLY below, not merely below. A run of losers is ordinary variance
even for a profitable system; halting on the raw mean would be the same
act-on-noise error the MI gates exist to prevent. Using the standard error
means a wide spread simply demands more trades before the rule can fire.
- NET of swap and commission (ResolveClose already sums all three). Deliberate
and load-bearing: when the edge is zero, cost IS the expectancy, so a gross
version would measure a strategy nobody can trade.
- Reported in R so symbols, lot sizes and balances share one scale and one
mean. Trades without a stop are not scored rather than assigned a guessed R.
- LATCHED across restarts, like the daily halt and for the same reason: a
latch a reattach clears is not a latch. Clearing it means deleting the risk
state file, deliberately, after looking at why.
State is appended to the risk file length-guarded, so files written before this
still load and start their sample at zero rather than misreading.
Defaults 40 trades / 2 sigma; ExpectancyMinTrades = 0 disables it.
This does not make the strategy profitable and is not meant to. It stops paying
tuition on one the results say is losing, and does it on measurement rather than
on a drawdown limit finally being reached.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
b3b7e7bceb |
fix: excursion window must not depend on the barrier it sizes
DIRECTION IS NOT THERE, and this run is what establishes it. Three symbols:
raw ASYMMETRY clears on all three (p=0.0199 / 0.0050 / 0.0050)
norm ASYMMETRY collapses on all three (p=0.3433 / 0.5075 / 0.2736),
USDCAD landing BELOW its own null
RANGE control strengthens to 3-5x its null everywhere
Divide sigma out and the apparent directional signal vanishes entirely. What
cleared was volatility leaking through an unnormalised difference. Note this
would have passed any replication test: three instruments at p=0.005 is exactly
the evidence one would accept before committing to a rebuild, and the confound
reproduces perfectly. Replication was never going to catch it - only the
normalisation could.
Two defects of mine, both surfaced by the same run.
1. THE GEOMETRY DERIVATION WAS DIVERGING, NOT CONVERGING. It produced a
14.57*ATR stop and a 29.14*ATR target that only 5.7% of bars ever reach.
Excursions were measured over the barrier horizon; the horizon scales with
the target; the target is a quantile of the excursions - so target ->
horizon -> excursions -> target ran away, and "settled" only because the
horizon ladder caps at 384 bars. A saturated runaway, which the iteration
guard could not catch because it watches for OSCILLATION.
Fixed at the root: excursions now accumulate only over m_swingMedianBars -
the UNSCALED median ZigZag leg, a property of the instrument that owes
nothing to the barrier. The barrier walk still runs the full horizon,
because that is how long the trade is held; only the MEASUREMENT used to
size the barrier is confined to a geometry-independent window.
(The Min_Risk_Reward_Ratio warning fired correctly and is what flagged it -
the diagnostic worked while the derivation behind it did not.)
2. THE CONFOUND VERDICT WAS UNREACHABLE. `sizeCleared && !asymCleared` was
tested first and is true whenever size clears - i.e. always - so the branch
that NAMES the volatility confound never printed; all three symbols showed
the generic size-not-direction message instead. Verdict chain rewritten with
the specific case first, and the dangling elses my first patch introduced
removed.
FORCES A FULL RETRAIN (the excursion window changes every derived barrier).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
32ffeb99f3 |
fix: normalise the asymmetry target - the raw one is confounded by volatility
Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three;
raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500
(p=0.1045). That looked like the first directional signal this project has
found. It probably is not, and the test as built could not tell.
(up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its
null on every instrument - and the directional part is symmetric noise eps, then
up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A
pure volatility predictor scores positive MI against a 3-bin (up-dn) while
carrying no directional information at all. Crucially that confound REPLICATES,
so reproducing on two instruments is not evidence against it - and the effect
sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries
~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a
leaked fraction of the volatility signal, not an independent one.
So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only
target a directional claim may rest on. The verdict now separates the cases and
NAMES the confound when raw clears while normalised does not, instead of
reporting the raw line as a finding.
Two bugs of mine in the same block, both caught by output rather than review:
- The derived-geometry line had a MISORDERED argument list: it printed
"stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the
multiple and the multiple as the quantile. Real values were 2.61 stop /
8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen.
- THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25
"so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the
stop - hit three times in four. The printed reachability said exactly that
("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate.
This is the entire reason reachability is measured and printed rather than
assumed.
Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop
still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And
bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the
end of the sorted array.
The geometry from the previous run is NOT usable and the asymmetry result is
unresolved, not established. Both are decided by the next run.
FORCES A FULL RETRAIN (the stop quantile changes every label).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
a7701f032b |
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in |
||
|
AnimateDread
|
2c78f3b90d |
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
3482b6c238 |
feat: entry/SL/TP stop being inputs - the barrier geometry is measured
Three enums left the Inputs tab. They were three things a user had to pick and, in the tester, three more axes for a genetic optimization to overfit. Entry_Multiplier is pinned to MARKET. Its pending modes place the entry at a LEVEL while the rest of the pipeline measures from the bar open - the exact mismatch that manufactured the +0.097 R "retail fade" result later retracted as a fill artifact. This codebase's fill model cannot honestly simulate a pending entry, so it is no longer offered. SL_Mode/TP_Mode become a STARTING pair. ReportBarrierGeometryScan now ADOPTS its winner instead of printing "set SL_Mode/TP_Mode to X and retrain": - only when it clears the family-wise gate from |
||
|
AnimateDread
|
9e1c72aacc |
fix: make the indicator tuner actually measure, and gate what it installs
ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates
returned exactly 0.00359 nats): the tune loop re-inits the indicators and then
scores, with no RefreshData() between.
ReInitADIndicators() does its part - Create() builds a NEW handle carrying the
new parameters, and the feature cache is flagged stale so features really are
recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and
only Refresh() copies data out of a handle into those. So every candidate was
scored on values still held from the PREVIOUS handle. My earlier guess in the
diagnostic ("suspect the feature cache") was wrong: the cache invalidation works.
Two things land together, because neither is safe alone:
1. RefreshData() after the re-init, so a candidate is scored on its own features.
2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and
the maximum of N draws from a null beats its incumbent almost every time - so
"it beat the incumbent" installs noise. This selector is the highest-stakes of
the three found in this audit because it ACTS: it overwrites the user's
configured indicator settings and forces BuildFreshTopology(), so the network
then trains on whatever the noise picked. Fixing (1) without (2) would have
made a dormant bug actively harmful.
The gate draws the winner's own permutation null once, then corrects the p-value
for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather
than the max-of-N resample used by the geometry scan because each candidate here
has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only
the one null. Exact under independence, mildly anti-conservative under positive
dependence - stated in the comment rather than hidden. A rejected winner restores
the configured settings, which best[] cannot do since the descent mutates it.
Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate()
calculates asynchronously, so if the spread is STILL zero the handles simply are
not done and the tuner needs to yield between candidates rather than score them
back to back - a state machine like the label prebuild. That distinction is now
readable from the log instead of requiring another guess.
No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
e5ceed6466 |
fix: MI diagnostics never ran on a resumed model - the stated intent was never achieved
A comment above the diagnostic branch says it "runs even when the sweep does not: on a resumed model ... tying it to that gate meant the only way to see the answer on a running model was to delete the model." It does not. Moving the diagnostic out of the tuner's gate left it behind m_labelCachePrebuilt, which has the same effect: the eager label pre-scan runs only on a FRESH start, because a net loaded from disk labels lazily per bar. So on a resumed model the flag is false forever and the whole MI block - headline, positive control, alignment scan, lag profile, geometry scan, winner test, and the auto-tune line - silently never runs. Measured on SP500 H1 2026-08-07: attached at era 271, still nothing by era 314, zero MI lines in the day's log, and the only "label cache pre-built" entry predates the attach. It also explains the shape of every capture on 08-05/06: each one came directly after a weights reset. The situation the comment was written to eliminate is exactly the situation that persisted. So drive the pre-scan when it is the only thing missing. Safe on a trained net: its one fresh-net side effect, pushing the output-layer bias toward the dominant class, is already gated on m_eraCount == 0, and the advance gate in Train() sits ABOVE if(!m_trainRunActive), so the era loop keeps its state - training pauses for the scan (~1s at 38k bars) and continues from where it was, not from 0. Announced only on a start that actually armed, since StartLabelCachePrebuild() returns unarmed when history is not ready and is retried per bar event. NOT sampled from the lazily-filled cache instead: BuildMiSample skips bars with no cached label, so that would score whichever subset training happened to have visited - a biased subsample presented as a measurement, which is the failure this diagnostic exists to catch. Also corrects a claim in 0d58923's comment. It argued four consecutive "no improvement" runs were ~1-in-100,000 evidence the indicator tuner is inert, by multiplying 5.6% across four runs. They are not independent trials: the MI scorer is deterministic and all four covered nearly the same bars, so an incumbent that is the maximum on this data is the maximum on every run. One ~1-in-18 observation with three correlated repeats, ~5.6% - unremarkable. The same independence assumption that made the uncorrected lag profile star four lags. The candidate-spread line stands: it settles inert-vs-live directly. No input, topology or label change: no retrain. Training in flight stays valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
0d5892357b |
diag: report the indicator tuner's candidate spread - "no improvement" is ambiguous
Auditing the other best-of-N scans after
|
||
|
AnimateDread
|
cccf94f9ca |
fix: correct the lag profile across lags too - it contradicted itself
|
||
|
AnimateDread
|
04ee2e113a |
fix: gate the barrier-geometry winner on a family-wise null, not its own
The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain".
That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two
numbers, with no test that either is distinguishable from zero.
bestExcess is a MAXIMUM over the eligible candidates. The maximum of several
draws from a null sits well above any single draw from it, so a max-shaped
statistic tested against a single-candidate null crowns a winner on noise
almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the
lag profile committed in
|
||
|
AnimateDread
|
3271f1ea93 |
diag: MI feature-lag profile - close the blind spot in every MI verdict so far
BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
8c5ea639ee | feat: extend ADWyckoffEventStream with new range-lifecycle parameters and update related features | ||
|
AnimateDread
|
8ccbddb051 |
Add new research scripts for trading strategy analysis
- Implemented sqx_audit.py to audit StrategyQuant X trade lists, focusing on performance metrics and cost analysis. - Created sqx_portfolio.py to evaluate portfolio performance based on uncorrelated components and their impact on risk and return. - Developed swing.py to analyze cost ratios across different holding periods and assess swing trading structures. - Introduced test_management.py to investigate the effectiveness of exit rules on random entries and their impact on expectancy. |
||
|
AnimateDread
|
f1b7dcf7f3 |
fix: correct MI sample alignment and improve BN weight diagnostic report
The MI sample builder used `MathAbs(labelBarOffset)` as a padding, causing rows from offset and non-offset builds to be paired with a double shift. This broke the positive control, failed the 5× gate, and voided all reported mutual‑information figures. Replace with the fixed `MiShiftPad` constant to ensure builds enumerate the same set of bars and row-k alignment is preserved. Add `BatchOptionsTotal()` to `CNeuronBatchNormOCL` and split the packed BN weight array in the learning report into separate norms for the outgoing dense matrix, gamma, beta, running statistics, and Adam moment buffers. This turns an ambiguous single‑norm reading into precise diagnostics that distinguish weight divergence from scaling issues. |
||
|
AnimateDread
|
5f647ba5db |
fix: improve error messages and suppress false sharing-violation logs
- BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run. |
||
|
AnimateDread
|
3459db887b |
infra: bid/ask M1 bars and an order engine that makes the fill bug unavailable
Three results today were invalidated by the same family of error: a price level used as an entry while the outcome was measured from somewhere else. The defence is not vigilance - it is an interface where the mistake cannot be expressed. bidask.py M1 bars carrying SEPARATE BID and ASK OHLC, built from the tick stream in one parallel pass. Every existing bar file stores MID, which is fine for measuring returns and useless for simulating orders, because no order ever executes at the mid. Mid bars force the spread to be bolted on afterwards as an average - the approximation that let today's artifacts through. With both sides carried, the spread is whatever it actually was, including the overnight and news blowouts an average hides. Fails loudly if ask < bid. fills.py owns the ENTIRE trade lifecycle; no test may open a position any other way. Longs enter at the ask and exit at the bid, shorts the reverse. A buy stop triggers on the ask and fills at the trigger price OR the bar's open if the bar gapped past it, which is where real slippage comes from. Limits fill on the opposite side and a gap is capped in their favour. The invariant that was violated: THE OUTCOME CLOCK STARTS AT THE FILL BAR. The fill index IS the start index - they are the same variable and cannot diverge. That is what went wrong before, and it is now unrepresentable rather than merely discouraged. Same-bar ambiguity is REPORTED, not assumed away: every result carries the fraction of trades decided by a bar containing both barriers, alongside fill rate and unresolved rate. A resolution-limited result now says so itself. M1 rather than raw ticks is a deliberate, stated bound: 513M ticks per symbol is ~8 GB packed and four symbols will not fit in memory, while M1 keeps it at ~500 MB and cuts the residual ambiguity 60x versus H1. It is an approximation with a visible error bar, not an exact simulator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
82cdbb40df |
research: Wyckoff context REPLICATES on a second, independent trigger
Applies the same five-trace context score from
|
||
|
AnimateDread
|
c998d655ee |
research: Wyckoff CONTEXT has real predictive content - 8/8 positive slopes
Earlier tests fired on the shakeout alone, which is not the method. Book 2 2.3
treats it as the third of four cumulative traces and reads the structure's own
history first. This scores all of them, oriented to the shakeout's direction:
1 Phase A test location (upper vs lower half of the structure)
2 Phase B test location
2b STRUCTURAL FAILURE - after the Phase B test, did price fail to reach the
opposite extreme
4 effort/result on the shakeout bar (close position + volume vs range average)
7.1 higher-timeframe context - is the larger move in the shakeout's favour
Conditioning on agreement shrinks the sample and multiplies the ways to slice
it, so the test is NOT 'find the combination that works'. It is the one
pre-specified prediction the books make and mining does not: expR must rise
MONOTONICALLY with the number of agreeing traces. One slope, no threshold to
tune, no best cell to pick.
POOLED (16,234 non-overlapping trades):
0 traces -0.332 3 traces -0.114
1 trace -0.214 4 traces -0.117
2 traces -0.184
slope +0.0464 R per agreeing trace, t +2.16
per-symbol slopes POSITIVE IN ALL 8 CELLS (p ~ 0.004 on sign alone)
So the context logic is real and measurable - it is not folklore. But the base
trade is in too deep a hole for it to matter: full confluence still returns
-0.117, and reaching break-even would need ~7 agreeing traces when only 5 exist.
The useful reading is that context is a MODIFIER worth about +0.05 R per trace,
which is only interesting when bolted to a trigger whose base expectancy is
already near zero. The shakeout's is not, because its structural target sits
4-6R away and is rarely reached.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
9a4ae635e0 |
research: the two canonical Wyckoff trades tested whole - both negative
Completes the programme on both books. Entries are MARKET ORDERS at a bar's
open throughout, so the fill-timing artifact that invalidated the last round is
designed out rather than remembered. Benchmark is analytic: entry, stop and
target fixed at entry means a driftless market gives expR = 0 exactly.
1. SPRING / UPTHRUST (book 1 ch.18, the event 'all Wyckoff operators wait for').
Pierce of a COMPRESSION-QUALIFIED range edge, close back inside, stop beyond
the shakeout extreme, target the far side of the range.
30 cells across 4 symbols x M15/H1/H4. Reward-to-risk averages 4-6:1, so the
break-even win rate is only 15-20%, and it still loses nearly everywhere:
M15 all four symbols -0.15 to -0.24 with 0/4 folds positive. Best cell is
EURUSD H1 climactic-volume +0.302 at t +2.27, which over 30 cells is inside
the family-wise band.
The books' volume requirement was applied - climactic (>1.5x range average)
and quiet (<0.8x) shakeouts scored separately. Neither rescues it.
2. LPS / LPSY, the test-after-breakout, and book 2's A/B (5.7.1, 5.8.3): it
claims the retest should be awaited at the VOLUME PROFILE level, not the
price edge. Same breakout, same stop, same 2R target, only the location
differs:
retest at typical expR
A price edge -0.041 .. -0.315
B value-area edge -0.128 .. -0.413
C range VPOC -0.129 .. -0.506
24/24 cells negative, and A > B > C in ALL EIGHT symbol/timeframe
combinations. That monotone ordering is not noise, and it inverts the book's
recommendation. Mechanism is adverse selection: the VPOC sits deep inside the
old range, so a retest that reaches it is disproportionately a breakout that
has already failed. The deeper the level you wait at, the more your fills are
selected against you.
Practical consequence: the volume profile is real (levels beat distance-matched
placebos at z +3 to +7.8) but using it to LOCATE ENTRIES makes this trade
worse, not better.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
79f108ab84 |
research: RETRACTION - the retail-fade edge was a fill-timing artifact
The +0.097 R EURUSD result in |
||
|
AnimateDread
|
821f16df8b |
research: model-selected retail fades hold up out of sample (after killing a big leak)
Reframes what the net is asked. Direction is the one thing the data says is barely predictable; but the fade of retail pin/inside setups has a MEASURED gross edge of ~0.139 R and a cost of spread/stop that varies 5x across instruments, sessions and vol regimes. So the answerable question is selection: spend the edge only where it survives. label realised R of the fade (what actually pays) features 28, all knowable at entry, incl. the volume-profile levels split CHRONOLOGICAL train/val/test; threshold picked on val, frozen for test TEST BLOCK 2022-08 .. 2026-07, 63,901 candidate trades, keep top 10%: mean R +0.1005 vs -0.0616 for taking everything by year +0.140 +0.075 +0.098 +0.105 +0.105 -> 5/5 positive with the COST FEATURE REMOVED: +0.0800, still 5/5 positive That last control matters: the model is not merely learning 'skip wide spreads'. Something in the setup geometry, session and level structure carries signal beyond the cost. HONEST SIGNIFICANCE. The naive t of +8.06 is not believable - with an 8-day horizon these trades overlap heavily and thousands share one price path. On a strictly non-overlapping subset (163 independent trades) it is +0.123 R at t +1.61; without the cost feature, +0.257 at t +3.45. The overlap filter applies the H1 horizon to M15 trades too, so 163 is a conservative floor and the true independent count is higher. Suggestive, not settled. THE LEAK THIS RUN NEARLY SHIPPED. First version scored +0.53 R on the held-out block, t +54. Bar-derived features were read at the FILL bar i2, but the order fills intrabar and the outcome race starts at the first M5 bar inside i2 - so i2's close, tick count and realised volatility are not knowable at entry. The model was seeing how the bar it entered on turned out. Second lookahead of this hunt (the first was worth +0.15 R in the sweep test). RULE: a clean chronological split does NOT protect against lookahead. The split was honest and the features were not. Any feature indexed at the entry bar must be re-derived from the bar before it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
c9b489e5d8 |
research: the retail fade DOES clear cost on EURUSD - correcting the earlier verdict
The previous commit pooled four instruments with very different spread-to-stop ratios and concluded the edge never beats the cost. That was too broad. Per cell (48 cells, M5 path, k=1), 8 clear - and they are not scattered: EURUSD H1 pin spread/stop 0.042 edge +0.160 cost 0.063 -> +0.097 R EURUSD H1 pin 0.045 +0.127 0.067 +0.060 EURUSD H1 inside 0.047 +0.123 0.060 +0.063 EURUSD H1 inside 0.050 +0.102 0.062 +0.040 EURUSD M15 pin 0.074 +0.141 0.100 +0.042 EURUSD M15 pin 0.072 +0.140 0.099 +0.041 Every clearing cell is on the tightest-spread instrument. XAUUSD carries the same gross edge (+0.09 to +0.135) and never clears, because its cost is 3x. That is the mechanism predicting where the effect should survive and being right - the opposite of the stop-run case, which inverted. WALK-FORWARD, 4 chronological folds: 6 of 8 hold at >=3/4. EURUSD H1 pin short side is +0.116 / +0.061 / +0.143 / +0.067 across 23 years, 4/4. WIDENING THE STOP still says what it said: EURUSD H1 pin goes +0.078 (m=1) -> +0.028 -> +0.009 -> +0.017 -> -0.003 (m=5). The gross edge collapses ~15x while the stop widens 5x, so this is NOT drift - it is reversion inside roughly one setup-risk of a stop order filled at a local extreme. It is only tradeable at the tight stop, which is exactly where cost bites hardest. WHAT IS NOT MODELLED, and it decides this: commission and stop slippage. Gross edge is ~0.139 R = ~2.4 pips on a 17.3-pip stop, against 0.75 pips of spread. That leaves ~1.6 pips of headroom for commission plus slippage before it is gone. A demo forward test measuring both is the next step, not more history. Also fixes a LOOKAHEAD found in the sweep-entry test: the protective stop was anchored to the low of the very bar that filled the limit order, which is not known until that bar closes. It was worth ~+0.15 R - larger than any real effect here - and it inflated the placebo equally, which is how it was caught. With it removed, buying at retail stop levels is no better than buying at an arbitrary level the same distance away: the 'stops are a farmable magnet' claim fails its own control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
c645c98f31 |
chore: keep the reference PDFs out of the repository
40 MB of third-party copyrighted books sit in references/ so the research scripts can read them. Untracked is not the same as safe: one broad `git add -A` puts them in history permanently and on a public remote. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
c2dd9eb6aa |
research: retail setups ARE anti-predictive - and the edge dies with the cost
Tests the user's thesis directly: if price is unpredictable, trade against the
people predicting it badly. Implements the three mechanical setups from 'How To
Day Trade Forex For Profit' ch.5 with their DOCUMENTED stop rules, so retail
stops are located exactly rather than by proxy.
THE MIRROR TEST. Retail's trade and its exact mirror, priced under identical
rules. Both sides pay the same spread and suffer the same same-bar tie
convention, so those cancel in the difference and double in the sum:
edge = (mirror - retail)/2 cost = -(mirror + retail)/2
pin EDGE +0.108 R COST +0.143 R
inside EDGE +0.068 R COST +0.140 R
engulf EDGE -0.001 R COST +0.095 R
So pin-bar and inside-bar setups really are anti-predictive - the first
confirmed directional edge in this project. Engulfing is a pure coin flip whose
loss is entirely the spread, i.e. money already gone to the broker.
Stable across three conventions: H1 bars pessimistic ties, M5 path pessimistic,
M5 path optimistic. Re-walking the barriers on M5 CUT the cost (0.195 -> 0.143)
and RAISED the edge (0.078 -> 0.108), so the coarse-bar convention was masking
the effect, not manufacturing it.
THEN THE TEST THAT KILLS IT. Cost in R is spread/stop-distance, so widening the
stop divides it. If the edge is directional drift it survives. Fade expR by stop
multiple (pin, k=1, 122k trades):
m=1.0 cost 0.146 expR -0.045 implied edge +0.101
m=1.5 cost 0.097 expR -0.067 +0.030
m=2.0 cost 0.073 expR -0.065 +0.008
m=3.0 cost 0.049 expR -0.051 -0.002
m=5.0 cost 0.029 expR -0.040 -0.011
The edge decays exactly as fast as the cost, then inverts. It was never drift:
it is reversion against a stop order filled AT a local extreme, and it lives
within one bar-range of the entry - the same short-horizon reversal the tick-flow
work already measured, meeting the same fate.
Also in this commit, the volume-profile claims from Wyckoff 2.0:
MAGNET all 10 tests positive vs a distance-matched placebo, z +3.0 to +7.8,
family-wise bar 2.79 - but the effect is +0.15 to +0.59pp on a ~74%
base rate, i.e. ~0.01 R.
REACTION naked VPOC and VPOC reject +0.71 to +0.86pp (z to +5.65); value-area
edges null or negative; HVN/LVN marginal.
80% RULE dead. 27.8% traversal against a 29.0% martingale benchmark. Acceptance
nearly DOUBLES the raw rate (14.1% -> 27.8%) and the benchmark doubles
with it - every bit of the apparent improvement is geometry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
8e6525d819 |
research: Wyckoff's law of cause and effect is real but SUBLINEAR
The 1:1 range projection is the target rule both books recommend (book 1 ch.8 discards point-and-figure counting as too subjective and keeps the vertical projection). Tested as a complete trade on 4 symbols x M15/H1: enter on the range breakout, stop at the far side of the range, target k x risk. A driftless market gives P(win) = 1/(1+k) and expR = 0 at EVERY k, so the benchmark here is analytic - no permutation null needed. Result: expR sits on that benchmark at every k on every symbol. Target placement does not move expectancy, which is what a martingale already said. But the law itself is measurable, and it is not 1:1. Regressing log(MFE) on log(range height) with log(ATR) as a FREE regressor (a shared ATR denominator correlates the errors and biases the exponent towards the hypothesis, so it cannot be used to argue against it): b = 0.10 .. 0.92, centred ~0.6 b = 1 rejected in 5 of 8 at >2sd, never significantly above 1 b = 0 rejected in 7 of 8 So a bigger cause does produce a bigger effect - sub-proportionally. The 1:1 projection systematically over-reaches after a large consolidation and under-reaches after a small one. Median travel in risk-multiples falls monotonically across height quartiles in 8 of 8 runs. Also adds the volume-profile machinery the second book is built on and which nothing in the EA has: tick-level volume-at-price on a fixed absolute grid, per-session VPOC/value area by the standard Market Profile walk, naked VPOCs, and HVN/LVN from a rolling causal composite. Two biases are left in deliberately, both against the hypothesis: a bar spanning stop and target books the loss, and spread is charged on entry and both barriers. Unresolved trades are marked to market at the horizon rather than discarded - discarding them deletes slow winners and manufactures a false deficit at large k. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
6f7aa4f1e8 |
research: the stop-run edge fails its own mechanism test - disconfirmed
The statistics had cleared a family-wise bar (z to +6.56) and a split-half. Both necessary, neither sufficient: thorough enough mining passes both. What mining cannot do is obey a mechanism it was never fitted to - so the decisive test is whether the effect appears WHERE THE THEORY SAYS IT MUST. Osler's stop-clustering predicts the edge concentrates where the stop reservoir is deepest and fresh forced orders arrive: London open and the London/NY overlap. Measured, by session in real UTC (broker is UTC+2): EURUSD Sydney/Tokyo +0.197* London open -0.063 London -0.156 USDJPY Rollover +0.233 London open -0.223 XAUUSD Sydney/Tokyo +0.070 London open -0.345 LDN/NY -0.185 SP500 Rollover +0.206 NY afternoon -0.098 Exactly inverted. The liquid sessions where stops actually cluster are the worst on every instrument; what remains lives in Sydney/Tokyo and rollover - the THINNEST hours, where fewest stops sit. That is not the mechanism, and thin hours are also where spreads are widest and fills worst, so even the surviving fragment points away from tradeability rather than toward it. Walk-forward by quarter agrees, and shows what the two-way split was hiding: EURUSD 2/4 positive (Q1 -0.029, Q2 +0.062, Q3 -0.010, Q4 +0.089) USDJPY 1/4 XAUUSD 0/4 SP500 2/4 No instrument reaches 3/4. The split-half HOLDS was Q2+Q4 carrying Q1+Q3. Verdict: not an edge. Recording it as disconfirmed rather than leaving an encouraging half-result in the log, because the next person to read this - me - would otherwise build on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
9ca0ffcc5c |
research: stop-run/liquidity sweep - the first setup to survive everything
Osler's currency-order-flow work is the published mechanism: stop-loss orders cluster just beyond recent swing extremes and cascade price; take-profits cluster and reverse it. So the claim is not "a pattern repeats" but "there is a reservoir of forced orders at a location computable in advance" - which is falsifiable in a way chart patterns are not. Tested as a COMPLETE trade rather than a signal with barriers bolted on: entry, stop and target all come from one structure. Price takes out the N-bar extreme MARGINALLY (<= over*ATR), closes back inside, enter the opposite way next open, stop just beyond the sweep extreme (where the liquidity actually was), target a multiple of that risk. 19 of 24 configs clear a family-wise max-statistic bar on EURUSD/USDJPY H1, all in the predicted direction, z to +6.56. R=1 configs are negative and R=2/R=3 turn positive, which is coherent: the edge is directional and a tight stop pays the spread as a large fraction of risk, so it needs a big R to clear. SPLIT-HALF then kills most of it, as it should: EURUSD N=50 ov=0.5 R=3 +0.013 / +0.042 HOLDS EURUSD N=20 ov=0.5 R=3 +0.014 / +0.034 HOLDS every R=2 config one half negative USDJPY one half negative Surviving configs are STRONGER in the second half, the opposite of a mined artifact decaying out of sample. But N=20 and N=50 overlap heavily and are not independent, so this is one instrument and one R - a lead, not a system. dukas.py: direct Dukascopy datafeed client. SQX mirrors through its own CDN (CdnCache/CdnDownloadJob) so there is nothing reusable there. Dukascopy publishes the raw feed - bi5, raw LZMA, 20-byte big-endian records, ZERO-BASED MONTH in the URL (fails silently into the wrong month otherwise). Cached, resumable, bounded concurrency. Two corrections it forced, per the user: SQX conforms Dukascopy data to the5ers' broker profile AND timestamps. Measured empirically, broker time = UTC+2 (EET), clean minimum. So (1) previously reported "hours" are BROKER time - gold's hour 1 is 23:00 UTC, the daily rollover and COMEX Globex reopen, a real mechanism; and (2) Dukascopy's raw 0.2-pip ECN spread must NOT be used for cost - the5ers' ~0.47 is what is actually paid, so the existing cost analysis was right and Dukascopy would have made every result look falsely tradeable. Its value is the bid/ask VOLUMES, which SQX lacks entirely - true signed flow instead of the event-count proxy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
05c6a5c484 |
research: four more hypothesis families - drift is real, timing still is not
Everything tested before this asked ONE question - can recent price or flow
predict the next bar's direction - and answered no four ways. These are different
families, each with a published prior rather than a hunch.
1 TIME-SERIES MOMENTUM (Moskowitz/Ooi/Pedersen). 34 configurations across 4
symbols x D1/H4 x 6 lookbacks. Nothing. The one rule that looks strong -
XAUUSD H4 250-bar, p=0.0038, t+3.30, +10.32%/yr - returns essentially exactly
buy-and-hold's +10.36%. It is not timing gold, it is being long gold. Hence the
vs-B&H column: on a drifting asset a rule that is merely long most of the time
looks skilful and is not.
2 SEASONALITY. The first thing in this project to survive a properly controlled
test: 5 of 8 clear a family-wise max-statistic bar, two at p=0.0002. Split-half
kills two of them (USDJPY dow-6 n=116 and SP500 hour-0 n=533 are thin
off-session buckets). Two HOLD with near-identical halves:
XAUUSD hour 1 +2.29 bp (t+6.44) / +2.43 bp (t+5.53)
EURUSD hour 13 -1.47 bp (t-6.80) / -0.64 bp (t-3.56)
Gold's hour 1 alone carries more than half the +4.22 bp/day drift.
And it is still not tradeable. Widening the window to amortise the 4.92 bp
round trip: the best of 144 windows (hour 1, 8h) nets +0.14 bp/day, t +0.23,
and splits +1.29 / -1.00 - the sign flips between halves. Every other window is
negative. Real, stable, well measured, and about 2x too small to cross its own
spread. Same shape as the flow result.
3 OVERNIGHT/INTRADAY - folded into the hour analysis above.
4 VOLATILITY-MANAGED DRIFT (Moreira/Muir) - the one needing no directional edge.
Does NOT reproduce here: flat on gold (-0.01), and it HURTS SP500 (0.71 -> 0.41
D1, 0.77 -> 0.54 H4). Honest negative against a strong prior.
What survives all of it is drift, which is large and significant while every
timing rule is noise: XAUUSD +10.24%/yr (t 2.88), SP500 +12.25%/yr (t 2.84),
against USDJPY +1.16%/yr (t 0.59) and EURUSD ~0.
resample.py gains D1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
0c2b025c16 |
fix(research): calendar recorder - separate LIVE from BACKFILL, fix seen-set key
First live run exposed both problems at once. It logged "+83 release(s) recorded", and every one of those rows shared a single observed_time up to 30 hours after its event_time: they were the startup backfill, not release-time observations. Their actual figures are whatever the terminal holds NOW - the post-revision values this recorder exists to avoid - and the very first batch proved that is not hypothetical: a Retail Sales row came back previous 3.5 / revised_prev 3.4, and a Core CPI row already carried revision=1. Backfill is still worth keeping (a fine snapshot of the revised series, and it carries the event metadata) but must never be silently mixed with release-time observations. Every row now records lag_sec and a capture class, so the distinction cannot be lost by whoever loads the CSV later: LIVE observed within InpLiveLagSeconds (default 600s) of release BACKFILL seen long after the fact - MUST NOT be used for surprise research The log now reports the split per poll and says so explicitly when a poll is entirely backfill. Second and worse, in LoadSeen: the FILE_CSV field walk was off by one and keyed the seen-set on event_id instead of value_id. event_id identifies the event TYPE, not the release, so after any restart every future release of every event already in the file would have been skipped - permanently, and silently, exactly for the recurring high-importance events (NFP, CPI) that matter most. Now reads whole lines and indexes a split array by a NAME-CHECKED column position, which cannot drift when the schema changes. Refuses to guess if value_id is absent. Schema change is handled by rotating any file with a non-matching header to <name>.<timestamp>.old rather than appending, since mixing layouts mis-parses every old row. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
2aa5bfbe28 |
feat(research): write-once live calendar recorder
The calendar is the only non-price source MQL5 carries with real content, and it is the one this project cannot research from history: MqlCalendarValue.actual_value returns the CURRENT figure, i.e. after every later revision. Reading 2019's NFP today returns a number nobody could have known in 2019, so any surprise = actual - forecast feature built from history carries lookahead - and the flattering kind, since it makes a model look most prescient exactly on the events that were revised most. The only sound fix is to write down what the terminal reported at the moment of release and never touch that row again. Write-once is the entire contract here: a row is appended the first time a value_id is seen carrying an actual figure, and is never rewritten, because re-recording on a later poll would silently import the revision this file exists to avoid. The seen-set is rebuilt from the file on init so a restart cannot duplicate or re-import either. Deliberately standalone - no includes from the EA tree, and not wired into Warrior_EA. It may run for months on a spare chart, and coupling it to the trading system would mean a refactor there can stop the recorder; a gap in a write-once series cannot be backfilled by definition. It also keeps a data-collection task from adding any failure mode to a system about to trade a prop account. Records actual/forecast/previous/revised_previous, revision number, impact, importance, units, plus observation time and the quote at observation. Every FileOpen carries FILE_SHARE_READ|FILE_SHARE_WRITE per the rule this codebase learned the hard way (exclusive opens fail 5004 and look like "no data"). Compiles 0 errors, 0 warnings. Value starts at zero and accrues with time, which is the argument for starting it now rather than when it is wanted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
50e1c7ef84 |
research: flow effect and spread cost decay together and never cross
resample.py composes M5 bars into M15/H1/H4 exactly - every column this pipeline produces is composable (sums sum, maxes max, OHLC nests, means re-weight by tick count), so this costs seconds instead of another 37-minute decode per timeframe. Asserts tick conservation and extreme preservation on every output. Motivation: ATR grows ~sqrt(time) while the spread does not, so spread/ATR should fall with timeframe and make a small edge affordable. It does, monotonically, and the measurement is clean (EURUSD, 1:1 barriers): M5 spread 0.099 ATR random wins 36.8% cost 13.2pp M15 0.057 39.7% 10.3pp H1 0.029 43.6% 6.4pp H4 0.015 47.6% 2.4pp But the signal decays at the same rate. Rows clearing the family-wise bar: M5 many, z to -10.1 M15 many, z to -5.8 H1 2 of 9, one POSITIVE and one negative - the shape of noise, not signal H4 none So the effect lives where the cost is fatal and is gone where the cost is affordable. They never cross. Also added --cheap=Q, which trades only the lowest-Q quantile of spread/ATR. This is the one honest use of an unsigned feature: it cannot point a direction but it can decline to trade, and both terms are known before entry. It does cut cost (EURUSD M15 10.3 -> 7.6pp, USDJPY 12.7 -> 6.4pp) and the effect does not survive there either - nothing clears the bar. test_flow.py gained --tf= and keeps the z-score window at ~1 day on every timeframe rather than a fixed 288 bars. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
f83425aad8 |
research: tick-flow verdict across 4 instruments - real reversal, untradeable
1.93 BILLION ticks -> 5.5M M5 bars (EURUSD/USDJPY/XAUUSD 2003-2026, SP500 2011-2026). Sequential non-overlapping trades, triple barriers, per-bar spread, direction-permutation null with a family-wise max-statistic bar. Order flow is genuinely ANTI-predictive at M5 - price mildly reverses the prior bar's flow. Same sign on all four instruments, clearing the family-wise bar on three: USDJPY z -10.10 -1.73pp vs null EURUSD z -7.95 -1.41pp XAUUSD z -5.34 -0.68pp SP500 z -2.60 -0.87pp (does not clear; half the sample) Agrees with the -0.0151 next-bar correlation (vs +0.4961 same-bar, which is the contemporaneous Cont/Kukanov/Stoikov effect and is not edge). And the cost dwarfs it. Random entry at 1 ATR barriers after spread: EURUSD spread 0.099 ATR -> wins 36.8% (13.2pp below the costless 50%) USDJPY 0.154 34.9% (15.1pp) SP500 0.292 26.5% (23.5pp) XAUUSD 0.450 20.0% (30.0pp) Cost rises monotonically with spread/ATR, which is an internal consistency check on the apparatus. A ~1pp effect against 13-30pp of cost is 10-100x short. Reversing does not rescue it - expR_rev is negative in every row of every geometry. Widening the barriers does not either: at 4-8 ATR nothing clears the bar (max |z| 2.51 vs 2.97). The effect lives exactly where the spread is fatal and vanishes where the spread would be affordable, which is what a seconds-to-minutes phenomenon predicts. Adds --narrow/--wide geometry sets so both regimes are reproducible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
eeacb609b6 |
fix(research): standardise flow test against the EMPIRICAL null, not a costless coin
The first run reported -23pp edges at -75 sigma, which is not a market effect - it is the tell this project has been burned by before (a lookahead, or here a wrong reference, inflates whatever sign it lands on). The give-away was in the output itself: a family-wise 5% bar of |z| > 71.67 where a centred null over 16 tests should sit near 2.5. Random entry was losing almost as badly as the signal. Cause: z and "edge pp" were measured against be = sl/(sl+tp), the break-even of a COSTLESS coin. These barriers charge the spread and book a loss when a single bar spans both levels, so random entry at 1 ATR on M5 wins ~36.8%, not 50%. The table was reporting the fixed cost of trading as if it were signal. Now every row shows the empirical null win rate, the gap against it, and z standardised by the null's own spread. Family-wise bar drops to 2.95 and the result becomes legible: order flow is genuinely ANTI-predictive at M5, about 1pp below random at z -5 to -8, clearing the bar in 12 of 16 tests and reproducing across three geometries and two independent signal families. It agrees with the -0.0151 next-bar correlation. It is also untradeable, which the table now says out loud: the spread is 0.099 ATR and costs 13pp of win rate against a 1pp effect. Reversing does not rescue it - expR_rev is reported per row and stays negative everywhere. Added a footer stating that beating the null is necessary but NOT sufficient; only exp R > 0 makes money. Also: - permutation null was allocating a single (nperm x nT) array, ~2 GB at these trade counts. Now batched. - null permutes the OBSERVED directions instead of flipping a fair coin, so a directionally skewed rule on a trending instrument cannot pass on drift alone. - timeouts reported separately rather than silently booked as stop-outs. - calibration falls back to a midpoint sample when the tick history predates the MT5 reference series (XAUUSD ticks start 2003-05-05, its H1 export 2004-06-11, so the head sample overlapped by nothing). 2M ticks, because the sample must span >=50 reference HOURS - 200k ticks of modern gold is nine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
91d67db737 |
perf(research): parallel tick decode, 12x, plus two correctness fixes
The SQX decoder is a per-record Python loop and cannot be vectorised - record LENGTH depends on the config nibbles, so record k+1's offset is unknowable without parsing record k. It therefore saturated exactly one core: 10% CPU on a 12-core box, ~3h for the four files. But the format is randomly seekable. Every BLOCK_LENGTH records SQX restates all four fields as absolute int64s, so byte ranges beginning at block headers decode with no shared history. split_offsets() cuts a file on those boundaries and decode_iter() gained start/stop. EURUSD: 55 min -> 9.4 min, 94% CPU. find_block() will not trust a bare MAGIC match: 0x00..0x0e is a byte run that delta payloads produce by coincidence, so a candidate is accepted only when the next header downstream carries the next sequential block index. Verified equal, not assumed equal: the same 315MB span decoded serially and in 6 chunks gives identical tick counts (31,056,000), identical bar counts (206,316) and identical OHLC. The only divergence is the documented seam artifact - the first tick of a chunk has no predecessor so its delta counts as zero, bounded at workers-1 ticks in 513M (~2e-8). Two fixes this shook out: - The feed is not perfectly time-ordered. EURUSD carries 2 backward steps in 513,494,303 ticks, both under an hour, both in 2003-2006. Bucketing is by absolute timestamp so every tick still lands in its true bar; the symptom is a bucket emitted twice out of order. finalise() now stable-sorts before the duplicate merge. The ordering assert is kept but keyed to MAGNITUDE, since a real chunking bug displaces a large fraction of rows and feed noise displaces a handful - only one of those is safe to continue past. - Chunk workers return undivided sums; means are divided once globally. Dividing per chunk would weight a straddling bar's mean-of-means wrong. test_flow.py: charge the PER-BAR spread instead of a single median across 2003-2026 - FX spreads narrowed by roughly an order of magnitude over that span, so one median charges modern cost to the 2000s and vice versa. Timeouts are now reported separately rather than silently booked as stop-outs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
fac9f620b0 |
research: harness for the M5 order-flow edge test
Ready ahead of the bar build so the analysis runs the moment EURUSD lands. Tests the two SIGNED features the tick pipeline can produce - tick-rule imbalance and event-count OFI - as entry triggers at 1:1/12, 1:2/24 and 2:3/48 bar geometries. Everything else the pipeline computes is unsigned and cannot point a trade however well it measures. Same discipline as every other test here: signal on bar i, entry at the OPEN of i+1, barriers scanned forward only, sequential NON-OVERLAPPING trades so each is independent (skipping that is what produced a fake +2.66pp at 2.9 sigma earlier in this project), break-even == chance by the gambler's-ruin identity, spread charged inside the barrier, and a sign-flip null taking the max over the whole family for the family-wise bar. The prior is written into the docstring before any result exists: OFI is established as a CONTEMPORANEOUS explainer whose predictive power decays within seconds, and it reproduced that here at +0.56 against the same-bar return. So the expectation is that it explains the bar it is measured in and says nothing about the next. What is actually being tested is the gap between that literature - equities, sub-second, size-weighted book data - and this setting: retail FX CFD feed, 5-minute bars, event counts without sizes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
e7a9fe22a7 |
research: signed order-flow imbalance instead of bare quote-move counts
Replaces bidmoves/askmoves with bid_up, bid_dn, ask_up, ask_dn. A bid ticking UP and an ask ticking DOWN both mean buy-side pressure, and a counter that only records "the bid changed" cannot tell them apart - it throws away the direction, which is the only part that could ever point a trade. This is order-flow imbalance in the Cont/Kukanov/Stoikov sense, in its event-count form; the feed carries no sizes so it cannot be size-weighted. Caught before the 3-hour build rather than after, which was the point of smoke-testing on a bounded sample first. Verified against the previous column set on the same 3M ticks: 21,971 bars, ticks/bar 76, up 38, dn 38, spread 0.000126, rvol 4.800e-07, gaps 3.95/31.4 - all identical - and bid_up+bid_dn reproduces the old bidmoves count of 74 exactly, as it must. The orientation check that matters: OFI correlates +0.56 with the SAME-bar return. That is the contemporaneous signature the literature reports, and it is also the cheapest guard against the failure mode that would otherwise pass silently - a sign flip would read -0.56 and every downstream test would then be fitting the negative of the intended feature. Expectations set in the docstring rather than discovered later: OFI is well established as a contemporaneous EXPLAINER of price change and its predictive power decays within seconds. At M5 with multi-hour horizons the prior should be that it explains the bar it is measured in, not the next one. Measuring it anyway is the point - but a +0.56 contemporaneous correlation is not evidence of an edge and must not be reported as one. Merge/mean bookkeeping is now index-driven off COLUMNS instead of positional, so adding a feature cannot silently mis-merge a bar that straddles a batch boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
0526f066ec |
research: stream tick files into bars with microstructure features
sqx.decode_iter() turns the decoder into a generator, and ticks_to_bars.py reduces a
symbol to bars in one bounded-memory pass. Necessary rather than tidy: EURUSD is ~458M
ticks, which is ~15 GB held as arrays, so nothing downstream can take the raw stream.
23 years collapses to ~2.4M M5 bars that every test can load instantly.
Aggregation is vectorised with reduceat rather than looping per tick. The only real
complexity is that a bar can straddle a batch boundary, so the last partial bar of each
batch is carried and merged into the first of the next; per-tick deltas are likewise seeded
from the previous batch's final tick, so the first tick of a batch is not silently treated
as having no predecessor. Verified against the per-tick implementation it replaces: 21,971
bars either way, and every reported median identical to the digit (ticks/bar 76, up 38, dn
38, bidmoves 74, askmoves 74, spread 0.000126/0.000250, rvol 4.800e-07, gaps 3.95/31.4).
Throughput 132k ticks/s, at which point the decoder itself is the bottleneck and the
aggregation costs ~12%.
Features are chosen by what the feed can honestly support. It carries (time, bid, ask,
volume) and no trade direction - SQX's record has one volume field and MT5's
TICK_FLAG_BUY/SELL are empty on FX - so true signed order flow does not exist here and is
not synthesised under a flattering name. What is available:
tick rule up/down mid-price changes; the standard Lee-Ready fallback
quote asymmetry bid updates vs ask updates - which side is being repriced harder
arrival rate inter-tick gaps, mean and max; urgency rather than size
realised variance sum of squared mid returns, a far better volatility estimate than
the bar range and only obtainable from ticks
spread mean and max within the bar
Of these only the tick rule and quote asymmetry can point a direction; the rest are
unsigned, like every feature that has measured above noise in this project so far.
Bars are stamped by the OPEN of their interval and built only from ticks inside it, so no
bar's features depend on a tick after it closes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
ecb8638996 |
chore: keep market data out of the repo
A 2.1 GB tick .dat was committed in c7e9777 by a broad 'git add -A' and removed again in 6bb3386 - but a delete does not remove the blob from history, so the 2096 MB object is still reachable and still gets pushed. That is what made syncing hang. Ignoring the paths only prevents a recurrence; clearing the existing blob needs a history rewrite, which is the user's call since it rewrites pushed commits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
ef126845f8 |
research: decode StrategyQuant .dat tick files
Reads SQX tick format 4.2 in pure Python. Derived from SQX's own writer, disassembled out
of internal/libs/SQDataLib.jar (TickDataWriter, NewDataFormat{,Writter}) with the javap
bundled in the install - so this follows the format as specified rather than as guessed.
Why, when Scripts/ExportTicks.mq5 pulls the same four fields from MT5: DEPTH. The broker's
MT5 tick history covers a few years; this file starts 2011-09-19. Sample size has been the
binding constraint on every question in this project - the H1/128-bar setup yields ~300
independent trades in 18 years, enough to resolve only a +6pp edge - so 15 years of ticks
is worth a decoder.
Format: four writeUTF strings, ten zero bytes, one more writeUTF, then records of
(time, ASK, BID, volume) - ask before bid, and the writer swaps them when bid>ask so ask is
always the larger. Every BLOCK_LENGTH=1000 records: MAGIC (15 bytes 0x00..0x0e) + int32
block index + config + four raw int64s. In between, deltas against the previous record.
Config is two bytes = four nibbles laid out high-first, nibble = (logicType << 2) |
dataType, where dataType 0..3 selects a 1/2/4/8-byte payload by magnitude and logicType
supplies the sign (MINUS/PLUS carry unsigned magnitudes; ASIS is a plain signed read).
The scale is the one thing NOT in the file. SQX keeps `decimals` in external metadata, and
1216010000 is equally plausible at 10^3, 10^5 or 10^6 - nothing in the bytes distinguishes
them. Guessing would be precisely the silent, plausible-looking error this project keeps
getting caught by: a 100x price scale error crashes nothing, it just quietly rescales every
ATR-normalised feature downstream. So calibrate_decimals() matches against a known
reference series instead. Against the MT5 SP500 H1 export the answer is not marginal:
decimals=5 median rel.err 9.001630
decimals=6 median rel.err 0.004078 <-
decimals=7 median rel.err 0.899984
Verified on 4M ticks: strictly monotonic timestamps, zero negative spreads, price range
1118.03-2048.38 over 2011-09 to 2014-11 (correct for SP500), median spread 0.43 (matches
this broker's H1 record). The 0.41% residual is the expected artefact of comparing a tick
ask against the nearest H1 bar close.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
87ee9c0a43 |
tools: MQL5 tick exporter for offline microstructure research
Dumps CopyTicksRange output to CSV in the COMMON files folder, where research/ already reads its rate exports from. Chunked by hour range because a single unbounded request over years is both slow and liable to ERR_HISTORY_SMALL_BUFFER; boundaries are half-open on purpose since CopyTicksRange is inclusive at both ends and adjacent chunks would otherwise duplicate any tick landing exactly on a split. Keeps MqlTick.flags RAW rather than decoding to a direction. On FX/CFD only TICK_FLAG_BID/ASK are ever set - TICK_FLAG_BUY/SELL and volume/volume_real are empty for Forex - so signed trade direction does not exist in this feed and has to be synthesised offline from quote dynamics. Exporting a decoded 'side' column would be inventing data. Written as the reliable alternative to decoding StrategyQuant's .dat: that format's base record parses cleanly (32 bytes, ms timestamp + bid + ask + one volume, prices x1e6, verified against a known SP500 level) but the delta stream is a custom bit-aligned dictionary scheme, and it carries only ONE volume field - so it offers nothing MT5 does not already provide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
ceb6342dfd |
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks
Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature,
default on). Spread is the one microstructure channel that is both FX-available and
genuinely historical in the Strategy Tester - "during testing, the spread is not modeled
but is taken from historical data" - so unlike swap, signed tick flow or depth of market it
is something a backtest can honestly validate.
What it encodes, stated precisely because the raw measurement overstates it.
research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5
of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges
the spread inside its own barriers, so a wide-spread bar is mechanically likelier to
resolve as a loss and the feature would partly be predicting its own cost model. Relabelling
at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology
and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime
reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when
realised volatility is below its own ATR estimate, which genuinely predicts whether
ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side.
Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated
in the spread series. Both cached on length alone:
if(m_crossAsset.Bars() >= bars) return true;
MQL5 series indices are relative to NOW, so one new closed candle shifts every index by
one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer
the newest, and every cross-asset value is read one bar out of step with the price features
sitting beside it in the same vector - silently, with no error and no shape change. This is
the same class of defect as the dtStudied watermark behind the zero-direction backtests.
Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the
label/feature bar caches already use.
And a performance fix that fell out of it: with correct invalidation the panel rebuilds on
every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one
full multi-symbol resample per simulated bar at training depth. Inference only reads bars
0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The
cache check is >=, so a deeper panel left from training still satisfies it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
4881ad22b5 |
research: measure the spread channel, and separate its real signal from its own cost model
Spread is the one microstructure channel that survived the API audit - FX-available, and
genuinely historical in the tester ("During testing, the spread is not modeled but is taken
from historical data"), unlike swap (no history), signed tick flow (empty on FX) or depth
of market (absent on retail FX, never replayed).
test_spread.py measures four spread features against the triple-barrier label with the same
block-permutation null as test_volume.py. spr/atr - cost relative to the volatility a trade
must overcome - is the strongest reading anywhere in this project so far: significant on 5
of 8 instrument/geometry cells and 2-4x the magnitude of any volume feature.
Which immediately looked too good, because the label is computed WITH the spread charged
inside the barriers. A bar with high spread/ATR has its barriers shifted more adversely and
is mechanically likelier to resolve as a loss - so the feature would partly predict its own
cost model, which is not tradeable information.
Tested directly by relabelling at zero cost and re-measuring the identical feature:
EURUSD 2:3 +0.000655 -> +0.000485 (p 0.030 -> 0.066, loses significance)
EURUSD 1:2 +0.000626 -> +0.000399 (p 0.003 -> 0.017)
USDJPY 2:3 +0.000418 -> +0.000164 (never significant either way)
USDJPY 1:2 +0.000876 -> +0.000532 (p 0.003 -> 0.003)
XAUUSD 2:3 +0.000769 -> +0.000744 (p 0.027 -> 0.027)
XAUUSD 1:2 +0.000876 -> +0.000698 (p 0.003 -> 0.003)
So roughly 20-40% of it WAS the tautology, and the majority is not. What remains is a
volatility-regime reading: spread is near-fixed while ATR is not, so spr/atr is high
exactly when realised volatility is running below its own ATR estimate - which genuinely
predicts whether ATR-scaled barriers get reached at all.
Note what that does and does not buy. Like volume, spread is UNSIGNED: it informs Neutral
vs directional, never Buy vs Sell. It is the best-measured feature in this project and it
still cannot pick a side.
No EA changes in this commit - measurement only, and the MQL5 side already has an
uncompiled backlog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
d80d9444a5 |
feat(ai): widen the volume feature block from 1 value to 4
The block fed exactly one number: (v[i] - v[i-1]) / v[i-1]. That is the first difference, and it cannot express three things that matter - the LEVEL relative to a baseline (two dead bars and two frantic bars both read ~0 change), and the two volume-vs-range interactions, where heavy participation that went NOWHERE (absorption) and heavy participation that travelled (continuation) mean opposite things and currently collapse onto the same value. research/test_volume.py measures each candidate's mutual information with the triple- barrier label across 3 instruments x 2 geometries, against a BLOCK-permutation null - blocks sized to the barrier horizon, because adjacent labels share almost their entire outcome window and a free shuffle yields a null so tight that everything looks significant. Finite-sample MI bias (~7/n here) is reported alongside rather than subtracted, since the permutation null already absorbs it. Result: volLevel beats the shipped change ratio outright on 4 of 6 cells (EURUSD 2:3 +0.000118 excess at p=0.006, USDJPY 1:2 +0.000284 at p=0.002); absorption is the single strongest reading anywhere in the sweep at EURUSD 1:2 (+0.000404, p=0.002) though it is null on XAUUSD; vol x range clears on 4 of 6. The shipped change ratio is itself significant on 5 of 6, so it stays. Kept OUT: a session-relative z-score against the same hour-of-day's own recent history. It was the weakest candidate - null on both EURUSD cells - and it is the only one needing per-hour rolling bookkeeping in MQL5. Not worth the state for a reading that did not survive its own null on the primary instrument. Magnitudes, stated plainly because they are the point: the excess MI is ~2e-4 nats against a label entropy near 1.05. That is under a tenth of one percent of the label's uncertainty. It is real, it repeats across instruments, and it is nowhere near an edge - this is worth having because it costs one 50-bar loop, not because it changes the answer. Prior work stands: the whole single-series feature family measured at the noise floor. m_neuronsCount is already in the fingerprint, so the width change re-keys existing caches by itself, which is correct - the input vector genuinely changed shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
b035ea29e5 |
feat(ai): cross-asset currency strength - the first feature not derived from one price series
Every feature the network sees today is a function of the traded symbol's own OHLCV: returns, ranges, oscillators, cloud distances, swing structure. Measured end to end that whole family sits at the noise floor (research/test_classic.py, and the mutual-information verdict before it). EURUSD moving is a statement about EUR and about USD, and which one moved is invisible from EURUSD alone - but plainly visible if you also look at EURJPY, GBPUSD and the rest. System\CrossAsset.mqh builds a currency-strength panel from the FX pairs in Market Watch: per bar, each currency's index is the average log return across every available pair containing it, signed so "up" always means that currency strengthened. Six features - base and quote strength at 1 and 20 bars, the DIVERGENCE between the pair and what its two currencies separately did, and the cross-sectional dispersion of currency moves as a regime term. The divergence is the thesis: it is the one value here that cannot be derived from the traded series at all, being defined only relative to the rest of the market. Built ONCE per training run against the traded symbol's bar grid, not per bar - a per-bar cross-symbol lookup would be pairs x 178k iBarShift calls. Correctness work, all of it driven by what MT5 actually guarantees rather than by what the API surface suggests: - Alignment is by TIMESTAMP, never by index. Bars do not open together across symbols, and in the tester each symbol gets its own generated tick sequence, so index k on GBPUSD and index k on USDJPY are not the same instant. Each traded bar takes the last reference bar at or BEFORE its timestamp - never after, which would be lookahead - and anything more than one bar period stale is treated as absent rather than carried forward across a holiday gap. - SeriesReady() gates every pair on SymbolSelect + SymbolIsSynchronized + the PER-TIMEFRAME SERIES_SYNCHRONIZED. The symbol-wide and per-timeframe flags can disagree because the terminal builds series on separate threads, so checking only the first is not enough. Non-blocking by design: an unready pair is skipped and picked up on a later build. - Failure is never fatal. Fewer than two usable pairs logs why and every Features() call 0-fills, so a missing reference symbol costs the context block rather than the whole run. Fingerprint: the flag goes in, the DISCOVERED REFERENCE SET does not. Which pairs exist in Market Watch is a measured property of the terminal, exactly like the bar count the existing comment warns about - keying the weights filename on it would orphan a trained model the moment the user adds a symbol, silently, because a missing cache reads as a normal first run. Defaults ON, which re-keys existing databases on first run. That is intended: the input vector genuinely changed shape. Deliberately NOT built, having checked what the platform actually provides: - swap/carry. SYMBOL_SWAP_LONG/SHORT have no history - "last values will be used for the whole test period" - so a backtest over 2020-2026 applies 2026 carry to 2020 bars. - signed order flow. TICK_FLAG_BUY/SELL and volume_real are empty on Forex; any feature assuming trade direction would silently be all zeros. - depth of market. Unavailable on retail FX symbols and never replayed in the tester. - calendar actual-vs-forecast surprise. MqlCalendarValue.actual_value is the FINAL, post-revision figure and the calendar keeps no as-of-release snapshot, so a surprise feature for a 2019 bar is built from a number nobody had in 2019. Needs a live recorder, not a historical read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
8710240cd5 |
fix(signals): revive a dead MA model, and demote Sanyaku from state to event
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
ae738f59f0 |
chore(research): drop committed __pycache__, add .gitignore
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
bd076ddbad |
research: test the shipped classic patterns for entry edge - and the lookahead that faked one
Transcribes all 26 classic vote models (MA 4, RSI 4, MACD 6, Ichimoku 12) from
Signals/*.mqh into vectorised Python, with their shipped constructor weights, then tests
them as entry triggers on 178k-bar FX histories.
Pre-registered by construction: the rules were written long before this test and nothing
about them is fitted here, so there is no in-sample/out-of-sample split to draw and the
whole history is usable. Break-even == chance by the gambler's-ruin identity, so "beats a
coin" and "makes money" are one question. Sequential non-overlapping trades only; a
sign-flip null over the whole pattern family gives the family-wise bar.
The result that matters is a negative one, and it took two lookahead fixes to see:
- _price_extremum reproduced the standard library's CENTRED MinValue(pos-2,5) window,
which reads up to 2 bars newer than the extremum it describes.
- turning_points marked a turn AT bar i, which is only knowable once bar i+1 closes.
Together those two bars of leakage WERE the entire apparent edge. MACD_p4 on EURUSD 1:2
read +5.05pp at +4.05 sigma before, -0.02pp at -0.02 sigma after; USDJPY 1:2 went +5.24pp
-> +0.02pp. RSI_p2's large NEGATIVE went the same way (-10.33pp -> -1.34pp), which is the
tell: a leak inflates whatever sign it lands on.
With both closed, across 4 instruments x 3 geometries: no pattern, no vote threshold, no
quorum and no event+confirmation rule separates from chance. One cell in ~180 tests stars
(SP500 2:6 vote>=30) and it is non-monotone in the threshold either side of the hit.
test_exits.py answers the trade-management half with the control that makes it mean
something: hold entries fixed, vary only the exit, and run every rule again on RANDOM
entries at the same bars. Breakeven-at-1R, chandelier trails, partials and time stops all
move E[R] - and move it by the same amount on random entries. No rule beats its own
control (max +0.99 sigma over 32 comparisons). Management reshapes the win-rate/payoff
split; it does not manufacture expectancy from a directionless entry.
Residual E[R] across every cell is -0.01 to -0.08 R, which is approximately the spread.
Incidental, both worth fixing in the EA:
- CSignalMA pattern 1 is unsatisfiable at the shipped EMA default. For an EMA,
MA[i]-MA[i-1] and c[i]-MA[i] are both positive multiples of (c[i]-MA[i-1]), so
"close below the MA while the MA rises" cannot occur. Dead code (weight 10).
- Ichimoku pattern 11 (Sanyaku, weight 100, the method's top signal) fires on 27% of
bars because it is a conjunction of three standing STATES with no event term, so it
dominates the averaged vote while carrying no trigger information.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
b6a067b266 |
research: offline validation kit, and the answer it produced
Keeps the Python that turned a per-hypothesis cost of minutes into
seconds, so the next person (or the next me) can re-run any of this
without MetaTrader in the loop.
kit.py vectorised triple-barrier labeller + scale-free feature set.
The labeller is a faithful port - stop tested before target
within a bar, so a bar spanning both scores as the loss -
and reproduces the EA's own distribution to 0.05pp
(24.93/22.01/53.06 vs 24.9/22.0/53.1) at 12x the speed.
wf.py purged, embargoed walk-forward gradient boosting.
sim.py sequential NON-OVERLAPPING trade simulation.
sweep.py the cell x geometry sweep.
detail.py full threshold profile for one configuration.
WHAT IT FOUND, and the order matters because the first answer was wrong:
Naive walk-forward looked like an edge - precision rising monotonically
with model confidence, 24.4/24.6/25.3/26.2/27.1%, topping out at +2.66pp
and 2.9 sigma. All of it pseudo-replication: a 128-bar barrier means
adjacent bars share almost their whole outcome window, so one trade was
being counted up to 128 times. Counting each trade ONCE (sim.py) the
ordering collapses to 25.9/31.1/26.8/28.1/25.8 and nothing is
significant. Same error family as the MI null that assumed independence.
That exposed a structural problem bigger than the result: at a 128-bar
horizon, 18 years of SP500 H1 yields at most ~300 independent trades,
which can only resolve a +6pp edge at 2 sigma. Real edges are 1-3pp. The
shipped configuration is not merely unproven - it is statistically
UNFALSIFIABLE on the available history.
The cost/power screen then showed SP500 is among the worst cells
available: 54k bars and spread/ATR 0.34, against EURUSD/USDJPY at 178k
bars and 0.05. Weeks of training went into the hardest instrument on the
list, 3x less data and 7x the relative cost.
Final sweep - 4 instruments x 3 geometries, purged walk-forward,
independent trades: nothing clears +2 sigma. The one survivor (EURUSD
2:3 h48, +2.76pp at +1.57 sigma) dissolves under its full threshold
profile: non-monotone across thresholds, and its per-fold win rate decays
monotonically through time (52.9 -> 45.9 -> 41.7 -> 35.2 -> 26.2).
Conclusion: with price/volume-derived technical features there is no
tradeable entry-direction edge on these instruments - now tested with a
model class that finds interactions, on 3x the data, with honest
statistics.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|