forked from animatedread/Warrior_EA
9a7c37f334
Three changes, one theme: the trade placed, the trade graded, and the trade computed are now the same trade. 1) GEOMETRY WIRE (correctness, the ranked #1 open issue). The measured barrier pair reached the LABELS only - OpenParams still placed orders at the enum geometry (2*ATR/6*ATR), so the deploy gate certified "reaches 1.62*ATR before 3.33*ATR above break-even" about trades the EA never placed. Published via g_DerivedSlAtrMult/g_DerivedTpAtrMult (ConfidenceBridge, same same-tick contract as the confidence globals, because OpenParams runs on the root signal which has no pointer to the AI filter). Two writers: DeriveBarrierGeometry at era 0, and the .cfg adoption a deployed model takes. Overrides both legs and both Intelligent modes - the certificate is exact or it is nothing. TP is ATR-anchored like the label, NOT risk-relative, so a floor-widened stop cannot reshape the certified target. 2) BATCH NORM RUNS DEVICE-SIDE ON OPENCL. Four kernels in Network.cl - forward, hidden gradient, gamma/beta accumulate, gamma/beta apply - each a line-for-line transcription of the host implementation (NormalizeHost / HiddenGradHost / StepGammaBeta) including every NaN guard, clamp, and the exact moment-write ordering. The host copies remain the runtime for the DLL and pure-MQL5 tiers and the reference the kernels must match. Because this box has no OpenCL platform, the safety story is layered: - shim validation: kernels compiled as C and driven against a fp64 host transcription over NaN-poisoned stats, NaN gamma, over-clamp inputs, the frozen path, both optimizers, 3 batches - ALL PASS, worst normalized diff 0.132 vs tolerance 1.0 - in-situ self-check: each kernel is compared against its host twin ON FIRST USE on the real device (SelfCheckBn*), covering what the shim cannot - arg indices and buffer bindings. Any disagreement resyncs from the good copy, latches all BN kernels off process-wide, and training continues host-side. A transcription bug costs a warning and some speed, never a poisoned .nnw. - sync discipline: BatchOptions is now a CBufferDouble with explicit authority tracking (m_bnDeviceAuthoritative). Checkpoints/saves pull read-only; restores/loads/resets push; a mid-batch handover drains the device gamma/beta accumulator into the host arrays so no sample is lost. 3) SMALL FIXES. Apply-kernel build failure now latches the dispatch path at init (one warning instead of warning + failed Execute). Build tag bumped to win-scoring-gpu-v1 - first tag change since expectancy-stop-v1 despite five binary-changing commits. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1374 lines
72 KiB
MQL5
1374 lines
72 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| NeuronBatchNorm.mqh |
|
|
//| AnimateDread |
|
|
//| https://www.mql5.com |
|
|
//+------------------------------------------------------------------+
|
|
//| CNeuronBatchNormOCL - batch normalization (Ioffe & Szegedy 2015). |
|
|
//| Needs CNeuronBaseOCL (AI\Network.mqh) already declared; included |
|
|
//| from there, not standalone. |
|
|
//| |
|
|
//| WHY THIS EXISTS |
|
|
//| Every hidden stage in this project is unbounded (PRELU dense, |
|
|
//| PRELU conv) and the only bounded stage in the whole forward path |
|
|
//| was the SIGMOID classification head. That is a network with no |
|
|
//| internal scale control at all, and it showed: across four |
|
|
//| topologies on SP500 H1 the models peaked early then decayed |
|
|
//| monotonically into all-Neutral, and the decay ordered exactly by |
|
|
//| DEPTH - the shallow perceptron held ~52% balanced accuracy while |
|
|
//| the deepest (conv+pool+LSTM+dense taper) sat at the 33.3% |
|
|
//| one-class floor. That depth ordering is the signature of internal |
|
|
//| covariate shift, which is precisely what this layer addresses. |
|
|
//| |
|
|
//| Two further consequences worth spelling out, because they are the |
|
|
//| actual mechanism by which this is expected to help here: |
|
|
//| |
|
|
//| 1. It decouples weight decay from the learned function. With a |
|
|
//| normalized layer downstream, scaling the weights that feed it |
|
|
//| leaves the output unchanged - so WEIGHT_DECAY can no longer |
|
|
//| grind the discriminative signal away, it only rescales the |
|
|
//| effective learning rate (van Laarhoven 2017). The observed |
|
|
//| failure was exactly a slow monotonic shrink of the per-bar |
|
|
//| logit spread (0.45 -> 0.38 over ~200 eras) until the evidence |
|
|
//| tilt fell under the class-prior tilt and argmax degenerated |
|
|
//| to constant-Neutral. See WEIGHT_DECAY's comment in |
|
|
//| AI\Network.mqh, which describes that mechanism first-hand. |
|
|
//| 2. It is what would make an UNBOUNDED logit head viable. The |
|
|
//| 2026-07-27 attempt to run the head at NONE blew up (IS error |
|
|
//| 5.6e15) specifically because nothing upstream constrained |
|
|
//| scale. Do not retry that without a normalization layer |
|
|
//| immediately before the head - and then also drop |
|
|
//| CLASS_LOGIT_SCALE to 1.0 and BIAS_MAGNITUDE to ~0.5. |
|
|
//| |
|
|
//| IMPLEMENTATION SHAPE - host-side reference, OpenCL kernels on top |
|
|
//| The math is elementwise O(n) and originally ran host-side on all |
|
|
//| four tiers, against the host mirrors of the device buffers, so |
|
|
//| the backends could not drift. The predicted cost - "on a real GPU |
|
|
//| it is a PCIe hop; if that ever matters, port these methods to |
|
|
//| kernels - the math here is the specification" - came due on |
|
|
//| 2026-08-09: the blocking syncs (four per BN layer per sample, |
|
|
//| forward AND backward) were a large part of why an RX 580 lost to |
|
|
//| a CPU thread pool, and Market builds forbid DLL imports, so the |
|
|
//| OpenCL tier is what paying clients actually run. |
|
|
//| The port did exactly what that sentence said: Network.cl's |
|
|
//| BatchNorm* kernels are transcriptions of NormalizeHost / |
|
|
//| HiddenGradHost / StepGammaBeta, the host methods REMAIN the |
|
|
//| runtime for the DLL and pure-MQL5 tiers and the reference the |
|
|
//| kernels must match, and each kernel is verified against its host |
|
|
//| twin on first use (SelfCheckBn*) - a disagreement latches the |
|
|
//| kernels off process-wide (g_bnKernelUsable) after resyncing from |
|
|
//| the good copy, so a transcription bug costs a warning and some |
|
|
//| speed, never a poisoned .nnw. Edit host and kernel together. |
|
|
//| |
|
|
//| STATISTICS: exponential moving, not a stored mini-batch. |
|
|
//| Training here is pure online SGD - one weight update per sample |
|
|
//| (CNet::backProp per bar), never a batched pass - so there is no |
|
|
//| mini-batch to average over. iBatchSize is therefore an EMA WINDOW |
|
|
//| LENGTH, not a buffer size: mean and variance are updated per |
|
|
//| sample toward the last ~iBatchSize samples. Same formulation as |
|
|
//| the NeuroNet_DNG reference's BatchFeedForward kernel; the book |
|
|
//| (ch. 6.1.1) explicitly endorses the exponential form to avoid |
|
|
//| storing per-neuron history. |
|
|
//| |
|
|
//| The statistics keep adapting on EVERY forward pass, including |
|
|
//| out-of-sample scoring and live inference - they are never frozen |
|
|
//| the way classic batch norm freezes them at inference time. That |
|
|
//| is deliberate and matches the rest of this system (see |
|
|
//| OnlineLearnStep: the deployed model is designed to keep tracking |
|
|
//| the market). Two consequences worth knowing: OOS scoring lets the |
|
|
//| statistics see OOS activations - unsupervised, no label |
|
|
//| information, but not a hermetic holdout - and a restored |
|
|
//| checkpoint rewinds the statistics along with gamma/beta, since |
|
|
//| getWeightsBN/setWeightsBN carry both. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_NEURON_BATCHNORM_MQH
|
|
#define WARRIOR_NEURON_BATCHNORM_MQH
|
|
//--- Per-neuron slot layout inside BatchOptions. A FIXED stride of 9 on every optimizer, unlike the
|
|
//--- reference's 7-or-9 split: the two unused doubles per neuron are noise next to a weight matrix,
|
|
//--- and a stride that changes with a persisted enum is an indexing bug waiting for the first time
|
|
//--- somebody switches TrainingOptimizer on an existing model.
|
|
#define BN_OPT_STRIDE 9
|
|
#define BN_OPT_MEAN 0 // running mean
|
|
#define BN_OPT_VAR 1 // running variance
|
|
#define BN_OPT_NX 2 // normalized input, cached from the forward pass for the backward pass
|
|
#define BN_OPT_GAMMA 3 // learned scale, init 1
|
|
#define BN_OPT_BETA 4 // learned shift, init 0
|
|
#define BN_OPT_MG 5 // gamma: Adam first momentum, or SGD previous delta
|
|
#define BN_OPT_MB 6 // beta: Adam first momentum, or SGD previous delta
|
|
#define BN_OPT_VG 7 // gamma: Adam second momentum (stored already square-rooted, as the
|
|
#define BN_OPT_VB 8 // beta: ... UpdateWeightsAdam kernels in this engine also do)
|
|
//--- Variance floor. Serves the same purpose as MIN_ACTIVATION_DERIVATIVE: a neuron whose input
|
|
//--- happens to be near-constant over the EMA window has a near-zero variance, and dividing by its
|
|
//--- square root turns rounding noise into an arbitrarily large activation that then propagates.
|
|
//--- Applied to the standard deviation (not the variance) so it reads as "no unit is amplified by
|
|
//--- more than 1e4", which is the property that actually matters.
|
|
#define BN_EPSILON 1.0e-10
|
|
#define BN_MIN_STD 1.0e-4
|
|
//--- Sanity ceiling on a single input value, applied before it can touch the running statistics below.
|
|
//--- Not a normalization choice - every legitimate feature in this codebase is clamped to single digits
|
|
//--- long before it gets here (see BufferTempDataCompute). This is purely the bound that keeps the
|
|
//--- statistics arithmetic inside double range, so a garbage value degrades one unit on one bar instead
|
|
//--- of permanently latching the layer. See NormalizeHost() for the failure this exists to stop.
|
|
#define BN_MAX_INPUT 1.0e6
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
class CNeuronBatchNormOCL : public CNeuronBaseOCL
|
|
{
|
|
protected:
|
|
int iBatchSize; // EMA window length; <=1 disables normalization entirely
|
|
//--- When true the running statistics are USED but not UPDATED, i.e. classic batch-norm inference
|
|
//--- semantics. Off by default (see the note on adaptation in the class header). Exists because a
|
|
//--- forward pass is otherwise not a pure function of its input: two evaluations of the same bar
|
|
//--- return slightly different answers because the first one moved the statistics. Anything that
|
|
//--- COMPARES two forward passes has to freeze them first or it is measuring its own side effect -
|
|
//--- CExpertSignalAIBase::ValidateCpuInference() is exactly that, and would have silently failed
|
|
//--- its DLL-free check on nothing but the one EMA step it took itself.
|
|
bool bStatsFrozen;
|
|
//--- Number of samples seen. Used only to ramp the effective window up from 1 to iBatchSize over
|
|
//--- the first iBatchSize samples (standard EMA bias correction). Without it the running mean
|
|
//--- starts at 0 and approaches the true mean over ~iBatchSize samples, during which the layer
|
|
//--- emits near-zero for everything - a cold-start dead zone indistinguishable from the very
|
|
//--- collapse this layer exists to prevent.
|
|
int iSamplesSeen;
|
|
//--- Persisted through Save/Load like any other parameter - gamma/beta are learned, and the running
|
|
//--- statistics ARE the layer's inference behaviour, so a model that loses them is not the model
|
|
//--- that was trained.
|
|
//--- A CBufferDouble since 2026-08-09 so the OpenCL kernels can own it device-side; on every other
|
|
//--- tier it behaves exactly as the CArrayDouble it was (no device buffer is ever created). WHO IS
|
|
//--- AUTHORITATIVE is tracked by m_bnDeviceAuthoritative below, and every host read/write of this
|
|
//--- buffer goes through the sync helpers - an unsynced BufferRead would clobber good host state
|
|
//--- with stale device state, and an unsynced host write would be silently overwritten by the next
|
|
//--- kernel. The 9-slot stride is baked into persisted .nnw files and into Network.cl's BN_OPT_*
|
|
//--- copies; the two must stay identical.
|
|
CBufferDouble *BatchOptions;
|
|
//--- Mini-batch running sums of dL/dgamma and dL/dbeta, one entry per unit. Host-only and NOT
|
|
//--- persisted: transient within a batch, and every save point flushes first. See
|
|
//--- accumulateInputWeightGrads for why they are not extra BatchOptions slots.
|
|
double m_accGamma[];
|
|
double m_accBeta[];
|
|
//--- PER-SAMPLE TRANSFER CACHES. This layer's math is host-side while its neighbours are device
|
|
//--- resident, so every value it touches crosses the bus - and each crossing is a BLOCKING sync,
|
|
//--- which is what makes it expensive rather than the bytes. Per sample it used to do four reads:
|
|
//--- feedForward -> previous layer's Output
|
|
//--- calcInputGradients -> own Gradient, and the previous layer's Output AGAIN
|
|
//--- update/accumulate -> own Gradient AGAIN
|
|
//--- The two repeats are exact duplicates. Nothing writes the previous layer's Output between the
|
|
//--- forward pass and the backward pass, and nothing writes this layer's Gradient between the
|
|
//--- gradient pass and the weight-update pass - CNet::backPropOCL runs those as two separate
|
|
//--- top-to-bottom loops, and only the first one writes gradients.
|
|
//---
|
|
//--- Caching them is therefore bit-exact, not an approximation: the same values, read once.
|
|
//--- Each cache is armed by its producer and DISARMED by feedForward, so a consumer that runs
|
|
//--- without its producer having run this sample falls back to reading the buffer rather than
|
|
//--- silently using the previous sample's data. That matters concretely: a batch-norm at layer 1
|
|
//--- never gets calcInputGradients called at all (backPropOCL's loop stops at layerNum > 0 - the
|
|
//--- same asymmetry documented there for a layer-1 LSTM), so its gradient cache is never armed.
|
|
double m_fwdInputCache[];
|
|
bool m_fwdInputCached;
|
|
double m_gradCache[];
|
|
bool m_gradCached;
|
|
//--- DEVICE PATH (OpenCL only, 2026-08-09). m_bnAcc holds the mini-batch gamma/beta gradient sums
|
|
//--- on the device (2 floats per unit: gamma then beta), the kernel twin of m_accGamma/m_accBeta.
|
|
//--- m_bnDeviceAuthoritative says the DEVICE copy of BatchOptions is the truth (kernels have
|
|
//--- written it since the last host sync); the m_bnChecked* flags latch each kernel's one-time
|
|
//--- self-check against its host twin. The checks are the whole safety story for shipping kernels
|
|
//--- that could not be built on the dev machine: a transcription or dispatch-binding error is
|
|
//--- caught on its first use, the layer resyncs from the good copy, latches the kernels off
|
|
//--- process-wide, and training continues host-side - a warning and some speed, never a poisoned
|
|
//--- .nnw.
|
|
CBufferDouble *m_bnAcc;
|
|
bool m_bnDeviceAuthoritative;
|
|
bool m_bnCheckedFwd;
|
|
bool m_bnCheckedGrad;
|
|
bool m_bnCheckedAccum;
|
|
bool m_bnCheckedApply;
|
|
//--- eligibility + buffer management for the kernel path
|
|
bool BnDeviceEligible(void);
|
|
bool EnsureBnDeviceBuffers(void);
|
|
//--- read-only pull of the device statistics into the host mirror (checkpoints/saves mid-training);
|
|
//--- device stays authoritative
|
|
void SyncOptionsToHost(void);
|
|
//--- full handover to the host path: pull statistics, drain the device accumulator into
|
|
//--- m_accGamma/m_accBeta so a mid-batch handover loses nothing, clear the flag
|
|
void EnsureHostAuthoritative(void);
|
|
//--- one-way process-wide latch + this layer's handover, with the reason printed once
|
|
void LatchBnKernelsOff(const string reason);
|
|
//--- kernel dispatches (return false on any SetArgument/Execute failure, no logging - the caller
|
|
//--- decides between latching and falling back)
|
|
bool DispatchBnForward(CNeuronBaseOCL *NeuronOCL, double w);
|
|
bool DispatchBnHiddenGrad(CNeuronBaseOCL *NeuronOCL);
|
|
bool DispatchBnAccum(void);
|
|
bool DispatchBnApply(double scale, double lt);
|
|
//--- one-time kernel-vs-host comparisons; each returns the OPERATION's result (true = the work got
|
|
//--- done correctly, by whichever path survived), never "the kernel matched"
|
|
bool SelfCheckBnForward(CNeuronBaseOCL *NeuronOCL);
|
|
bool SelfCheckBnHiddenGrad(CNeuronBaseOCL *NeuronOCL);
|
|
bool SelfCheckBnAccum(void);
|
|
bool SelfCheckBnApply(double scale, double lt);
|
|
//--- normalized disagreement: |got-ref| / (1e-3 * max(1,|ref|)), <=1 passes. DBL_MAX when exactly
|
|
//--- one side is non-finite. The 1e-3 relative band is ~4 decades above fp32-vs-fp64 noise and ~3
|
|
//--- below any real transcription error (wrong slot, wrong sign, wrong buffer), so it cannot
|
|
//--- confuse the two.
|
|
double BnDiffScore(double ref, double got);
|
|
//--- host twin of the backward elementwise math, factored out of calcInputGradients so the
|
|
//--- self-check compares against literally the same code the host path runs
|
|
void HiddenGradHost(const double &grad[], const double &prevOut[],
|
|
ENUM_ACTIVATION act, double &ig[], int n);
|
|
//---
|
|
virtual bool feedForward(CNeuronBaseOCL *NeuronOCL);
|
|
virtual bool feedForwardCPU(CNeuronBaseOCL *NeuronOCL);
|
|
virtual bool updateInputWeights(CNeuronBaseOCL *NeuronOCL);
|
|
virtual bool accumulateInputWeightGrads(CNeuronBaseOCL *NeuronOCL);
|
|
//--- one unit's gamma/beta step, shared by the per-sample and per-batch paths
|
|
bool StepGammaBeta(int shift, double gGamma, double gBeta, double lt);
|
|
|
|
public:
|
|
virtual bool BeginGradAccum(void);
|
|
virtual bool ApplyAccumulatedGradients(double scale);
|
|
protected:
|
|
//--- shared by feedForward/feedForwardCPU: the whole forward transform for one already-read input
|
|
//--- vector, writing straight into the host mirror of Output.
|
|
bool NormalizeHost(const double &inputs[], int count);
|
|
//--- ensures BatchOptions exists and is sized/seeded for `neurons` units
|
|
bool InitOptions(int neurons);
|
|
|
|
public:
|
|
CNeuronBatchNormOCL(void);
|
|
~CNeuronBatchNormOCL(void);
|
|
//---
|
|
virtual bool Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type);
|
|
virtual bool Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type);
|
|
virtual bool calcInputGradients(CNeuronBaseOCL *NeuronOCL);
|
|
//--- see bStatsFrozen. Deliberately NOT persisted: it is a transient evaluation mode, not model state.
|
|
void SetStatsFrozen(bool v) { bStatsFrozen = v; }
|
|
//--- Checkpoint/blend support. CNet::CaptureWeights/RestoreWeights/BlendWeightsFrom snapshot one
|
|
//--- flat array per neuron object, so gamma/beta/mean/variance are appended AFTER the outgoing
|
|
//--- dense weight matrix and split back out on restore. Without this the plateau ladder's
|
|
//--- "restore best checkpoint" would put the dense weights back while leaving this layer's
|
|
//--- parameters at whatever the diverged era left behind - a silently mismatched pair.
|
|
virtual int getWeightsBN(double &values[]);
|
|
virtual bool setWeightsBN(double &values[]);
|
|
//--- How many of the trailing entries in getWeightsBN's array are BatchOptions rather than the
|
|
//--- outgoing dense matrix. Published so the health report can split that packed block into its
|
|
//--- four very different parts (weights / gamma-beta / running statistics / Adam moments) instead
|
|
//--- of quoting one norm over all of them, which cannot say which is moving.
|
|
int BatchOptionsTotal(void) const
|
|
{
|
|
return (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
|
|
}
|
|
//--- Zero ONLY the gamma/beta moment slots (BN_OPT_MG/MB/VG/VB) plus the base class's buffers for
|
|
//--- the outgoing dense matrix. The running mean/variance and the learned gamma/beta are MODEL
|
|
//--- state, not optimizer state - they are exactly what getWeightsBN checkpoints and what a restore
|
|
//--- puts back, so wiping them here would undo the restore this reset exists to complete. See
|
|
//--- CNet::ResetOptimizerState.
|
|
virtual bool ResetOptimizerState(void)
|
|
{
|
|
bool ok = CNeuronBaseOCL::ResetOptimizerState();
|
|
if(CheckPointer(BatchOptions) != POINTER_INVALID)
|
|
{
|
|
//--- Kernel-mode discipline: this zeroes SOME slots of a block whose truth may live on the
|
|
//--- device, so pull first (or the untouched slots would be written back stale), zero, push.
|
|
SyncOptionsToHost();
|
|
int totalSlots = BatchOptions.Total();
|
|
for(int shift = 0; shift + BN_OPT_VB < totalSlots; shift += BN_OPT_STRIDE)
|
|
{
|
|
ok = BatchOptions.Update(shift + BN_OPT_MG, 0.0) && ok;
|
|
ok = BatchOptions.Update(shift + BN_OPT_MB, 0.0) && ok;
|
|
ok = BatchOptions.Update(shift + BN_OPT_VG, 0.0) && ok;
|
|
ok = BatchOptions.Update(shift + BN_OPT_VB, 0.0) && ok;
|
|
}
|
|
if(BatchOptions.GetIndex() >= 0)
|
|
ok = BatchOptions.BufferWrite() && ok;
|
|
}
|
|
return ok;
|
|
}
|
|
//---
|
|
virtual bool Save(int const file_handle);
|
|
virtual bool Load(int const file_handle);
|
|
//---
|
|
virtual int Type(void) const { return defNeuronBatchNormOCL; }
|
|
};
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
CNeuronBatchNormOCL::CNeuronBatchNormOCL(void) : iBatchSize(1), bStatsFrozen(false), iSamplesSeen(0),
|
|
m_fwdInputCached(false), m_gradCached(false), m_bnDeviceAuthoritative(false),
|
|
m_bnCheckedFwd(false), m_bnCheckedGrad(false), m_bnCheckedAccum(false), m_bnCheckedApply(false)
|
|
{
|
|
BatchOptions = NULL;
|
|
m_bnAcc = NULL;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
CNeuronBatchNormOCL::~CNeuronBatchNormOCL(void)
|
|
{
|
|
if(CheckPointer(BatchOptions) != POINTER_INVALID)
|
|
delete BatchOptions;
|
|
if(CheckPointer(m_bnAcc) != POINTER_INVALID)
|
|
delete m_bnAcc;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| gamma=1 / beta=0 / zeroed statistics for every unit. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::InitOptions(int neurons)
|
|
{
|
|
if(neurons <= 0)
|
|
return false;
|
|
if(CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
{
|
|
BatchOptions = new CBufferDouble();
|
|
if(CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
return false;
|
|
}
|
|
BatchOptions.Clear();
|
|
if(!BatchOptions.Reserve(neurons * BN_OPT_STRIDE))
|
|
return false;
|
|
for(int n = 0; n < neurons; n++)
|
|
for(int s = 0; s < BN_OPT_STRIDE; s++)
|
|
if(!BatchOptions.Add(s == BN_OPT_GAMMA ? 1.0 : 0.0))
|
|
return false;
|
|
iSamplesSeen = 0;
|
|
//--- Full host overwrite: if a device copy exists, push it so the two cannot disagree. The host is
|
|
//--- authoritative after a re-init by definition.
|
|
if(BatchOptions.GetIndex() >= 0)
|
|
{
|
|
if(!BatchOptions.BufferWrite())
|
|
return false;
|
|
m_bnDeviceAuthoritative = false;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type)
|
|
{
|
|
if(!CNeuronBaseOCL::Init(numOutputs, myIndex, open_cl, numNeurons, optimization_type))
|
|
return false;
|
|
//--- Identity forward transform. The activation belongs to the layer AFTER this one; normalizing
|
|
//--- and then squashing in the same step would defeat the point (Ioffe & Szegedy place the
|
|
//--- normalization immediately BEFORE the non-linearity, not around it).
|
|
activation = NONE;
|
|
iBatchSize = (int)batchSize;
|
|
return InitOptions((int)numNeurons);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type)
|
|
{
|
|
if(!CNeuronBaseOCL::Init(numOutputs, myIndex, direct_ml, numNeurons, optimization_type))
|
|
return false;
|
|
activation = NONE;
|
|
iBatchSize = (int)batchSize;
|
|
return InitOptions((int)numNeurons);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| The forward transform, host-side. Mirrors the NeuroNet_DNG |
|
|
//| BatchFeedForward kernel, plus the bias-corrected warm-up window. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::NormalizeHost(const double &inputs[], int count)
|
|
{
|
|
if(CheckPointer(Output) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
return false;
|
|
if(count > Output.Total())
|
|
count = Output.Total();
|
|
if(BatchOptions.Total() < count * BN_OPT_STRIDE)
|
|
return false;
|
|
//--- A window of 1 makes mean==x and variance==0 for every sample, i.e. a constant-zero output.
|
|
//--- Treat it as "normalization off" and pass the input through untouched rather than emit zeros.
|
|
if(iBatchSize <= 1)
|
|
{
|
|
for(int i = 0; i < count; i++)
|
|
if(!Output.Update(i, inputs[i]))
|
|
return false;
|
|
return true;
|
|
}
|
|
if(!bStatsFrozen && iSamplesSeen < iBatchSize)
|
|
iSamplesSeen++;
|
|
//--- Effective window: ramps 1,2,3... up to iBatchSize, so the first samples produce an honest
|
|
//--- running mean instead of one biased toward the zero initializer.
|
|
double w = (double)MathMax(1, iSamplesSeen);
|
|
for(int i = 0; i < count; i++)
|
|
{
|
|
int shift = i * BN_OPT_STRIDE;
|
|
double x = inputs[i];
|
|
//--- The running mean/variance below are PERSISTENT: they live in BatchOptions, are carried into
|
|
//--- the .nnw by getWeightsBN, and every later sample normalizes against them. That makes them a
|
|
//--- LATCH. One non-finite or astronomically large x poisons them permanently, and from then on
|
|
//--- this layer emits NaN on every bar for the rest of the run - the arithmetic gets there in
|
|
//--- three steps: (x-mean)^2 overflows variance to +inf, then mean*(w-1) overflows mean to +inf,
|
|
//--- then delta/sd is inf/inf = NaN. Note it does NOT need x to be non-finite to start: a merely
|
|
//--- huge finite x (an unguarded EMPTY_VALUE feature divided by ATR is ~1e307) passes every
|
|
//--- upstream check and still overflows the square.
|
|
//--- 2026-08-02: this presented as 13,776 identical "BufferWrite failed for buffer 3" lines and
|
|
//--- nothing else. Buffer 3 is the first batch-norm layer's Output - the one sitting on the raw
|
|
//--- input vector - and the CPU DLL's isfinite() boundary check was correctly refusing to store
|
|
//--- the NaN, so the layer's device-side output silently froze at its last good value while
|
|
//--- training carried on against it for a full era.
|
|
//--- A value this large is meaningless whatever produced it, so clamp rather than propagate.
|
|
if(!MathIsValidNumber(x))
|
|
x = 0.0;
|
|
x = MathMax(-BN_MAX_INPUT, MathMin(BN_MAX_INPUT, x));
|
|
double mean = BatchOptions.At(shift + BN_OPT_MEAN);
|
|
double variance = BatchOptions.At(shift + BN_OPT_VAR);
|
|
//--- Self-heal statistics that were already poisoned before this guard existed. Without it an
|
|
//--- affected .nnw stays dead across restarts, because Load faithfully restores the NaN.
|
|
if(!MathIsValidNumber(mean))
|
|
mean = x;
|
|
if(!MathIsValidNumber(variance) || variance < 0.0)
|
|
variance = 0.0;
|
|
if(!bStatsFrozen)
|
|
{
|
|
mean = (mean * (w - 1.0) + x) / w;
|
|
variance = (variance * (w - 1.0) + (x - mean) * (x - mean)) / w;
|
|
}
|
|
double delta = x - mean;
|
|
double sd = MathMax(MathSqrt(variance + BN_EPSILON), BN_MIN_STD);
|
|
double nx = delta / sd;
|
|
//--- gamma/beta cannot go non-finite going forward (updateInputWeights validates and clamps every
|
|
//--- step), but a model SAVED before that guard existed can carry NaN in here through Load. Same
|
|
//--- self-heal as the statistics above: fall back to the identity transform for that unit.
|
|
double gamma = BatchOptions.At(shift + BN_OPT_GAMMA);
|
|
double beta = BatchOptions.At(shift + BN_OPT_BETA);
|
|
if(!MathIsValidNumber(gamma))
|
|
gamma = 1.0;
|
|
if(!MathIsValidNumber(beta))
|
|
beta = 0.0;
|
|
double y = gamma * nx + beta;
|
|
//--- nx is still cached when frozen: it costs nothing and keeps the buffer consistent with the
|
|
//--- output just produced. There is no backward pass while frozen, so nothing reads it.
|
|
if(!bStatsFrozen &&
|
|
(!BatchOptions.Update(shift + BN_OPT_MEAN, mean) ||
|
|
!BatchOptions.Update(shift + BN_OPT_VAR, variance)))
|
|
return false;
|
|
if(!BatchOptions.Update(shift + BN_OPT_NX, nx))
|
|
return false;
|
|
if(!Output.Update(i, y))
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Forward. On OpenCL this is a KERNEL since 2026-08-09 (verified |
|
|
//| against NormalizeHost on first use, see SelfCheckBnForward); on |
|
|
//| every other backend it pulls the previous layer's output to the |
|
|
//| host, runs the transform, and pushes its output back. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::feedForward(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
if(CheckPointer(NeuronOCL) == POINTER_INVALID)
|
|
return false;
|
|
//--- A new sample starts here, so last sample's gradient is no longer valid to reuse. Disarmed
|
|
//--- BEFORE anything can fail below, so an aborted forward pass cannot leave a consumer holding a
|
|
//--- cache that looks current - see the declaration comment.
|
|
m_gradCached = false;
|
|
m_fwdInputCached = false;
|
|
//--- DEVICE PATH: nothing crosses the bus at all. The self-check runs the first time through and
|
|
//--- proves the kernel against NormalizeHost before the device is trusted with the persisted
|
|
//--- statistics; after that it is one dispatch per sample.
|
|
if(BnDeviceEligible() && NeuronOCL.getOutputIndex() >= 0 && getOutputIndex() >= 0 &&
|
|
EnsureBnDeviceBuffers())
|
|
{
|
|
if(!m_bnCheckedFwd)
|
|
return SelfCheckBnForward(NeuronOCL);
|
|
//--- Ramp bookkeeping stays host-side with iSamplesSeen; committed only on success so a failed
|
|
//--- dispatch that falls through to NormalizeHost (which increments itself) cannot double-count.
|
|
int seen = iSamplesSeen;
|
|
if(!bStatsFrozen && seen < iBatchSize)
|
|
seen++;
|
|
if(DispatchBnForward(NeuronOCL, (double)MathMax(1, seen)))
|
|
{
|
|
iSamplesSeen = seen;
|
|
return true;
|
|
}
|
|
LatchBnKernelsOff("BatchNormForward dispatch failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
}
|
|
//--- HOST PATH (every non-OpenCL tier, and OpenCL after a latch). If the device had been
|
|
//--- authoritative, pull its state first - NormalizeHost below must advance the REAL statistics,
|
|
//--- not a stale mirror.
|
|
EnsureHostAuthoritative();
|
|
double inputs[];
|
|
int count = NeuronOCL.getOutputVal(inputs);
|
|
if(count <= 0)
|
|
return false;
|
|
if(!NormalizeHost(inputs, count))
|
|
return false;
|
|
//--- Armed for calcInputGradients, which needs exactly these values and would otherwise read the
|
|
//--- same buffer back a second time this sample.
|
|
if(ArrayCopy(m_fwdInputCache, inputs, 0, 0, count) == count)
|
|
m_fwdInputCached = true;
|
|
//--- Unlike the dense/conv kernels this layer's Output does NOT stay device-resident by itself -
|
|
//--- it was just written host-side, so it has to be pushed before the next layer's kernel reads
|
|
//--- it through getOutputIndex().
|
|
return Output.BufferWrite();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Pure-MQL5 inference path (no backend at all): the previous |
|
|
//| layer's values live only in its host mirror, so read them there. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::feedForwardCPU(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
if(CheckPointer(NeuronOCL) == POINTER_INVALID)
|
|
return false;
|
|
int count = NeuronOCL.Neurons();
|
|
if(count <= 0)
|
|
return false;
|
|
double inputs[];
|
|
if(ArrayResize(inputs, count) != count)
|
|
return false;
|
|
for(int i = 0; i < count; i++)
|
|
inputs[i] = NeuronOCL.OutputHost(i);
|
|
//--- No BufferWrite: there is no device buffer on this path, and Output's host mirror is what
|
|
//--- the next layer's feedForwardCPU (and GetOutputsCPU) read.
|
|
return NormalizeHost(inputs, count);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| DEVICE PATH plumbing - see the m_bnAcc declaration comment for |
|
|
//| the design. Everything below is OpenCL-only. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::BnDeviceEligible(void)
|
|
{
|
|
return g_bnKernelUsable && iBatchSize > 1 &&
|
|
CheckPointer(OpenCL) != POINTER_INVALID &&
|
|
CheckPointer(BatchOptions) != POINTER_INVALID;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::EnsureBnDeviceBuffers(void)
|
|
{
|
|
//--- BufferCreate pushes the current host contents, so a freshly-created device copy is exactly
|
|
//--- the state the host was authoritative over - the invariant every self-check relies on.
|
|
if(BatchOptions.GetIndex() < 0 && !BatchOptions.BufferCreate(OpenCL))
|
|
return false;
|
|
if(CheckPointer(m_bnAcc) == POINTER_INVALID)
|
|
{
|
|
m_bnAcc = new CBufferDouble();
|
|
if(CheckPointer(m_bnAcc) == POINTER_INVALID)
|
|
return false;
|
|
if(!m_bnAcc.BufferInit(Neurons() * 2, 0.0))
|
|
return false;
|
|
}
|
|
if(m_bnAcc.GetIndex() < 0 && !m_bnAcc.BufferCreate(OpenCL))
|
|
return false;
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
void CNeuronBatchNormOCL::SyncOptionsToHost(void)
|
|
{
|
|
//--- Read-only pull for checkpoints/saves taken mid-training; the device REMAINS authoritative.
|
|
//--- Guarded by the flag, because an unguarded BufferRead on a host-authoritative layer would
|
|
//--- clobber good host state with whatever stale bytes the device still holds.
|
|
if(m_bnDeviceAuthoritative && CheckPointer(BatchOptions) != POINTER_INVALID &&
|
|
BatchOptions.GetIndex() >= 0)
|
|
BatchOptions.BufferRead();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
void CNeuronBatchNormOCL::EnsureHostAuthoritative(void)
|
|
{
|
|
if(!m_bnDeviceAuthoritative)
|
|
return;
|
|
if(CheckPointer(BatchOptions) != POINTER_INVALID && BatchOptions.GetIndex() >= 0)
|
|
BatchOptions.BufferRead();
|
|
//--- Drain the device-side batch accumulator into the host arrays, so a handover in the MIDDLE of
|
|
//--- a batch keeps the samples the kernels already accumulated - without this, latching off after
|
|
//--- sample 3 of 8 would silently drop three samples' gradients from the batch.
|
|
if(CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0 && m_bnAcc.BufferRead())
|
|
{
|
|
int units = Neurons();
|
|
if(ArraySize(m_accGamma) != units || ArraySize(m_accBeta) != units)
|
|
{
|
|
ArrayResize(m_accGamma, units);
|
|
ArrayResize(m_accBeta, units);
|
|
ArrayInitialize(m_accGamma, 0.0);
|
|
ArrayInitialize(m_accBeta, 0.0);
|
|
}
|
|
for(int i = 0; i < units && 2 * i + 1 < m_bnAcc.Total(); i++)
|
|
{
|
|
m_accGamma[i] += m_bnAcc.At(2 * i);
|
|
m_accBeta[i] += m_bnAcc.At(2 * i + 1);
|
|
}
|
|
ZeroOptimizerBuffer(m_bnAcc);
|
|
}
|
|
m_bnDeviceAuthoritative = false;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
void CNeuronBatchNormOCL::LatchBnKernelsOff(const string reason)
|
|
{
|
|
if(g_bnKernelUsable)
|
|
{
|
|
g_bnKernelUsable = false;
|
|
Print(__FUNCTION__ + ": BATCH-NORM KERNELS DISABLED for the rest of this run - " + reason +
|
|
". Every batch-norm layer falls back to the host implementation: results stay correct "
|
|
"(the host path is the reference the kernels were transcribed from), each layer just pays "
|
|
"its device round-trips again. State was resynced from the good copy before anything "
|
|
"could persist.");
|
|
}
|
|
EnsureHostAuthoritative();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
double CNeuronBatchNormOCL::BnDiffScore(double ref, double got)
|
|
{
|
|
bool fRef = MathIsValidNumber(ref), fGot = MathIsValidNumber(got);
|
|
if(!fRef || !fGot)
|
|
return (fRef == fGot) ? 0.0 : DBL_MAX; // both poisoned the same way is agreement
|
|
return MathAbs(got - ref) / (1.0e-3 * MathMax(1.0, MathAbs(ref)));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::DispatchBnForward(CNeuronBaseOCL *NeuronOCL, double w)
|
|
{
|
|
uint offset[1] = {0};
|
|
uint size[1];
|
|
size[0] = (uint)Neurons();
|
|
if(!OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_matrix_i, NeuronOCL.getOutputIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_matrix_o, getOutputIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_options, BatchOptions.GetIndex()) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormForward, def_k_bnf_w, (float)w) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormForward, def_k_bnf_frozen, bStatsFrozen ? 1 : 0))
|
|
return false;
|
|
ResetLastError();
|
|
if(!OpenCL.Execute(def_k_BatchNormForward, 1, offset, size))
|
|
return false;
|
|
//--- The kernel just wrote the running statistics on the device: it is now the authority.
|
|
m_bnDeviceAuthoritative = true;
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::DispatchBnHiddenGrad(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
uint offset[1] = {0};
|
|
uint size[1];
|
|
size[0] = (uint)Neurons();
|
|
if(!OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_matrix_g, getGradientIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_prev_o, NeuronOCL.getOutputIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_prev_g, NeuronOCL.getGradientIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_options, BatchOptions.GetIndex()) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormHiddenGrad, def_k_bnh_activation, NativeActivationCode(NeuronOCL.Activation())))
|
|
return false;
|
|
ResetLastError();
|
|
return OpenCL.Execute(def_k_BatchNormHiddenGrad, 1, offset, size);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::DispatchBnAccum(void)
|
|
{
|
|
uint offset[1] = {0};
|
|
uint size[1];
|
|
size[0] = (uint)Neurons();
|
|
if(!OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_matrix_g, getGradientIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_options, BatchOptions.GetIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_acc, m_bnAcc.GetIndex()))
|
|
return false;
|
|
ResetLastError();
|
|
return OpenCL.Execute(def_k_BatchNormAccumGammaBeta, 1, offset, size);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::DispatchBnApply(double scale, double lt)
|
|
{
|
|
uint offset[1] = {0};
|
|
uint size[1];
|
|
size[0] = (uint)Neurons();
|
|
if(!OpenCL.SetArgumentBuffer(def_k_BatchNormApplyGammaBeta, def_k_bnp_options, BatchOptions.GetIndex()) ||
|
|
!OpenCL.SetArgumentBuffer(def_k_BatchNormApplyGammaBeta, def_k_bnp_acc, m_bnAcc.GetIndex()) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_scale, (float)scale) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_lt, (float)lt) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_b1, (float)b1) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_b2, (float)b2) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_lr, (float)eta) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_momentum, (float)alpha) ||
|
|
!OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_optimizer, (optimization == SGD) ? 0 : 1))
|
|
return false;
|
|
ResetLastError();
|
|
if(!OpenCL.Execute(def_k_BatchNormApplyGammaBeta, 1, offset, size))
|
|
return false;
|
|
m_bnDeviceAuthoritative = true;
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| SELF-CHECK: forward. Runs the HOST transform first (on the same |
|
|
//| pre-state the device holds), dispatches the kernel, then compares |
|
|
//| output + statistics elementwise. Pass -> the device is verified |
|
|
//| and becomes authoritative. Fail -> the host result is restored to |
|
|
//| both copies and the kernels latch off. Cost: one extra input read |
|
|
//| and one output+options read, ONCE per layer per process. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::SelfCheckBnForward(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
int units = Neurons();
|
|
double inputs[];
|
|
int count = NeuronOCL.getOutputVal(inputs);
|
|
if(count <= 0)
|
|
return false;
|
|
//--- Host reference. NormalizeHost mutates the host mirrors (Output, BatchOptions, iSamplesSeen);
|
|
//--- the device still holds the untouched pre-state, which is exactly what the kernel must see.
|
|
if(!NormalizeHost(inputs, count))
|
|
return false;
|
|
int oTotal = BatchOptions.Total();
|
|
double refY[], refOpt[];
|
|
ArrayResize(refY, units);
|
|
ArrayResize(refOpt, oTotal);
|
|
for(int i = 0; i < units; i++)
|
|
refY[i] = Output.At(i);
|
|
for(int i = 0; i < oTotal; i++)
|
|
refOpt[i] = BatchOptions.At(i);
|
|
//--- Kernel on the same sample. NormalizeHost already advanced iSamplesSeen, so the ramped window
|
|
//--- it used is exactly MathMax(1, iSamplesSeen) now.
|
|
bool ok = DispatchBnForward(NeuronOCL, (double)MathMax(1, iSamplesSeen));
|
|
if(ok)
|
|
ok = Output.BufferRead() && BatchOptions.BufferRead();
|
|
double worst = 0.0;
|
|
int worstAt = -1;
|
|
if(ok)
|
|
{
|
|
for(int i = 0; i < units; i++)
|
|
{
|
|
double s = BnDiffScore(refY[i], Output.At(i));
|
|
if(s > worst) { worst = s; worstAt = i; }
|
|
}
|
|
for(int i = 0; i < oTotal; i++)
|
|
{
|
|
double s = BnDiffScore(refOpt[i], BatchOptions.At(i));
|
|
if(s > worst) { worst = s; worstAt = units + i; }
|
|
}
|
|
}
|
|
if(ok && worst <= 1.0)
|
|
{
|
|
m_bnCheckedFwd = true;
|
|
m_bnDeviceAuthoritative = true;
|
|
Print(__FUNCTION__ + StringFormat(": BatchNormForward kernel VERIFIED against the host math on "
|
|
"%d units (worst normalized diff %.2e) - this layer's forward "
|
|
"pass now runs device-side.", units, worst));
|
|
return true;
|
|
}
|
|
//--- Kernel wrong or unreachable: the host result is the answer. Put it back in both copies.
|
|
for(int i = 0; i < units; i++)
|
|
Output.Update(i, refY[i]);
|
|
for(int i = 0; i < oTotal; i++)
|
|
BatchOptions.Update(i, refOpt[i]);
|
|
if(BatchOptions.GetIndex() >= 0)
|
|
BatchOptions.BufferWrite();
|
|
m_bnDeviceAuthoritative = false;
|
|
LatchBnKernelsOff(ok ? StringFormat("BatchNormForward disagrees with the host math (worst "
|
|
"normalized diff %.2e at element %d)", worst, worstAt)
|
|
: "BatchNormForward dispatch/readback failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
return Output.BufferWrite();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| SELF-CHECK: backward. Same pattern; the reference is |
|
|
//| HiddenGradHost, which IS the host path's own loop. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::SelfCheckBnHiddenGrad(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
int units = Neurons();
|
|
SyncOptionsToHost(); // the host reference must see the statistics the kernels have been updating
|
|
double grad[];
|
|
if(getGradient(grad) < units)
|
|
return false;
|
|
double prevOut[];
|
|
if(NeuronOCL.getOutputVal(prevOut) < units)
|
|
return false;
|
|
double ig[];
|
|
if(ArrayResize(ig, units) != units)
|
|
return false;
|
|
HiddenGradHost(grad, prevOut, NeuronOCL.Activation(), ig, units);
|
|
bool ok = DispatchBnHiddenGrad(NeuronOCL);
|
|
double got[];
|
|
if(ok)
|
|
ok = (NeuronOCL.getGradient(got) >= units);
|
|
double worst = 0.0;
|
|
int worstAt = -1;
|
|
if(ok)
|
|
for(int i = 0; i < units; i++)
|
|
{
|
|
double s = BnDiffScore(ig[i], got[i]);
|
|
if(s > worst) { worst = s; worstAt = i; }
|
|
}
|
|
if(ok && worst <= 1.0)
|
|
{
|
|
m_bnCheckedGrad = true;
|
|
Print(__FUNCTION__ + StringFormat(": BatchNormHiddenGrad kernel VERIFIED against the host math "
|
|
"on %d units (worst normalized diff %.2e).", units, worst));
|
|
return true;
|
|
}
|
|
LatchBnKernelsOff(ok ? StringFormat("BatchNormHiddenGrad disagrees with the host math (worst "
|
|
"normalized diff %.2e at unit %d)", worst, worstAt)
|
|
: "BatchNormHiddenGrad dispatch/readback failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
//--- The host result is the answer either way; setGradient writes host and device copies.
|
|
return NeuronOCL.setGradient(ig);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| SELF-CHECK: gamma/beta accumulate. First use ever, so the device |
|
|
//| accumulator holds the zeros it was created with. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::SelfCheckBnAccum(void)
|
|
{
|
|
int units = Neurons();
|
|
SyncOptionsToHost(); // NX is device-fresh in kernel mode
|
|
double grad[];
|
|
if(getGradient(grad) < units)
|
|
return false;
|
|
double preAcc[];
|
|
if(m_bnAcc.GetIndex() >= 0 && !m_bnAcc.BufferRead())
|
|
{
|
|
//--- Cannot even read the accumulator: latch (which drains whatever earlier samples the kernels
|
|
//--- put there) before reporting the failure, so the batch is not silently truncated.
|
|
LatchBnKernelsOff("BatchNormAccumGammaBeta pre-check read failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
return false;
|
|
}
|
|
ArrayResize(preAcc, m_bnAcc.Total());
|
|
for(int i = 0; i < m_bnAcc.Total(); i++)
|
|
preAcc[i] = m_bnAcc.At(i);
|
|
bool ok = DispatchBnAccum();
|
|
if(ok)
|
|
ok = m_bnAcc.BufferRead();
|
|
double worst = 0.0;
|
|
int worstAt = -1;
|
|
if(ok)
|
|
for(int i = 0; i < units; i++)
|
|
{
|
|
double expG = preAcc[2 * i] + grad[i] * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX);
|
|
double expB = preAcc[2 * i + 1] + grad[i];
|
|
double s = MathMax(BnDiffScore(expG, m_bnAcc.At(2 * i)), BnDiffScore(expB, m_bnAcc.At(2 * i + 1)));
|
|
if(s > worst) { worst = s; worstAt = i; }
|
|
}
|
|
if(ok && worst <= 1.0)
|
|
{
|
|
m_bnCheckedAccum = true;
|
|
Print(__FUNCTION__ + StringFormat(": BatchNormAccumGammaBeta kernel VERIFIED against the host "
|
|
"math on %d units (worst normalized diff %.2e).", units, worst));
|
|
return true;
|
|
}
|
|
//--- Restore the truth (pre-state plus this sample's host-computed contribution) into the device
|
|
//--- accumulator BEFORE latching, so the drain inside the latch hands the host arrays exactly the
|
|
//--- right sums.
|
|
for(int i = 0; i < units; i++)
|
|
{
|
|
m_bnAcc.Update(2 * i, preAcc[2 * i] + grad[i] * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX));
|
|
m_bnAcc.Update(2 * i + 1, preAcc[2 * i + 1] + grad[i]);
|
|
}
|
|
if(m_bnAcc.GetIndex() >= 0)
|
|
m_bnAcc.BufferWrite();
|
|
LatchBnKernelsOff(ok ? StringFormat("BatchNormAccumGammaBeta disagrees with the host math (worst "
|
|
"normalized diff %.2e at unit %d)", worst, worstAt)
|
|
: "BatchNormAccumGammaBeta dispatch/readback failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| SELF-CHECK: gamma/beta apply. The host reference is StepGammaBeta |
|
|
//| itself, run on the synced host mirror - literally the code the |
|
|
//| host path executes, so the comparison cannot drift from it. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::SelfCheckBnApply(double scale, double lt)
|
|
{
|
|
int units = Neurons();
|
|
SyncOptionsToHost();
|
|
if(m_bnAcc.GetIndex() >= 0 && !m_bnAcc.BufferRead())
|
|
{
|
|
//--- Same reasoning as the accumulate pre-check: latch drains the device sums into the host
|
|
//--- arrays, so the caller's host fallback steps the REAL batch rather than an empty one.
|
|
LatchBnKernelsOff("BatchNormApplyGammaBeta pre-check read failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
return false;
|
|
}
|
|
//--- Host reference: run the real host step on the host mirror (currently the pre-state).
|
|
double accG[], accB[];
|
|
ArrayResize(accG, units);
|
|
ArrayResize(accB, units);
|
|
for(int i = 0; i < units; i++)
|
|
{
|
|
accG[i] = m_bnAcc.At(2 * i);
|
|
accB[i] = m_bnAcc.At(2 * i + 1);
|
|
if(!StepGammaBeta(i * BN_OPT_STRIDE, accG[i] * scale, accB[i] * scale, lt))
|
|
return false;
|
|
}
|
|
int oTotal = BatchOptions.Total();
|
|
double refOpt[];
|
|
ArrayResize(refOpt, oTotal);
|
|
for(int i = 0; i < oTotal; i++)
|
|
refOpt[i] = BatchOptions.At(i);
|
|
bool ok = DispatchBnApply(scale, lt);
|
|
if(ok)
|
|
ok = BatchOptions.BufferRead() && m_bnAcc.BufferRead();
|
|
double worst = 0.0;
|
|
int worstAt = -1;
|
|
if(ok)
|
|
{
|
|
for(int i = 0; i < oTotal; i++)
|
|
{
|
|
double s = BnDiffScore(refOpt[i], BatchOptions.At(i));
|
|
if(s > worst) { worst = s; worstAt = i; }
|
|
}
|
|
//--- and the kernel must have zeroed the accumulator
|
|
for(int i = 0; i < m_bnAcc.Total(); i++)
|
|
if(MathAbs(m_bnAcc.At(i)) > 1.0e-12)
|
|
{ worst = DBL_MAX; worstAt = oTotal + i; break; }
|
|
}
|
|
if(ok && worst <= 1.0)
|
|
{
|
|
m_bnCheckedApply = true;
|
|
m_bnDeviceAuthoritative = true;
|
|
Print(__FUNCTION__ + StringFormat(": BatchNormApplyGammaBeta kernel VERIFIED against the host "
|
|
"math on %d units (worst normalized diff %.2e).", units, worst));
|
|
return true;
|
|
}
|
|
//--- The host step already produced the correct post-state in the host mirror - push it, zero the
|
|
//--- accumulator everywhere, and latch.
|
|
for(int i = 0; i < oTotal; i++)
|
|
BatchOptions.Update(i, refOpt[i]);
|
|
if(BatchOptions.GetIndex() >= 0)
|
|
BatchOptions.BufferWrite();
|
|
ZeroOptimizerBuffer(m_bnAcc);
|
|
m_bnDeviceAuthoritative = false;
|
|
LatchBnKernelsOff(ok ? StringFormat("BatchNormApplyGammaBeta disagrees with the host math (worst "
|
|
"normalized diff %.2e at slot %d)", worst, worstAt)
|
|
: "BatchNormApplyGammaBeta dispatch/readback failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Backward. Called by the layer BELOW this one (inverted-call |
|
|
//| convention, same as Conv/Pool/LSTM): consumes this layer's own |
|
|
//| Gradient (dL/dy, already filled by calcHiddenGradients against |
|
|
//| the layer above) and writes dL/dx into NeuronOCL's Gradient. |
|
|
//| |
|
|
//| dL/dx = gamma/sd * dL/dy. |
|
|
//| |
|
|
//| The mean/variance are treated as CONSTANTS with respect to the |
|
|
//| current sample. That is exact for the running-statistics form |
|
|
//| this layer implements, and it is also what batch norm does at |
|
|
//| inference time. The reference's kernel additionally carries two |
|
|
//| correction terms for the current sample's own contribution to the |
|
|
//| statistics; both are O(1/batch) and vanish at the window lengths |
|
|
//| used here, and its variance term (mean*x / 2*var^1.5) does not |
|
|
//| follow from the derivative in the paper - so they are deliberately|
|
|
//| NOT copied. Numerical robustness matters far more in this engine |
|
|
//| than a correction that is already below the noise floor. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::calcInputGradients(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
if(CheckPointer(NeuronOCL) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
return false;
|
|
int units = Neurons();
|
|
int inputsCount = NeuronOCL.Neurons();
|
|
if(units <= 0 || inputsCount != units)
|
|
return false; // batch norm is elementwise - a size mismatch means the topology is wrong
|
|
//--- DEVICE PATH: the gradient, the previous output and the statistics are all already on the
|
|
//--- device; the result lands in the previous layer's device gradient where its own backward
|
|
//--- kernels read it. Zero transfers.
|
|
if(BnDeviceEligible() && NeuronOCL.getOutputIndex() >= 0 && NeuronOCL.getGradientIndex() >= 0 &&
|
|
getGradientIndex() >= 0 && EnsureBnDeviceBuffers())
|
|
{
|
|
if(!m_bnCheckedGrad)
|
|
return SelfCheckBnHiddenGrad(NeuronOCL);
|
|
if(DispatchBnHiddenGrad(NeuronOCL))
|
|
return true;
|
|
LatchBnKernelsOff("BatchNormHiddenGrad dispatch failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
}
|
|
//--- HOST PATH.
|
|
EnsureHostAuthoritative();
|
|
double grad[];
|
|
int count = getGradient(grad);
|
|
//--- Gradient is allocated with one slot more than Neurons() (see CNeuronBaseOCL::Init); only the
|
|
//--- real units carry a value.
|
|
if(count < units)
|
|
return false;
|
|
//--- Armed for the weight-update pass, which reads this same Gradient again - see the declaration.
|
|
if(ArrayCopy(m_gradCache, grad, 0, 0, count) == count)
|
|
m_gradCached = true;
|
|
//--- The previous layer's Output, from this sample's forward pass rather than a second read of the
|
|
//--- same device buffer. Falls back to reading whenever the cache is not armed or does not match.
|
|
double prevOut[];
|
|
if(m_fwdInputCached && ArraySize(m_fwdInputCache) >= inputsCount)
|
|
{
|
|
if(ArrayCopy(prevOut, m_fwdInputCache, 0, 0, inputsCount) != inputsCount)
|
|
return false;
|
|
}
|
|
else
|
|
if(NeuronOCL.getOutputVal(prevOut) < inputsCount)
|
|
return false;
|
|
double ig[];
|
|
if(ArrayResize(ig, inputsCount) != inputsCount)
|
|
return false;
|
|
HiddenGradHost(grad, prevOut, NeuronOCL.Activation(), ig, inputsCount);
|
|
return NeuronOCL.setGradient(ig);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| The elementwise backward math, factored out of calcInputGradients |
|
|
//| (2026-08-09) so the kernel self-check compares against LITERALLY |
|
|
//| the code the host path runs, not a second copy that could drift. |
|
|
//+------------------------------------------------------------------+
|
|
void CNeuronBatchNormOCL::HiddenGradHost(const double &grad[], const double &prevOut[],
|
|
ENUM_ACTIVATION act, double &ig[], int n)
|
|
{
|
|
for(int i = 0; i < n; i++)
|
|
{
|
|
int shift = i * BN_OPT_STRIDE;
|
|
double g = grad[i];
|
|
if(iBatchSize > 1)
|
|
{
|
|
double sd = MathMax(MathSqrt(BatchOptions.At(shift + BN_OPT_VAR) + BN_EPSILON), BN_MIN_STD);
|
|
g = g * BatchOptions.At(shift + BN_OPT_GAMMA) / sd;
|
|
}
|
|
//--- Then the previous layer's own activation derivative, byte-for-byte the same treatment
|
|
//--- Network.cl's CaclHiddenGradient applies - including the clamp-to-range "implied target"
|
|
//--- reformulation - so that from the previous layer's point of view a batch-norm layer is
|
|
//--- indistinguishable from any other. NONE falls through unscaled.
|
|
double out = prevOut[i];
|
|
switch(act)
|
|
{
|
|
case TANH:
|
|
g = MathMax(-1.0, MathMin(1.0, g + out)) - out;
|
|
g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, 1.0 - out * out);
|
|
break;
|
|
case SIGMOID:
|
|
g = MathMax(0.0, MathMin(1.0, g + out)) - out;
|
|
g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, out * (1.0 - out));
|
|
break;
|
|
case PRELU:
|
|
g = g * (out >= 0 ? 1.0 : 0.01);
|
|
break;
|
|
default:
|
|
break;
|
|
}
|
|
ig[i] = g;
|
|
}
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| gamma/beta update. NeuronOCL is the previous layer and is |
|
|
//| deliberately unused: batch norm consumes its output elementwise, |
|
|
//| so there is no incoming weight matrix to update (the CNet |
|
|
//| constructor gives the previous layer 0 outgoing weights when this |
|
|
//| layer follows it). |
|
|
//| |
|
|
//| NO WEIGHT DECAY on gamma/beta - unlike every other parameter in |
|
|
//| this engine, and on purpose. Decaying gamma toward zero shrinks |
|
|
//| the layer's output toward the constant beta, which is the exact |
|
|
//| pathology this layer was added to stop; excluding normalization |
|
|
//| parameters from weight decay is standard practice for the same |
|
|
//| reason. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::updateInputWeights(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
if(CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
return false;
|
|
if(iBatchSize <= 1)
|
|
return true; // normalization off - gamma/beta are not in the graph
|
|
int units = Neurons();
|
|
//--- This per-sample path stays HOST-side even in kernel mode, deliberately: it only runs when the
|
|
//--- net's train batch is 1 (online learning on a deployed model - once per confirmed bar, not per
|
|
//--- training sample), so kernelizing it buys nothing, and the host step is the reference
|
|
//--- implementation. What kernel mode DOES require is the sync bracket: pull the device statistics
|
|
//--- before stepping (NX and the running stats are device-fresh), push the stepped gamma/beta back
|
|
//--- after, so the next forward kernel sees them.
|
|
EnsureHostAuthoritative();
|
|
//--- Gradient from calcInputGradients' read this same sample, not a second device round trip.
|
|
double grad[];
|
|
int count = m_gradCached ? ArrayCopy(grad, m_gradCache) : getGradient(grad);
|
|
if(count < units || units <= 0)
|
|
return false;
|
|
//--- Same bias-corrected step size the dense Adam path computes, so gamma/beta move on the same
|
|
//--- schedule as the weights around them.
|
|
double lt = (optimization == SGD) ? 0.0 : eta * sqrt(1 - pow(b2, t)) / (1 - pow(b1, t));
|
|
for(int i = 0; i < units; i++)
|
|
{
|
|
int shift = i * BN_OPT_STRIDE;
|
|
double g = grad[i];
|
|
//--- dL/dgamma = dL/dy * nx | dL/dbeta = dL/dy
|
|
double gGamma = g * BatchOptions.At(shift + BN_OPT_NX);
|
|
double gBeta = g;
|
|
if(!StepGammaBeta(shift, gGamma, gBeta, lt))
|
|
return false;
|
|
}
|
|
if(optimization != SGD && t < INT_MAX)
|
|
t++;
|
|
//--- Keep the device copy coherent so a later kernel forward reads the stepped gamma/beta.
|
|
if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite())
|
|
return false;
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| ONE unit's gamma/beta optimizer step, factored out of |
|
|
//| updateInputWeights so the mini-batch path |
|
|
//| (ApplyAccumulatedGradients) takes the identical step on the batch |
|
|
//| mean instead of carrying a second copy of this arithmetic. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::StepGammaBeta(int shift, double gGamma, double gBeta, double lt)
|
|
{
|
|
{
|
|
double gamma = BatchOptions.At(shift + BN_OPT_GAMMA);
|
|
double beta = BatchOptions.At(shift + BN_OPT_BETA);
|
|
//--- Matches NormalizeHost's self-heal, and this is the copy that makes it STICK: the clamped
|
|
//--- write at the end of this loop persists the repaired value, whereas the forward pass only
|
|
//--- substitutes one locally. Without this a model that loaded a NaN gamma would normalize
|
|
//--- correctly but never train that unit's scale again, since NaN + anything stays NaN.
|
|
if(!MathIsValidNumber(gamma))
|
|
gamma = 1.0;
|
|
if(!MathIsValidNumber(beta))
|
|
beta = 0.0;
|
|
double dGamma = 0.0, dBeta = 0.0;
|
|
if(optimization == SGD)
|
|
{
|
|
dGamma = eta * gGamma + alpha * BatchOptions.At(shift + BN_OPT_MG);
|
|
dBeta = eta * gBeta + alpha * BatchOptions.At(shift + BN_OPT_MB);
|
|
if(!BatchOptions.Update(shift + BN_OPT_MG, dGamma) ||
|
|
!BatchOptions.Update(shift + BN_OPT_MB, dBeta))
|
|
return false;
|
|
}
|
|
else
|
|
{
|
|
//--- Second momentum is stored ALREADY square-rooted so it can be used as the denominator
|
|
//--- directly, the same convention this engine's UpdateWeightsAdam kernels use
|
|
//--- (matrix_v[wi] = sqrt(...)).
|
|
//--- This layer squares the stored value back before re-entering the recursion, which is the
|
|
//--- actual Adam second moment. RECONCILED 2026-08-09: the weight kernels used to feed the
|
|
//--- stored sqrt straight back in as `v` - mixing a standard deviation with a variance - and
|
|
//--- that recursion has a fixed point at v ~= b2 for any |g| below 1, so its denominator
|
|
//--- stopped tracking the gradient scale and Adam degraded to plain SGD. It was left alone
|
|
//--- here on the grounds that fixing it would change every existing model on all four
|
|
//--- backends; it changed them for the better, and the reason this layer kept training while
|
|
//--- the conv/LSTM stages behind it froze was precisely that gamma/beta got it right. All
|
|
//--- four tiers now match this form - see AI\Network.cl's UpdateWeightsAdam.
|
|
double mg = b1 * BatchOptions.At(shift + BN_OPT_MG) + (1 - b1) * gGamma;
|
|
double mb = b1 * BatchOptions.At(shift + BN_OPT_MB) + (1 - b1) * gBeta;
|
|
double vg = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VG), 2) + (1 - b2) * gGamma * gGamma);
|
|
double vb = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VB), 2) + (1 - b2) * gBeta * gBeta);
|
|
dGamma = lt * mg / (vg > 0 ? vg : lt * 10);
|
|
dBeta = lt * mb / (vb > 0 ? vb : lt * 10);
|
|
if(!BatchOptions.Update(shift + BN_OPT_MG, mg) ||
|
|
!BatchOptions.Update(shift + BN_OPT_MB, mb) ||
|
|
!BatchOptions.Update(shift + BN_OPT_VG, vg) ||
|
|
!BatchOptions.Update(shift + BN_OPT_VB, vb))
|
|
return false;
|
|
}
|
|
dGamma = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dGamma));
|
|
dBeta = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dBeta));
|
|
//--- `return true` and not `continue`: this is one unit's step now that the loop lives in the
|
|
//--- caller, and a non-finite delta means SKIP this unit, never fail the layer.
|
|
if(!MathIsValidNumber(dGamma) || !MathIsValidNumber(dBeta))
|
|
return true;
|
|
if(!BatchOptions.Update(shift + BN_OPT_GAMMA, MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, gamma + dGamma))) ||
|
|
!BatchOptions.Update(shift + BN_OPT_BETA, MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, beta + dBeta))))
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| MINI-BATCH (2026-08-09 audit, F4). gamma/beta are host-side |
|
|
//| parameters, so their accumulation is a plain host sum - no kernel |
|
|
//| and no extra device buffer. The per-sample normalized input |
|
|
//| (BN_OPT_NX) is overwritten by each forward pass, which is exactly |
|
|
//| why dL/dgamma has to be formed HERE, per sample, rather than |
|
|
//| reconstructed at the end of the batch. |
|
|
//| Held in plain host arrays rather than new BatchOptions slots on |
|
|
//| purpose: BN_OPT_STRIDE is baked into every persisted .nnw through |
|
|
//| getWeightsBN, so widening it would invalidate every saved model. |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::accumulateInputWeightGrads(CNeuronBaseOCL *NeuronOCL)
|
|
{
|
|
//--- The outgoing dense matrix is accumulated by the layer above, exactly as for a plain dense
|
|
//--- neuron; this adds only the normalization parameters' own gradients.
|
|
if(CheckPointer(BatchOptions) == POINTER_INVALID || iBatchSize <= 1)
|
|
return true; // normalization off - gamma/beta are not in the graph
|
|
int units = Neurons();
|
|
//--- DEVICE PATH: the gradient and NX are both device-fresh, so the accumulation happens where
|
|
//--- they already live. One dispatch, no transfers.
|
|
if(BnDeviceEligible() && getGradientIndex() >= 0 && EnsureBnDeviceBuffers())
|
|
{
|
|
if(!m_bnCheckedAccum)
|
|
return SelfCheckBnAccum();
|
|
if(DispatchBnAccum())
|
|
return true;
|
|
LatchBnKernelsOff("BatchNormAccumGammaBeta dispatch failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
//--- The latch drained whatever the kernels had accumulated this batch into m_accGamma/
|
|
//--- m_accBeta; the host code below adds THIS sample on top, so nothing is lost or doubled.
|
|
}
|
|
EnsureHostAuthoritative();
|
|
//--- Gradient from calcInputGradients' read this same sample - see the declaration comment. This is
|
|
//--- the batched twin of updateInputWeights above and takes the value from the same place.
|
|
double grad[];
|
|
int count = m_gradCached ? ArrayCopy(grad, m_gradCache) : getGradient(grad);
|
|
if(count < units || units <= 0)
|
|
return false;
|
|
if(ArraySize(m_accGamma) != units || ArraySize(m_accBeta) != units)
|
|
{
|
|
ArrayResize(m_accGamma, units);
|
|
ArrayResize(m_accBeta, units);
|
|
ArrayInitialize(m_accGamma, 0.0);
|
|
ArrayInitialize(m_accBeta, 0.0);
|
|
}
|
|
for(int i = 0; i < units; i++)
|
|
{
|
|
double g = grad[i];
|
|
m_accGamma[i] += g * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX);
|
|
m_accBeta[i] += g;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::BeginGradAccum(void)
|
|
{
|
|
bool ok = CNeuronBaseOCL::BeginGradAccum();
|
|
ArrayInitialize(m_accGamma, 0.0);
|
|
ArrayInitialize(m_accBeta, 0.0);
|
|
//--- Belt and braces for the device accumulator: the apply kernel zeroes it itself, but a batch
|
|
//--- abandoned mid-way (era boundary, restore) must not leak its partial sums into the next one.
|
|
if(CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0)
|
|
ok = ZeroOptimizerBuffer(m_bnAcc) && ok;
|
|
return ok;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::ApplyAccumulatedGradients(double scale)
|
|
{
|
|
//--- Outgoing dense matrix first, through the shared block optimizer.
|
|
bool ok = ApplyAccumToBlock(Weights, GradAccum, FirstMomentum, SecondMomentum, DeltaWeights, scale);
|
|
if(CheckPointer(BatchOptions) != POINTER_INVALID && iBatchSize > 1)
|
|
{
|
|
double lt = (optimization == SGD) ? 0.0 : eta * sqrt(1 - pow(b2, t)) / (1 - pow(b1, t));
|
|
//--- DEVICE PATH: the batch was accumulated by the kernels, so it is stepped by the kernel too.
|
|
//--- A dispatch failure latches (which drains the device sums into m_accGamma/m_accBeta) and
|
|
//--- drops to the host loop below, so the batch is stepped exactly once either way.
|
|
bool stepped = false;
|
|
if(BnDeviceEligible() && CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0)
|
|
{
|
|
if(!m_bnCheckedApply)
|
|
stepped = SelfCheckBnApply(scale, lt);
|
|
else
|
|
if(DispatchBnApply(scale, lt))
|
|
stepped = true;
|
|
else
|
|
LatchBnKernelsOff("BatchNormApplyGammaBeta dispatch failed (error " +
|
|
IntegerToString(GetLastError()) + ")");
|
|
}
|
|
if(!stepped)
|
|
{
|
|
EnsureHostAuthoritative();
|
|
int units = MathMin(Neurons(), MathMin(ArraySize(m_accGamma), ArraySize(m_accBeta)));
|
|
for(int i = 0; i < units; i++)
|
|
if(!StepGammaBeta(i * BN_OPT_STRIDE, m_accGamma[i] * scale, m_accBeta[i] * scale, lt))
|
|
{
|
|
ok = false;
|
|
break;
|
|
}
|
|
ArrayInitialize(m_accGamma, 0.0);
|
|
ArrayInitialize(m_accBeta, 0.0);
|
|
//--- Keep a created-but-idle device copy coherent with the host step.
|
|
if(BatchOptions.GetIndex() >= 0)
|
|
ok = BatchOptions.BufferWrite() && ok;
|
|
}
|
|
}
|
|
//--- One step, so t advances once - matching every other layer's batched apply.
|
|
if(optimization != SGD && t < INT_MAX)
|
|
t++;
|
|
return ok;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Outgoing dense weight matrix followed by the whole BatchOptions |
|
|
//| block, as one flat array - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CNeuronBatchNormOCL::getWeightsBN(double &values[])
|
|
{
|
|
//--- Checkpoints and the health report call this mid-training; in kernel mode the statistics live
|
|
//--- on the device, so pull them first. Read-only - the device stays authoritative.
|
|
SyncOptionsToHost();
|
|
double w[];
|
|
int wCount = getWeights(w);
|
|
if(wCount < 0)
|
|
wCount = 0;
|
|
int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
|
|
if(ArrayResize(values, wCount + oCount) != wCount + oCount)
|
|
return 0;
|
|
for(int i = 0; i < wCount; i++)
|
|
values[i] = w[i];
|
|
for(int i = 0; i < oCount; i++)
|
|
values[wCount + i] = BatchOptions.At(i);
|
|
return wCount + oCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::setWeightsBN(double &values[])
|
|
{
|
|
int total = ArraySize(values);
|
|
int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
|
|
int wCount = total - oCount;
|
|
if(wCount < 0)
|
|
return false; // snapshot predates this layer's parameters - refuse rather than half-restore
|
|
if(wCount > 0)
|
|
{
|
|
double w[];
|
|
if(ArrayResize(w, wCount) != wCount)
|
|
return false;
|
|
for(int i = 0; i < wCount; i++)
|
|
w[i] = values[i];
|
|
if(!setWeights(w))
|
|
return false;
|
|
}
|
|
for(int i = 0; i < oCount; i++)
|
|
if(!BatchOptions.Update(i, values[wCount + i]))
|
|
return false;
|
|
//--- Full host overwrite of every slot: push it so a kernel-mode net keeps computing on the
|
|
//--- RESTORED statistics rather than the diverged ones the device still holds. After this the two
|
|
//--- copies are identical, so whichever side was authoritative remains consistent.
|
|
if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite())
|
|
return false;
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::Save(const int file_handle)
|
|
{
|
|
//--- The statistics ARE the layer's inference behaviour; in kernel mode their current values live
|
|
//--- on the device, and a .nnw written from the stale host mirror would be a different model.
|
|
SyncOptionsToHost();
|
|
if(!CNeuronBaseOCL::Save(file_handle))
|
|
return false;
|
|
if(FileWriteInteger(file_handle, iBatchSize, INT_VALUE) < INT_VALUE)
|
|
return false;
|
|
if(FileWriteInteger(file_handle, iSamplesSeen, INT_VALUE) < INT_VALUE)
|
|
return false;
|
|
if(CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
return false;
|
|
return BatchOptions.Save(file_handle);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CNeuronBatchNormOCL::Load(const int file_handle)
|
|
{
|
|
if(!CNeuronBaseOCL::Load(file_handle))
|
|
return false;
|
|
iBatchSize = FileReadInteger(file_handle, INT_VALUE);
|
|
iSamplesSeen = FileReadInteger(file_handle, INT_VALUE);
|
|
if(CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
{
|
|
BatchOptions = new CBufferDouble();
|
|
if(CheckPointer(BatchOptions) == POINTER_INVALID)
|
|
return false;
|
|
}
|
|
if(!BatchOptions.Load(file_handle))
|
|
return false;
|
|
//--- Loaded state is a full host overwrite - push it if a device copy already exists (a reload into
|
|
//--- a live net), and the host is authoritative until the first kernel forward.
|
|
if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite())
|
|
return false;
|
|
m_bnDeviceAuthoritative = false;
|
|
//--- A model whose statistics block does not match its own width is not usable for inference -
|
|
//--- fail loudly here rather than index past the end on the first forward pass.
|
|
if(BatchOptions.Total() != Neurons() * BN_OPT_STRIDE)
|
|
{
|
|
Print(__FUNCTION__ + ": batch-norm parameter block is " + IntegerToString(BatchOptions.Total()) +
|
|
" values but this layer has " + IntegerToString(Neurons()) + " units (expected " +
|
|
IntegerToString(Neurons() * BN_OPT_STRIDE) + ") - file does not match the topology");
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
#endif // WARRIOR_NEURON_BATCHNORM_MQH
|
|
//+------------------------------------------------------------------+
|