//+------------------------------------------------------------------+ //| NeuronBatchNorm.mqh | //| AnimateDread | //| https://www.mql5.com | //+------------------------------------------------------------------+ //| CNeuronBatchNormOCL - batch normalization (Ioffe & Szegedy 2015). | //| Needs CNeuronBaseOCL (AI\Network.mqh) already declared; included | //| from there, not standalone. | //| | //| WHY THIS EXISTS | //| Every hidden stage in this project is unbounded (PRELU dense, | //| PRELU conv) and the only bounded stage in the whole forward path | //| was the SIGMOID classification head. That is a network with no | //| internal scale control at all, and it showed: across four | //| topologies on SP500 H1 the models peaked early then decayed | //| monotonically into all-Neutral, and the decay ordered exactly by | //| DEPTH - the shallow perceptron held ~52% balanced accuracy while | //| the deepest (conv+pool+LSTM+dense taper) sat at the 33.3% | //| one-class floor. That depth ordering is the signature of internal | //| covariate shift, which is precisely what this layer addresses. | //| | //| Two further consequences worth spelling out, because they are the | //| actual mechanism by which this is expected to help here: | //| | //| 1. It decouples weight decay from the learned function. With a | //| normalized layer downstream, scaling the weights that feed it | //| leaves the output unchanged - so WEIGHT_DECAY can no longer | //| grind the discriminative signal away, it only rescales the | //| effective learning rate (van Laarhoven 2017). The observed | //| failure was exactly a slow monotonic shrink of the per-bar | //| logit spread (0.45 -> 0.38 over ~200 eras) until the evidence | //| tilt fell under the class-prior tilt and argmax degenerated | //| to constant-Neutral. See WEIGHT_DECAY's comment in | //| AI\Network.mqh, which describes that mechanism first-hand. | //| 2. It is what would make an UNBOUNDED logit head viable. The | //| 2026-07-27 attempt to run the head at NONE blew up (IS error | //| 5.6e15) specifically because nothing upstream constrained | //| scale. Do not retry that without a normalization layer | //| immediately before the head - and then also drop | //| CLASS_LOGIT_SCALE to 1.0 and BIAS_MAGNITUDE to ~0.5. | //| | //| IMPLEMENTATION SHAPE - host-side reference, OpenCL kernels on top | //| The math is elementwise O(n) and originally ran host-side on all | //| four tiers, against the host mirrors of the device buffers, so | //| the backends could not drift. The predicted cost - "on a real GPU | //| it is a PCIe hop; if that ever matters, port these methods to | //| kernels - the math here is the specification" - came due on | //| 2026-08-09: the blocking syncs (four per BN layer per sample, | //| forward AND backward) were a large part of why an RX 580 lost to | //| a CPU thread pool, and Market builds forbid DLL imports, so the | //| OpenCL tier is what paying clients actually run. | //| The port did exactly what that sentence said: Network.cl's | //| BatchNorm* kernels are transcriptions of NormalizeHost / | //| HiddenGradHost / StepGammaBeta, the host methods REMAIN the | //| runtime for the DLL and pure-MQL5 tiers and the reference the | //| kernels must match, and each kernel is verified against its host | //| twin on first use (SelfCheckBn*) - a disagreement latches the | //| kernels off process-wide (g_bnKernelUsable) after resyncing from | //| the good copy, so a transcription bug costs a warning and some | //| speed, never a poisoned .nnw. Edit host and kernel together. | //| | //| STATISTICS: exponential moving, not a stored mini-batch. | //| Training here is pure online SGD - one weight update per sample | //| (CNet::backProp per bar), never a batched pass - so there is no | //| mini-batch to average over. iBatchSize is therefore an EMA WINDOW | //| LENGTH, not a buffer size: mean and variance are updated per | //| sample toward the last ~iBatchSize samples. Same formulation as | //| the NeuroNet_DNG reference's BatchFeedForward kernel; the book | //| (ch. 6.1.1) explicitly endorses the exponential form to avoid | //| storing per-neuron history. | //| | //| The statistics keep adapting on EVERY forward pass, including | //| out-of-sample scoring and live inference - they are never frozen | //| the way classic batch norm freezes them at inference time. That | //| is deliberate and matches the rest of this system (see | //| OnlineLearnStep: the deployed model is designed to keep tracking | //| the market). Two consequences worth knowing: OOS scoring lets the | //| statistics see OOS activations - unsupervised, no label | //| information, but not a hermetic holdout - and a restored | //| checkpoint rewinds the statistics along with gamma/beta, since | //| getWeightsBN/setWeightsBN carry both. | //+------------------------------------------------------------------+ #ifndef WARRIOR_NEURON_BATCHNORM_MQH #define WARRIOR_NEURON_BATCHNORM_MQH //--- Per-neuron slot layout inside BatchOptions. A FIXED stride of 9 on every optimizer, unlike the //--- reference's 7-or-9 split: the two unused doubles per neuron are noise next to a weight matrix, //--- and a stride that changes with a persisted enum is an indexing bug waiting for the first time //--- somebody switches TrainingOptimizer on an existing model. #define BN_OPT_STRIDE 9 #define BN_OPT_MEAN 0 // running mean #define BN_OPT_VAR 1 // running variance #define BN_OPT_NX 2 // normalized input, cached from the forward pass for the backward pass #define BN_OPT_GAMMA 3 // learned scale, init 1 #define BN_OPT_BETA 4 // learned shift, init 0 #define BN_OPT_MG 5 // gamma: Adam first momentum, or SGD previous delta #define BN_OPT_MB 6 // beta: Adam first momentum, or SGD previous delta #define BN_OPT_VG 7 // gamma: Adam second momentum (stored already square-rooted, as the #define BN_OPT_VB 8 // beta: ... UpdateWeightsAdam kernels in this engine also do) //--- Variance floor. Serves the same purpose as MIN_ACTIVATION_DERIVATIVE: a neuron whose input //--- happens to be near-constant over the EMA window has a near-zero variance, and dividing by its //--- square root turns rounding noise into an arbitrarily large activation that then propagates. //--- Applied to the standard deviation (not the variance) so it reads as "no unit is amplified by //--- more than 1e4", which is the property that actually matters. #define BN_EPSILON 1.0e-10 #define BN_MIN_STD 1.0e-4 //--- Sanity ceiling on a single input value, applied before it can touch the running statistics below. //--- Not a normalization choice - every legitimate feature in this codebase is clamped to single digits //--- long before it gets here (see BufferTempDataCompute). This is purely the bound that keeps the //--- statistics arithmetic inside double range, so a garbage value degrades one unit on one bar instead //--- of permanently latching the layer. See NormalizeHost() for the failure this exists to stop. #define BN_MAX_INPUT 1.0e6 //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ class CNeuronBatchNormOCL : public CNeuronBaseOCL { protected: int iBatchSize; // EMA window length; <=1 disables normalization entirely //--- When true the running statistics are USED but not UPDATED, i.e. classic batch-norm inference //--- semantics. Off by default (see the note on adaptation in the class header). Exists because a //--- forward pass is otherwise not a pure function of its input: two evaluations of the same bar //--- return slightly different answers because the first one moved the statistics. Anything that //--- COMPARES two forward passes has to freeze them first or it is measuring its own side effect - //--- CExpertSignalAIBase::ValidateCpuInference() is exactly that, and would have silently failed //--- its DLL-free check on nothing but the one EMA step it took itself. bool bStatsFrozen; //--- Number of samples seen. Used only to ramp the effective window up from 1 to iBatchSize over //--- the first iBatchSize samples (standard EMA bias correction). Without it the running mean //--- starts at 0 and approaches the true mean over ~iBatchSize samples, during which the layer //--- emits near-zero for everything - a cold-start dead zone indistinguishable from the very //--- collapse this layer exists to prevent. int iSamplesSeen; //--- Persisted through Save/Load like any other parameter - gamma/beta are learned, and the running //--- statistics ARE the layer's inference behaviour, so a model that loses them is not the model //--- that was trained. //--- A CBufferDouble since 2026-08-09 so the OpenCL kernels can own it device-side; on every other //--- tier it behaves exactly as the CArrayDouble it was (no device buffer is ever created). WHO IS //--- AUTHORITATIVE is tracked by m_bnDeviceAuthoritative below, and every host read/write of this //--- buffer goes through the sync helpers - an unsynced BufferRead would clobber good host state //--- with stale device state, and an unsynced host write would be silently overwritten by the next //--- kernel. The 9-slot stride is baked into persisted .nnw files and into Network.cl's BN_OPT_* //--- copies; the two must stay identical. CBufferDouble *BatchOptions; //--- Mini-batch running sums of dL/dgamma and dL/dbeta, one entry per unit. Host-only and NOT //--- persisted: transient within a batch, and every save point flushes first. See //--- accumulateInputWeightGrads for why they are not extra BatchOptions slots. double m_accGamma[]; double m_accBeta[]; //--- PER-SAMPLE TRANSFER CACHES. This layer's math is host-side while its neighbours are device //--- resident, so every value it touches crosses the bus - and each crossing is a BLOCKING sync, //--- which is what makes it expensive rather than the bytes. Per sample it used to do four reads: //--- feedForward -> previous layer's Output //--- calcInputGradients -> own Gradient, and the previous layer's Output AGAIN //--- update/accumulate -> own Gradient AGAIN //--- The two repeats are exact duplicates. Nothing writes the previous layer's Output between the //--- forward pass and the backward pass, and nothing writes this layer's Gradient between the //--- gradient pass and the weight-update pass - CNet::backPropOCL runs those as two separate //--- top-to-bottom loops, and only the first one writes gradients. //--- //--- Caching them is therefore bit-exact, not an approximation: the same values, read once. //--- Each cache is armed by its producer and DISARMED by feedForward, so a consumer that runs //--- without its producer having run this sample falls back to reading the buffer rather than //--- silently using the previous sample's data. That matters concretely: a batch-norm at layer 1 //--- never gets calcInputGradients called at all (backPropOCL's loop stops at layerNum > 0 - the //--- same asymmetry documented there for a layer-1 LSTM), so its gradient cache is never armed. double m_fwdInputCache[]; bool m_fwdInputCached; double m_gradCache[]; bool m_gradCached; //--- DEVICE PATH (OpenCL only, 2026-08-09). m_bnAcc holds the mini-batch gamma/beta gradient sums //--- on the device (2 floats per unit: gamma then beta), the kernel twin of m_accGamma/m_accBeta. //--- m_bnDeviceAuthoritative says the DEVICE copy of BatchOptions is the truth (kernels have //--- written it since the last host sync); the m_bnChecked* flags latch each kernel's one-time //--- self-check against its host twin. The checks are the whole safety story for shipping kernels //--- that could not be built on the dev machine: a transcription or dispatch-binding error is //--- caught on its first use, the layer resyncs from the good copy, latches the kernels off //--- process-wide, and training continues host-side - a warning and some speed, never a poisoned //--- .nnw. CBufferDouble *m_bnAcc; bool m_bnDeviceAuthoritative; bool m_bnCheckedFwd; bool m_bnCheckedGrad; bool m_bnCheckedAccum; bool m_bnCheckedApply; //--- eligibility + buffer management for the kernel path bool BnDeviceEligible(void); bool EnsureBnDeviceBuffers(void); //--- read-only pull of the device statistics into the host mirror (checkpoints/saves mid-training); //--- device stays authoritative void SyncOptionsToHost(void); //--- full handover to the host path: pull statistics, drain the device accumulator into //--- m_accGamma/m_accBeta so a mid-batch handover loses nothing, clear the flag void EnsureHostAuthoritative(void); //--- one-way process-wide latch + this layer's handover, with the reason printed once void LatchBnKernelsOff(const string reason); //--- kernel dispatches (return false on any SetArgument/Execute failure, no logging - the caller //--- decides between latching and falling back) bool DispatchBnForward(CNeuronBaseOCL *NeuronOCL, double w); bool DispatchBnHiddenGrad(CNeuronBaseOCL *NeuronOCL); bool DispatchBnAccum(void); bool DispatchBnApply(double scale, double lt); //--- one-time kernel-vs-host comparisons; each returns the OPERATION's result (true = the work got //--- done correctly, by whichever path survived), never "the kernel matched" bool SelfCheckBnForward(CNeuronBaseOCL *NeuronOCL); bool SelfCheckBnHiddenGrad(CNeuronBaseOCL *NeuronOCL); bool SelfCheckBnAccum(void); bool SelfCheckBnApply(double scale, double lt); //--- normalized disagreement: |got-ref| / (1e-3 * max(1,|ref|)), <=1 passes. DBL_MAX when exactly //--- one side is non-finite. The 1e-3 relative band is ~4 decades above fp32-vs-fp64 noise and ~3 //--- below any real transcription error (wrong slot, wrong sign, wrong buffer), so it cannot //--- confuse the two. double BnDiffScore(double ref, double got); //--- host twin of the backward elementwise math, factored out of calcInputGradients so the //--- self-check compares against literally the same code the host path runs void HiddenGradHost(const double &grad[], const double &prevOut[], ENUM_ACTIVATION act, double &ig[], int n); //--- virtual bool feedForward(CNeuronBaseOCL *NeuronOCL); virtual bool feedForwardCPU(CNeuronBaseOCL *NeuronOCL); virtual bool updateInputWeights(CNeuronBaseOCL *NeuronOCL); virtual bool accumulateInputWeightGrads(CNeuronBaseOCL *NeuronOCL); //--- one unit's gamma/beta step, shared by the per-sample and per-batch paths bool StepGammaBeta(int shift, double gGamma, double gBeta, double lt); public: virtual bool BeginGradAccum(void); virtual bool ApplyAccumulatedGradients(double scale); protected: //--- shared by feedForward/feedForwardCPU: the whole forward transform for one already-read input //--- vector, writing straight into the host mirror of Output. bool NormalizeHost(const double &inputs[], int count); //--- ensures BatchOptions exists and is sized/seeded for `neurons` units bool InitOptions(int neurons); public: CNeuronBatchNormOCL(void); ~CNeuronBatchNormOCL(void); //--- virtual bool Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type); virtual bool Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type); virtual bool calcInputGradients(CNeuronBaseOCL *NeuronOCL); //--- see bStatsFrozen. Deliberately NOT persisted: it is a transient evaluation mode, not model state. void SetStatsFrozen(bool v) { bStatsFrozen = v; } //--- Checkpoint/blend support. CNet::CaptureWeights/RestoreWeights/BlendWeightsFrom snapshot one //--- flat array per neuron object, so gamma/beta/mean/variance are appended AFTER the outgoing //--- dense weight matrix and split back out on restore. Without this the plateau ladder's //--- "restore best checkpoint" would put the dense weights back while leaving this layer's //--- parameters at whatever the diverged era left behind - a silently mismatched pair. virtual int getWeightsBN(double &values[]); virtual bool setWeightsBN(double &values[]); //--- How many of the trailing entries in getWeightsBN's array are BatchOptions rather than the //--- outgoing dense matrix. Published so the health report can split that packed block into its //--- four very different parts (weights / gamma-beta / running statistics / Adam moments) instead //--- of quoting one norm over all of them, which cannot say which is moving. int BatchOptionsTotal(void) const { return (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total(); } //--- Zero ONLY the gamma/beta moment slots (BN_OPT_MG/MB/VG/VB) plus the base class's buffers for //--- the outgoing dense matrix. The running mean/variance and the learned gamma/beta are MODEL //--- state, not optimizer state - they are exactly what getWeightsBN checkpoints and what a restore //--- puts back, so wiping them here would undo the restore this reset exists to complete. See //--- CNet::ResetOptimizerState. virtual bool ResetOptimizerState(void) { bool ok = CNeuronBaseOCL::ResetOptimizerState(); if(CheckPointer(BatchOptions) != POINTER_INVALID) { //--- Kernel-mode discipline: this zeroes SOME slots of a block whose truth may live on the //--- device, so pull first (or the untouched slots would be written back stale), zero, push. SyncOptionsToHost(); int totalSlots = BatchOptions.Total(); for(int shift = 0; shift + BN_OPT_VB < totalSlots; shift += BN_OPT_STRIDE) { ok = BatchOptions.Update(shift + BN_OPT_MG, 0.0) && ok; ok = BatchOptions.Update(shift + BN_OPT_MB, 0.0) && ok; ok = BatchOptions.Update(shift + BN_OPT_VG, 0.0) && ok; ok = BatchOptions.Update(shift + BN_OPT_VB, 0.0) && ok; } if(BatchOptions.GetIndex() >= 0) ok = BatchOptions.BufferWrite() && ok; } return ok; } //--- virtual bool Save(int const file_handle); virtual bool Load(int const file_handle); //--- virtual int Type(void) const { return defNeuronBatchNormOCL; } }; //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ CNeuronBatchNormOCL::CNeuronBatchNormOCL(void) : iBatchSize(1), bStatsFrozen(false), iSamplesSeen(0), m_fwdInputCached(false), m_gradCached(false), m_bnDeviceAuthoritative(false), m_bnCheckedFwd(false), m_bnCheckedGrad(false), m_bnCheckedAccum(false), m_bnCheckedApply(false) { BatchOptions = NULL; m_bnAcc = NULL; } //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ CNeuronBatchNormOCL::~CNeuronBatchNormOCL(void) { if(CheckPointer(BatchOptions) != POINTER_INVALID) delete BatchOptions; if(CheckPointer(m_bnAcc) != POINTER_INVALID) delete m_bnAcc; } //+------------------------------------------------------------------+ //| gamma=1 / beta=0 / zeroed statistics for every unit. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::InitOptions(int neurons) { if(neurons <= 0) return false; if(CheckPointer(BatchOptions) == POINTER_INVALID) { BatchOptions = new CBufferDouble(); if(CheckPointer(BatchOptions) == POINTER_INVALID) return false; } BatchOptions.Clear(); if(!BatchOptions.Reserve(neurons * BN_OPT_STRIDE)) return false; for(int n = 0; n < neurons; n++) for(int s = 0; s < BN_OPT_STRIDE; s++) if(!BatchOptions.Add(s == BN_OPT_GAMMA ? 1.0 : 0.0)) return false; iSamplesSeen = 0; //--- Full host overwrite: if a device copy exists, push it so the two cannot disagree. The host is //--- authoritative after a re-init by definition. if(BatchOptions.GetIndex() >= 0) { if(!BatchOptions.BufferWrite()) return false; m_bnDeviceAuthoritative = false; } return true; } //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type) { if(!CNeuronBaseOCL::Init(numOutputs, myIndex, open_cl, numNeurons, optimization_type)) return false; //--- Identity forward transform. The activation belongs to the layer AFTER this one; normalizing //--- and then squashing in the same step would defeat the point (Ioffe & Szegedy place the //--- normalization immediately BEFORE the non-linearity, not around it). activation = NONE; iBatchSize = (int)batchSize; return InitOptions((int)numNeurons); } //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type) { if(!CNeuronBaseOCL::Init(numOutputs, myIndex, direct_ml, numNeurons, optimization_type)) return false; activation = NONE; iBatchSize = (int)batchSize; return InitOptions((int)numNeurons); } //+------------------------------------------------------------------+ //| The forward transform, host-side. Mirrors the NeuroNet_DNG | //| BatchFeedForward kernel, plus the bias-corrected warm-up window. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::NormalizeHost(const double &inputs[], int count) { if(CheckPointer(Output) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID) return false; if(count > Output.Total()) count = Output.Total(); if(BatchOptions.Total() < count * BN_OPT_STRIDE) return false; //--- A window of 1 makes mean==x and variance==0 for every sample, i.e. a constant-zero output. //--- Treat it as "normalization off" and pass the input through untouched rather than emit zeros. if(iBatchSize <= 1) { for(int i = 0; i < count; i++) if(!Output.Update(i, inputs[i])) return false; return true; } if(!bStatsFrozen && iSamplesSeen < iBatchSize) iSamplesSeen++; //--- Effective window: ramps 1,2,3... up to iBatchSize, so the first samples produce an honest //--- running mean instead of one biased toward the zero initializer. double w = (double)MathMax(1, iSamplesSeen); for(int i = 0; i < count; i++) { int shift = i * BN_OPT_STRIDE; double x = inputs[i]; //--- The running mean/variance below are PERSISTENT: they live in BatchOptions, are carried into //--- the .nnw by getWeightsBN, and every later sample normalizes against them. That makes them a //--- LATCH. One non-finite or astronomically large x poisons them permanently, and from then on //--- this layer emits NaN on every bar for the rest of the run - the arithmetic gets there in //--- three steps: (x-mean)^2 overflows variance to +inf, then mean*(w-1) overflows mean to +inf, //--- then delta/sd is inf/inf = NaN. Note it does NOT need x to be non-finite to start: a merely //--- huge finite x (an unguarded EMPTY_VALUE feature divided by ATR is ~1e307) passes every //--- upstream check and still overflows the square. //--- 2026-08-02: this presented as 13,776 identical "BufferWrite failed for buffer 3" lines and //--- nothing else. Buffer 3 is the first batch-norm layer's Output - the one sitting on the raw //--- input vector - and the CPU DLL's isfinite() boundary check was correctly refusing to store //--- the NaN, so the layer's device-side output silently froze at its last good value while //--- training carried on against it for a full era. //--- A value this large is meaningless whatever produced it, so clamp rather than propagate. if(!MathIsValidNumber(x)) x = 0.0; x = MathMax(-BN_MAX_INPUT, MathMin(BN_MAX_INPUT, x)); double mean = BatchOptions.At(shift + BN_OPT_MEAN); double variance = BatchOptions.At(shift + BN_OPT_VAR); //--- Self-heal statistics that were already poisoned before this guard existed. Without it an //--- affected .nnw stays dead across restarts, because Load faithfully restores the NaN. if(!MathIsValidNumber(mean)) mean = x; if(!MathIsValidNumber(variance) || variance < 0.0) variance = 0.0; if(!bStatsFrozen) { mean = (mean * (w - 1.0) + x) / w; variance = (variance * (w - 1.0) + (x - mean) * (x - mean)) / w; } double delta = x - mean; double sd = MathMax(MathSqrt(variance + BN_EPSILON), BN_MIN_STD); double nx = delta / sd; //--- gamma/beta cannot go non-finite going forward (updateInputWeights validates and clamps every //--- step), but a model SAVED before that guard existed can carry NaN in here through Load. Same //--- self-heal as the statistics above: fall back to the identity transform for that unit. double gamma = BatchOptions.At(shift + BN_OPT_GAMMA); double beta = BatchOptions.At(shift + BN_OPT_BETA); if(!MathIsValidNumber(gamma)) gamma = 1.0; if(!MathIsValidNumber(beta)) beta = 0.0; double y = gamma * nx + beta; //--- nx is still cached when frozen: it costs nothing and keeps the buffer consistent with the //--- output just produced. There is no backward pass while frozen, so nothing reads it. if(!bStatsFrozen && (!BatchOptions.Update(shift + BN_OPT_MEAN, mean) || !BatchOptions.Update(shift + BN_OPT_VAR, variance))) return false; if(!BatchOptions.Update(shift + BN_OPT_NX, nx)) return false; if(!Output.Update(i, y)) return false; } return true; } //+------------------------------------------------------------------+ //| Forward. On OpenCL this is a KERNEL since 2026-08-09 (verified | //| against NormalizeHost on first use, see SelfCheckBnForward); on | //| every other backend it pulls the previous layer's output to the | //| host, runs the transform, and pushes its output back. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::feedForward(CNeuronBaseOCL *NeuronOCL) { if(CheckPointer(NeuronOCL) == POINTER_INVALID) return false; //--- A new sample starts here, so last sample's gradient is no longer valid to reuse. Disarmed //--- BEFORE anything can fail below, so an aborted forward pass cannot leave a consumer holding a //--- cache that looks current - see the declaration comment. m_gradCached = false; m_fwdInputCached = false; //--- DEVICE PATH: nothing crosses the bus at all. The self-check runs the first time through and //--- proves the kernel against NormalizeHost before the device is trusted with the persisted //--- statistics; after that it is one dispatch per sample. if(BnDeviceEligible() && NeuronOCL.getOutputIndex() >= 0 && getOutputIndex() >= 0 && EnsureBnDeviceBuffers()) { if(!m_bnCheckedFwd) return SelfCheckBnForward(NeuronOCL); //--- Ramp bookkeeping stays host-side with iSamplesSeen; committed only on success so a failed //--- dispatch that falls through to NormalizeHost (which increments itself) cannot double-count. int seen = iSamplesSeen; if(!bStatsFrozen && seen < iBatchSize) seen++; if(DispatchBnForward(NeuronOCL, (double)MathMax(1, seen))) { iSamplesSeen = seen; return true; } LatchBnKernelsOff("BatchNormForward dispatch failed (error " + IntegerToString(GetLastError()) + ")"); } //--- HOST PATH (every non-OpenCL tier, and OpenCL after a latch). If the device had been //--- authoritative, pull its state first - NormalizeHost below must advance the REAL statistics, //--- not a stale mirror. EnsureHostAuthoritative(); double inputs[]; int count = NeuronOCL.getOutputVal(inputs); if(count <= 0) return false; if(!NormalizeHost(inputs, count)) return false; //--- Armed for calcInputGradients, which needs exactly these values and would otherwise read the //--- same buffer back a second time this sample. if(ArrayCopy(m_fwdInputCache, inputs, 0, 0, count) == count) m_fwdInputCached = true; //--- Unlike the dense/conv kernels this layer's Output does NOT stay device-resident by itself - //--- it was just written host-side, so it has to be pushed before the next layer's kernel reads //--- it through getOutputIndex(). return Output.BufferWrite(); } //+------------------------------------------------------------------+ //| Pure-MQL5 inference path (no backend at all): the previous | //| layer's values live only in its host mirror, so read them there. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::feedForwardCPU(CNeuronBaseOCL *NeuronOCL) { if(CheckPointer(NeuronOCL) == POINTER_INVALID) return false; int count = NeuronOCL.Neurons(); if(count <= 0) return false; double inputs[]; if(ArrayResize(inputs, count) != count) return false; for(int i = 0; i < count; i++) inputs[i] = NeuronOCL.OutputHost(i); //--- No BufferWrite: there is no device buffer on this path, and Output's host mirror is what //--- the next layer's feedForwardCPU (and GetOutputsCPU) read. return NormalizeHost(inputs, count); } //+------------------------------------------------------------------+ //| DEVICE PATH plumbing - see the m_bnAcc declaration comment for | //| the design. Everything below is OpenCL-only. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::BnDeviceEligible(void) { return g_bnKernelUsable && iBatchSize > 1 && CheckPointer(OpenCL) != POINTER_INVALID && CheckPointer(BatchOptions) != POINTER_INVALID; } //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::EnsureBnDeviceBuffers(void) { //--- BufferCreate pushes the current host contents, so a freshly-created device copy is exactly //--- the state the host was authoritative over - the invariant every self-check relies on. if(BatchOptions.GetIndex() < 0 && !BatchOptions.BufferCreate(OpenCL)) return false; if(CheckPointer(m_bnAcc) == POINTER_INVALID) { m_bnAcc = new CBufferDouble(); if(CheckPointer(m_bnAcc) == POINTER_INVALID) return false; if(!m_bnAcc.BufferInit(Neurons() * 2, 0.0)) return false; } if(m_bnAcc.GetIndex() < 0 && !m_bnAcc.BufferCreate(OpenCL)) return false; return true; } //+------------------------------------------------------------------+ void CNeuronBatchNormOCL::SyncOptionsToHost(void) { //--- Read-only pull for checkpoints/saves taken mid-training; the device REMAINS authoritative. //--- Guarded by the flag, because an unguarded BufferRead on a host-authoritative layer would //--- clobber good host state with whatever stale bytes the device still holds. if(m_bnDeviceAuthoritative && CheckPointer(BatchOptions) != POINTER_INVALID && BatchOptions.GetIndex() >= 0) BatchOptions.BufferRead(); } //+------------------------------------------------------------------+ void CNeuronBatchNormOCL::EnsureHostAuthoritative(void) { if(!m_bnDeviceAuthoritative) return; if(CheckPointer(BatchOptions) != POINTER_INVALID && BatchOptions.GetIndex() >= 0) BatchOptions.BufferRead(); //--- Drain the device-side batch accumulator into the host arrays, so a handover in the MIDDLE of //--- a batch keeps the samples the kernels already accumulated - without this, latching off after //--- sample 3 of 8 would silently drop three samples' gradients from the batch. if(CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0 && m_bnAcc.BufferRead()) { int units = Neurons(); if(ArraySize(m_accGamma) != units || ArraySize(m_accBeta) != units) { ArrayResize(m_accGamma, units); ArrayResize(m_accBeta, units); ArrayInitialize(m_accGamma, 0.0); ArrayInitialize(m_accBeta, 0.0); } for(int i = 0; i < units && 2 * i + 1 < m_bnAcc.Total(); i++) { m_accGamma[i] += m_bnAcc.At(2 * i); m_accBeta[i] += m_bnAcc.At(2 * i + 1); } ZeroOptimizerBuffer(m_bnAcc); } m_bnDeviceAuthoritative = false; } //+------------------------------------------------------------------+ void CNeuronBatchNormOCL::LatchBnKernelsOff(const string reason) { if(g_bnKernelUsable) { g_bnKernelUsable = false; Print(__FUNCTION__ + ": BATCH-NORM KERNELS DISABLED for the rest of this run - " + reason + ". Every batch-norm layer falls back to the host implementation: results stay correct " "(the host path is the reference the kernels were transcribed from), each layer just pays " "its device round-trips again. State was resynced from the good copy before anything " "could persist."); } EnsureHostAuthoritative(); } //+------------------------------------------------------------------+ double CNeuronBatchNormOCL::BnDiffScore(double ref, double got) { bool fRef = MathIsValidNumber(ref), fGot = MathIsValidNumber(got); if(!fRef || !fGot) return (fRef == fGot) ? 0.0 : DBL_MAX; // both poisoned the same way is agreement return MathAbs(got - ref) / (1.0e-3 * MathMax(1.0, MathAbs(ref))); } //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::DispatchBnForward(CNeuronBaseOCL *NeuronOCL, double w) { uint offset[1] = {0}; uint size[1]; size[0] = (uint)Neurons(); if(!OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_matrix_i, NeuronOCL.getOutputIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_matrix_o, getOutputIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_options, BatchOptions.GetIndex()) || !OpenCL.SetArgument(def_k_BatchNormForward, def_k_bnf_w, (float)w) || !OpenCL.SetArgument(def_k_BatchNormForward, def_k_bnf_frozen, bStatsFrozen ? 1 : 0)) return false; ResetLastError(); if(!OpenCL.Execute(def_k_BatchNormForward, 1, offset, size)) return false; //--- The kernel just wrote the running statistics on the device: it is now the authority. m_bnDeviceAuthoritative = true; return true; } //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::DispatchBnHiddenGrad(CNeuronBaseOCL *NeuronOCL) { uint offset[1] = {0}; uint size[1]; size[0] = (uint)Neurons(); if(!OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_matrix_g, getGradientIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_prev_o, NeuronOCL.getOutputIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_prev_g, NeuronOCL.getGradientIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_options, BatchOptions.GetIndex()) || !OpenCL.SetArgument(def_k_BatchNormHiddenGrad, def_k_bnh_activation, NativeActivationCode(NeuronOCL.Activation()))) return false; ResetLastError(); return OpenCL.Execute(def_k_BatchNormHiddenGrad, 1, offset, size); } //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::DispatchBnAccum(void) { uint offset[1] = {0}; uint size[1]; size[0] = (uint)Neurons(); if(!OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_matrix_g, getGradientIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_options, BatchOptions.GetIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_acc, m_bnAcc.GetIndex())) return false; ResetLastError(); return OpenCL.Execute(def_k_BatchNormAccumGammaBeta, 1, offset, size); } //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::DispatchBnApply(double scale, double lt) { uint offset[1] = {0}; uint size[1]; size[0] = (uint)Neurons(); if(!OpenCL.SetArgumentBuffer(def_k_BatchNormApplyGammaBeta, def_k_bnp_options, BatchOptions.GetIndex()) || !OpenCL.SetArgumentBuffer(def_k_BatchNormApplyGammaBeta, def_k_bnp_acc, m_bnAcc.GetIndex()) || !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_scale, (float)scale) || !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_lt, (float)lt) || !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_b1, (float)b1) || !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_b2, (float)b2) || !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_lr, (float)eta) || !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_momentum, (float)alpha) || !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_optimizer, (optimization == SGD) ? 0 : 1)) return false; ResetLastError(); if(!OpenCL.Execute(def_k_BatchNormApplyGammaBeta, 1, offset, size)) return false; m_bnDeviceAuthoritative = true; return true; } //+------------------------------------------------------------------+ //| SELF-CHECK: forward. Runs the HOST transform first (on the same | //| pre-state the device holds), dispatches the kernel, then compares | //| output + statistics elementwise. Pass -> the device is verified | //| and becomes authoritative. Fail -> the host result is restored to | //| both copies and the kernels latch off. Cost: one extra input read | //| and one output+options read, ONCE per layer per process. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::SelfCheckBnForward(CNeuronBaseOCL *NeuronOCL) { int units = Neurons(); double inputs[]; int count = NeuronOCL.getOutputVal(inputs); if(count <= 0) return false; //--- Host reference. NormalizeHost mutates the host mirrors (Output, BatchOptions, iSamplesSeen); //--- the device still holds the untouched pre-state, which is exactly what the kernel must see. if(!NormalizeHost(inputs, count)) return false; int oTotal = BatchOptions.Total(); double refY[], refOpt[]; ArrayResize(refY, units); ArrayResize(refOpt, oTotal); for(int i = 0; i < units; i++) refY[i] = Output.At(i); for(int i = 0; i < oTotal; i++) refOpt[i] = BatchOptions.At(i); //--- Kernel on the same sample. NormalizeHost already advanced iSamplesSeen, so the ramped window //--- it used is exactly MathMax(1, iSamplesSeen) now. bool ok = DispatchBnForward(NeuronOCL, (double)MathMax(1, iSamplesSeen)); if(ok) ok = Output.BufferRead() && BatchOptions.BufferRead(); double worst = 0.0; int worstAt = -1; if(ok) { for(int i = 0; i < units; i++) { double s = BnDiffScore(refY[i], Output.At(i)); if(s > worst) { worst = s; worstAt = i; } } for(int i = 0; i < oTotal; i++) { double s = BnDiffScore(refOpt[i], BatchOptions.At(i)); if(s > worst) { worst = s; worstAt = units + i; } } } if(ok && worst <= 1.0) { m_bnCheckedFwd = true; m_bnDeviceAuthoritative = true; Print(__FUNCTION__ + StringFormat(": BatchNormForward kernel VERIFIED against the host math on " "%d units (worst normalized diff %.2e) - this layer's forward " "pass now runs device-side.", units, worst)); return true; } //--- Kernel wrong or unreachable: the host result is the answer. Put it back in both copies. for(int i = 0; i < units; i++) Output.Update(i, refY[i]); for(int i = 0; i < oTotal; i++) BatchOptions.Update(i, refOpt[i]); if(BatchOptions.GetIndex() >= 0) BatchOptions.BufferWrite(); m_bnDeviceAuthoritative = false; LatchBnKernelsOff(ok ? StringFormat("BatchNormForward disagrees with the host math (worst " "normalized diff %.2e at element %d)", worst, worstAt) : "BatchNormForward dispatch/readback failed (error " + IntegerToString(GetLastError()) + ")"); return Output.BufferWrite(); } //+------------------------------------------------------------------+ //| SELF-CHECK: backward. Same pattern; the reference is | //| HiddenGradHost, which IS the host path's own loop. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::SelfCheckBnHiddenGrad(CNeuronBaseOCL *NeuronOCL) { int units = Neurons(); SyncOptionsToHost(); // the host reference must see the statistics the kernels have been updating double grad[]; if(getGradient(grad) < units) return false; double prevOut[]; if(NeuronOCL.getOutputVal(prevOut) < units) return false; double ig[]; if(ArrayResize(ig, units) != units) return false; HiddenGradHost(grad, prevOut, NeuronOCL.Activation(), ig, units); bool ok = DispatchBnHiddenGrad(NeuronOCL); double got[]; if(ok) ok = (NeuronOCL.getGradient(got) >= units); double worst = 0.0; int worstAt = -1; if(ok) for(int i = 0; i < units; i++) { double s = BnDiffScore(ig[i], got[i]); if(s > worst) { worst = s; worstAt = i; } } if(ok && worst <= 1.0) { m_bnCheckedGrad = true; Print(__FUNCTION__ + StringFormat(": BatchNormHiddenGrad kernel VERIFIED against the host math " "on %d units (worst normalized diff %.2e).", units, worst)); return true; } LatchBnKernelsOff(ok ? StringFormat("BatchNormHiddenGrad disagrees with the host math (worst " "normalized diff %.2e at unit %d)", worst, worstAt) : "BatchNormHiddenGrad dispatch/readback failed (error " + IntegerToString(GetLastError()) + ")"); //--- The host result is the answer either way; setGradient writes host and device copies. return NeuronOCL.setGradient(ig); } //+------------------------------------------------------------------+ //| SELF-CHECK: gamma/beta accumulate. First use ever, so the device | //| accumulator holds the zeros it was created with. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::SelfCheckBnAccum(void) { int units = Neurons(); SyncOptionsToHost(); // NX is device-fresh in kernel mode double grad[]; if(getGradient(grad) < units) return false; double preAcc[]; if(m_bnAcc.GetIndex() >= 0 && !m_bnAcc.BufferRead()) { //--- Cannot even read the accumulator: latch (which drains whatever earlier samples the kernels //--- put there) before reporting the failure, so the batch is not silently truncated. LatchBnKernelsOff("BatchNormAccumGammaBeta pre-check read failed (error " + IntegerToString(GetLastError()) + ")"); return false; } ArrayResize(preAcc, m_bnAcc.Total()); for(int i = 0; i < m_bnAcc.Total(); i++) preAcc[i] = m_bnAcc.At(i); bool ok = DispatchBnAccum(); if(ok) ok = m_bnAcc.BufferRead(); double worst = 0.0; int worstAt = -1; if(ok) for(int i = 0; i < units; i++) { double expG = preAcc[2 * i] + grad[i] * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX); double expB = preAcc[2 * i + 1] + grad[i]; double s = MathMax(BnDiffScore(expG, m_bnAcc.At(2 * i)), BnDiffScore(expB, m_bnAcc.At(2 * i + 1))); if(s > worst) { worst = s; worstAt = i; } } if(ok && worst <= 1.0) { m_bnCheckedAccum = true; Print(__FUNCTION__ + StringFormat(": BatchNormAccumGammaBeta kernel VERIFIED against the host " "math on %d units (worst normalized diff %.2e).", units, worst)); return true; } //--- Restore the truth (pre-state plus this sample's host-computed contribution) into the device //--- accumulator BEFORE latching, so the drain inside the latch hands the host arrays exactly the //--- right sums. for(int i = 0; i < units; i++) { m_bnAcc.Update(2 * i, preAcc[2 * i] + grad[i] * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX)); m_bnAcc.Update(2 * i + 1, preAcc[2 * i + 1] + grad[i]); } if(m_bnAcc.GetIndex() >= 0) m_bnAcc.BufferWrite(); LatchBnKernelsOff(ok ? StringFormat("BatchNormAccumGammaBeta disagrees with the host math (worst " "normalized diff %.2e at unit %d)", worst, worstAt) : "BatchNormAccumGammaBeta dispatch/readback failed (error " + IntegerToString(GetLastError()) + ")"); return true; } //+------------------------------------------------------------------+ //| SELF-CHECK: gamma/beta apply. The host reference is StepGammaBeta | //| itself, run on the synced host mirror - literally the code the | //| host path executes, so the comparison cannot drift from it. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::SelfCheckBnApply(double scale, double lt) { int units = Neurons(); SyncOptionsToHost(); if(m_bnAcc.GetIndex() >= 0 && !m_bnAcc.BufferRead()) { //--- Same reasoning as the accumulate pre-check: latch drains the device sums into the host //--- arrays, so the caller's host fallback steps the REAL batch rather than an empty one. LatchBnKernelsOff("BatchNormApplyGammaBeta pre-check read failed (error " + IntegerToString(GetLastError()) + ")"); return false; } //--- Host reference: run the real host step on the host mirror (currently the pre-state). double accG[], accB[]; ArrayResize(accG, units); ArrayResize(accB, units); for(int i = 0; i < units; i++) { accG[i] = m_bnAcc.At(2 * i); accB[i] = m_bnAcc.At(2 * i + 1); if(!StepGammaBeta(i * BN_OPT_STRIDE, accG[i] * scale, accB[i] * scale, lt)) return false; } int oTotal = BatchOptions.Total(); double refOpt[]; ArrayResize(refOpt, oTotal); for(int i = 0; i < oTotal; i++) refOpt[i] = BatchOptions.At(i); bool ok = DispatchBnApply(scale, lt); if(ok) ok = BatchOptions.BufferRead() && m_bnAcc.BufferRead(); double worst = 0.0; int worstAt = -1; if(ok) { for(int i = 0; i < oTotal; i++) { double s = BnDiffScore(refOpt[i], BatchOptions.At(i)); if(s > worst) { worst = s; worstAt = i; } } //--- and the kernel must have zeroed the accumulator for(int i = 0; i < m_bnAcc.Total(); i++) if(MathAbs(m_bnAcc.At(i)) > 1.0e-12) { worst = DBL_MAX; worstAt = oTotal + i; break; } } if(ok && worst <= 1.0) { m_bnCheckedApply = true; m_bnDeviceAuthoritative = true; Print(__FUNCTION__ + StringFormat(": BatchNormApplyGammaBeta kernel VERIFIED against the host " "math on %d units (worst normalized diff %.2e).", units, worst)); return true; } //--- The host step already produced the correct post-state in the host mirror - push it, zero the //--- accumulator everywhere, and latch. for(int i = 0; i < oTotal; i++) BatchOptions.Update(i, refOpt[i]); if(BatchOptions.GetIndex() >= 0) BatchOptions.BufferWrite(); ZeroOptimizerBuffer(m_bnAcc); m_bnDeviceAuthoritative = false; LatchBnKernelsOff(ok ? StringFormat("BatchNormApplyGammaBeta disagrees with the host math (worst " "normalized diff %.2e at slot %d)", worst, worstAt) : "BatchNormApplyGammaBeta dispatch/readback failed (error " + IntegerToString(GetLastError()) + ")"); return true; } //+------------------------------------------------------------------+ //| Backward. Called by the layer BELOW this one (inverted-call | //| convention, same as Conv/Pool/LSTM): consumes this layer's own | //| Gradient (dL/dy, already filled by calcHiddenGradients against | //| the layer above) and writes dL/dx into NeuronOCL's Gradient. | //| | //| dL/dx = gamma/sd * dL/dy. | //| | //| The mean/variance are treated as CONSTANTS with respect to the | //| current sample. That is exact for the running-statistics form | //| this layer implements, and it is also what batch norm does at | //| inference time. The reference's kernel additionally carries two | //| correction terms for the current sample's own contribution to the | //| statistics; both are O(1/batch) and vanish at the window lengths | //| used here, and its variance term (mean*x / 2*var^1.5) does not | //| follow from the derivative in the paper - so they are deliberately| //| NOT copied. Numerical robustness matters far more in this engine | //| than a correction that is already below the noise floor. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::calcInputGradients(CNeuronBaseOCL *NeuronOCL) { if(CheckPointer(NeuronOCL) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID) return false; int units = Neurons(); int inputsCount = NeuronOCL.Neurons(); if(units <= 0 || inputsCount != units) return false; // batch norm is elementwise - a size mismatch means the topology is wrong //--- DEVICE PATH: the gradient, the previous output and the statistics are all already on the //--- device; the result lands in the previous layer's device gradient where its own backward //--- kernels read it. Zero transfers. if(BnDeviceEligible() && NeuronOCL.getOutputIndex() >= 0 && NeuronOCL.getGradientIndex() >= 0 && getGradientIndex() >= 0 && EnsureBnDeviceBuffers()) { if(!m_bnCheckedGrad) return SelfCheckBnHiddenGrad(NeuronOCL); if(DispatchBnHiddenGrad(NeuronOCL)) return true; LatchBnKernelsOff("BatchNormHiddenGrad dispatch failed (error " + IntegerToString(GetLastError()) + ")"); } //--- HOST PATH. EnsureHostAuthoritative(); double grad[]; int count = getGradient(grad); //--- Gradient is allocated with one slot more than Neurons() (see CNeuronBaseOCL::Init); only the //--- real units carry a value. if(count < units) return false; //--- Armed for the weight-update pass, which reads this same Gradient again - see the declaration. if(ArrayCopy(m_gradCache, grad, 0, 0, count) == count) m_gradCached = true; //--- The previous layer's Output, from this sample's forward pass rather than a second read of the //--- same device buffer. Falls back to reading whenever the cache is not armed or does not match. double prevOut[]; if(m_fwdInputCached && ArraySize(m_fwdInputCache) >= inputsCount) { if(ArrayCopy(prevOut, m_fwdInputCache, 0, 0, inputsCount) != inputsCount) return false; } else if(NeuronOCL.getOutputVal(prevOut) < inputsCount) return false; double ig[]; if(ArrayResize(ig, inputsCount) != inputsCount) return false; HiddenGradHost(grad, prevOut, NeuronOCL.Activation(), ig, inputsCount); return NeuronOCL.setGradient(ig); } //+------------------------------------------------------------------+ //| The elementwise backward math, factored out of calcInputGradients | //| (2026-08-09) so the kernel self-check compares against LITERALLY | //| the code the host path runs, not a second copy that could drift. | //+------------------------------------------------------------------+ void CNeuronBatchNormOCL::HiddenGradHost(const double &grad[], const double &prevOut[], ENUM_ACTIVATION act, double &ig[], int n) { for(int i = 0; i < n; i++) { int shift = i * BN_OPT_STRIDE; double g = grad[i]; if(iBatchSize > 1) { double sd = MathMax(MathSqrt(BatchOptions.At(shift + BN_OPT_VAR) + BN_EPSILON), BN_MIN_STD); g = g * BatchOptions.At(shift + BN_OPT_GAMMA) / sd; } //--- Then the previous layer's own activation derivative, byte-for-byte the same treatment //--- Network.cl's CaclHiddenGradient applies - including the clamp-to-range "implied target" //--- reformulation - so that from the previous layer's point of view a batch-norm layer is //--- indistinguishable from any other. NONE falls through unscaled. double out = prevOut[i]; switch(act) { case TANH: g = MathMax(-1.0, MathMin(1.0, g + out)) - out; g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, 1.0 - out * out); break; case SIGMOID: g = MathMax(0.0, MathMin(1.0, g + out)) - out; g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, out * (1.0 - out)); break; case PRELU: g = g * (out >= 0 ? 1.0 : 0.01); break; default: break; } ig[i] = g; } } //+------------------------------------------------------------------+ //| gamma/beta update. NeuronOCL is the previous layer and is | //| deliberately unused: batch norm consumes its output elementwise, | //| so there is no incoming weight matrix to update (the CNet | //| constructor gives the previous layer 0 outgoing weights when this | //| layer follows it). | //| | //| NO WEIGHT DECAY on gamma/beta - unlike every other parameter in | //| this engine, and on purpose. Decaying gamma toward zero shrinks | //| the layer's output toward the constant beta, which is the exact | //| pathology this layer was added to stop; excluding normalization | //| parameters from weight decay is standard practice for the same | //| reason. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::updateInputWeights(CNeuronBaseOCL *NeuronOCL) { if(CheckPointer(BatchOptions) == POINTER_INVALID) return false; if(iBatchSize <= 1) return true; // normalization off - gamma/beta are not in the graph int units = Neurons(); //--- This per-sample path stays HOST-side even in kernel mode, deliberately: it only runs when the //--- net's train batch is 1 (online learning on a deployed model - once per confirmed bar, not per //--- training sample), so kernelizing it buys nothing, and the host step is the reference //--- implementation. What kernel mode DOES require is the sync bracket: pull the device statistics //--- before stepping (NX and the running stats are device-fresh), push the stepped gamma/beta back //--- after, so the next forward kernel sees them. EnsureHostAuthoritative(); //--- Gradient from calcInputGradients' read this same sample, not a second device round trip. double grad[]; int count = m_gradCached ? ArrayCopy(grad, m_gradCache) : getGradient(grad); if(count < units || units <= 0) return false; //--- Same bias-corrected step size the dense Adam path computes, so gamma/beta move on the same //--- schedule as the weights around them. double lt = (optimization == SGD) ? 0.0 : eta * sqrt(1 - pow(b2, t)) / (1 - pow(b1, t)); for(int i = 0; i < units; i++) { int shift = i * BN_OPT_STRIDE; double g = grad[i]; //--- dL/dgamma = dL/dy * nx | dL/dbeta = dL/dy double gGamma = g * BatchOptions.At(shift + BN_OPT_NX); double gBeta = g; if(!StepGammaBeta(shift, gGamma, gBeta, lt)) return false; } if(optimization != SGD && t < INT_MAX) t++; //--- Keep the device copy coherent so a later kernel forward reads the stepped gamma/beta. if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite()) return false; return true; } //+------------------------------------------------------------------+ //| ONE unit's gamma/beta optimizer step, factored out of | //| updateInputWeights so the mini-batch path | //| (ApplyAccumulatedGradients) takes the identical step on the batch | //| mean instead of carrying a second copy of this arithmetic. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::StepGammaBeta(int shift, double gGamma, double gBeta, double lt) { { double gamma = BatchOptions.At(shift + BN_OPT_GAMMA); double beta = BatchOptions.At(shift + BN_OPT_BETA); //--- Matches NormalizeHost's self-heal, and this is the copy that makes it STICK: the clamped //--- write at the end of this loop persists the repaired value, whereas the forward pass only //--- substitutes one locally. Without this a model that loaded a NaN gamma would normalize //--- correctly but never train that unit's scale again, since NaN + anything stays NaN. if(!MathIsValidNumber(gamma)) gamma = 1.0; if(!MathIsValidNumber(beta)) beta = 0.0; double dGamma = 0.0, dBeta = 0.0; if(optimization == SGD) { dGamma = eta * gGamma + alpha * BatchOptions.At(shift + BN_OPT_MG); dBeta = eta * gBeta + alpha * BatchOptions.At(shift + BN_OPT_MB); if(!BatchOptions.Update(shift + BN_OPT_MG, dGamma) || !BatchOptions.Update(shift + BN_OPT_MB, dBeta)) return false; } else { //--- Second momentum is stored ALREADY square-rooted so it can be used as the denominator //--- directly, the same convention this engine's UpdateWeightsAdam kernels use //--- (matrix_v[wi] = sqrt(...)). //--- This layer squares the stored value back before re-entering the recursion, which is the //--- actual Adam second moment. RECONCILED 2026-08-09: the weight kernels used to feed the //--- stored sqrt straight back in as `v` - mixing a standard deviation with a variance - and //--- that recursion has a fixed point at v ~= b2 for any |g| below 1, so its denominator //--- stopped tracking the gradient scale and Adam degraded to plain SGD. It was left alone //--- here on the grounds that fixing it would change every existing model on all four //--- backends; it changed them for the better, and the reason this layer kept training while //--- the conv/LSTM stages behind it froze was precisely that gamma/beta got it right. All //--- four tiers now match this form - see AI\Network.cl's UpdateWeightsAdam. double mg = b1 * BatchOptions.At(shift + BN_OPT_MG) + (1 - b1) * gGamma; double mb = b1 * BatchOptions.At(shift + BN_OPT_MB) + (1 - b1) * gBeta; double vg = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VG), 2) + (1 - b2) * gGamma * gGamma); double vb = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VB), 2) + (1 - b2) * gBeta * gBeta); dGamma = lt * mg / (vg > 0 ? vg : lt * 10); dBeta = lt * mb / (vb > 0 ? vb : lt * 10); if(!BatchOptions.Update(shift + BN_OPT_MG, mg) || !BatchOptions.Update(shift + BN_OPT_MB, mb) || !BatchOptions.Update(shift + BN_OPT_VG, vg) || !BatchOptions.Update(shift + BN_OPT_VB, vb)) return false; } dGamma = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dGamma)); dBeta = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dBeta)); //--- `return true` and not `continue`: this is one unit's step now that the loop lives in the //--- caller, and a non-finite delta means SKIP this unit, never fail the layer. if(!MathIsValidNumber(dGamma) || !MathIsValidNumber(dBeta)) return true; if(!BatchOptions.Update(shift + BN_OPT_GAMMA, MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, gamma + dGamma))) || !BatchOptions.Update(shift + BN_OPT_BETA, MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, beta + dBeta)))) return false; } return true; } //+------------------------------------------------------------------+ //| MINI-BATCH (2026-08-09 audit, F4). gamma/beta are host-side | //| parameters, so their accumulation is a plain host sum - no kernel | //| and no extra device buffer. The per-sample normalized input | //| (BN_OPT_NX) is overwritten by each forward pass, which is exactly | //| why dL/dgamma has to be formed HERE, per sample, rather than | //| reconstructed at the end of the batch. | //| Held in plain host arrays rather than new BatchOptions slots on | //| purpose: BN_OPT_STRIDE is baked into every persisted .nnw through | //| getWeightsBN, so widening it would invalidate every saved model. | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::accumulateInputWeightGrads(CNeuronBaseOCL *NeuronOCL) { //--- The outgoing dense matrix is accumulated by the layer above, exactly as for a plain dense //--- neuron; this adds only the normalization parameters' own gradients. if(CheckPointer(BatchOptions) == POINTER_INVALID || iBatchSize <= 1) return true; // normalization off - gamma/beta are not in the graph int units = Neurons(); //--- DEVICE PATH: the gradient and NX are both device-fresh, so the accumulation happens where //--- they already live. One dispatch, no transfers. if(BnDeviceEligible() && getGradientIndex() >= 0 && EnsureBnDeviceBuffers()) { if(!m_bnCheckedAccum) return SelfCheckBnAccum(); if(DispatchBnAccum()) return true; LatchBnKernelsOff("BatchNormAccumGammaBeta dispatch failed (error " + IntegerToString(GetLastError()) + ")"); //--- The latch drained whatever the kernels had accumulated this batch into m_accGamma/ //--- m_accBeta; the host code below adds THIS sample on top, so nothing is lost or doubled. } EnsureHostAuthoritative(); //--- Gradient from calcInputGradients' read this same sample - see the declaration comment. This is //--- the batched twin of updateInputWeights above and takes the value from the same place. double grad[]; int count = m_gradCached ? ArrayCopy(grad, m_gradCache) : getGradient(grad); if(count < units || units <= 0) return false; if(ArraySize(m_accGamma) != units || ArraySize(m_accBeta) != units) { ArrayResize(m_accGamma, units); ArrayResize(m_accBeta, units); ArrayInitialize(m_accGamma, 0.0); ArrayInitialize(m_accBeta, 0.0); } for(int i = 0; i < units; i++) { double g = grad[i]; m_accGamma[i] += g * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX); m_accBeta[i] += g; } return true; } //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::BeginGradAccum(void) { bool ok = CNeuronBaseOCL::BeginGradAccum(); ArrayInitialize(m_accGamma, 0.0); ArrayInitialize(m_accBeta, 0.0); //--- Belt and braces for the device accumulator: the apply kernel zeroes it itself, but a batch //--- abandoned mid-way (era boundary, restore) must not leak its partial sums into the next one. if(CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0) ok = ZeroOptimizerBuffer(m_bnAcc) && ok; return ok; } //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::ApplyAccumulatedGradients(double scale) { //--- Outgoing dense matrix first, through the shared block optimizer. bool ok = ApplyAccumToBlock(Weights, GradAccum, FirstMomentum, SecondMomentum, DeltaWeights, scale); if(CheckPointer(BatchOptions) != POINTER_INVALID && iBatchSize > 1) { double lt = (optimization == SGD) ? 0.0 : eta * sqrt(1 - pow(b2, t)) / (1 - pow(b1, t)); //--- DEVICE PATH: the batch was accumulated by the kernels, so it is stepped by the kernel too. //--- A dispatch failure latches (which drains the device sums into m_accGamma/m_accBeta) and //--- drops to the host loop below, so the batch is stepped exactly once either way. bool stepped = false; if(BnDeviceEligible() && CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0) { if(!m_bnCheckedApply) stepped = SelfCheckBnApply(scale, lt); else if(DispatchBnApply(scale, lt)) stepped = true; else LatchBnKernelsOff("BatchNormApplyGammaBeta dispatch failed (error " + IntegerToString(GetLastError()) + ")"); } if(!stepped) { EnsureHostAuthoritative(); int units = MathMin(Neurons(), MathMin(ArraySize(m_accGamma), ArraySize(m_accBeta))); for(int i = 0; i < units; i++) if(!StepGammaBeta(i * BN_OPT_STRIDE, m_accGamma[i] * scale, m_accBeta[i] * scale, lt)) { ok = false; break; } ArrayInitialize(m_accGamma, 0.0); ArrayInitialize(m_accBeta, 0.0); //--- Keep a created-but-idle device copy coherent with the host step. if(BatchOptions.GetIndex() >= 0) ok = BatchOptions.BufferWrite() && ok; } } //--- One step, so t advances once - matching every other layer's batched apply. if(optimization != SGD && t < INT_MAX) t++; return ok; } //+------------------------------------------------------------------+ //| Outgoing dense weight matrix followed by the whole BatchOptions | //| block, as one flat array - see the declaration comment. | //+------------------------------------------------------------------+ int CNeuronBatchNormOCL::getWeightsBN(double &values[]) { //--- Checkpoints and the health report call this mid-training; in kernel mode the statistics live //--- on the device, so pull them first. Read-only - the device stays authoritative. SyncOptionsToHost(); double w[]; int wCount = getWeights(w); if(wCount < 0) wCount = 0; int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total(); if(ArrayResize(values, wCount + oCount) != wCount + oCount) return 0; for(int i = 0; i < wCount; i++) values[i] = w[i]; for(int i = 0; i < oCount; i++) values[wCount + i] = BatchOptions.At(i); return wCount + oCount; } //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::setWeightsBN(double &values[]) { int total = ArraySize(values); int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total(); int wCount = total - oCount; if(wCount < 0) return false; // snapshot predates this layer's parameters - refuse rather than half-restore if(wCount > 0) { double w[]; if(ArrayResize(w, wCount) != wCount) return false; for(int i = 0; i < wCount; i++) w[i] = values[i]; if(!setWeights(w)) return false; } for(int i = 0; i < oCount; i++) if(!BatchOptions.Update(i, values[wCount + i])) return false; //--- Full host overwrite of every slot: push it so a kernel-mode net keeps computing on the //--- RESTORED statistics rather than the diverged ones the device still holds. After this the two //--- copies are identical, so whichever side was authoritative remains consistent. if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite()) return false; return true; } //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::Save(const int file_handle) { //--- The statistics ARE the layer's inference behaviour; in kernel mode their current values live //--- on the device, and a .nnw written from the stale host mirror would be a different model. SyncOptionsToHost(); if(!CNeuronBaseOCL::Save(file_handle)) return false; if(FileWriteInteger(file_handle, iBatchSize, INT_VALUE) < INT_VALUE) return false; if(FileWriteInteger(file_handle, iSamplesSeen, INT_VALUE) < INT_VALUE) return false; if(CheckPointer(BatchOptions) == POINTER_INVALID) return false; return BatchOptions.Save(file_handle); } //+------------------------------------------------------------------+ //| | //+------------------------------------------------------------------+ bool CNeuronBatchNormOCL::Load(const int file_handle) { if(!CNeuronBaseOCL::Load(file_handle)) return false; iBatchSize = FileReadInteger(file_handle, INT_VALUE); iSamplesSeen = FileReadInteger(file_handle, INT_VALUE); if(CheckPointer(BatchOptions) == POINTER_INVALID) { BatchOptions = new CBufferDouble(); if(CheckPointer(BatchOptions) == POINTER_INVALID) return false; } if(!BatchOptions.Load(file_handle)) return false; //--- Loaded state is a full host overwrite - push it if a device copy already exists (a reload into //--- a live net), and the host is authoritative until the first kernel forward. if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite()) return false; m_bnDeviceAuthoritative = false; //--- A model whose statistics block does not match its own width is not usable for inference - //--- fail loudly here rather than index past the end on the first forward pass. if(BatchOptions.Total() != Neurons() * BN_OPT_STRIDE) { Print(__FUNCTION__ + ": batch-norm parameter block is " + IntegerToString(BatchOptions.Total()) + " values but this layer has " + IntegerToString(Neurons()) + " units (expected " + IntegerToString(Neurons() * BN_OPT_STRIDE) + ") - file does not match the topology"); return false; } return true; } #endif // WARRIOR_NEURON_BATCHNORM_MQH //+------------------------------------------------------------------+