Warrior_EA/AI/NeuronBatchNorm.mqh

//+------------------------------------------------------------------+
//|                                              NeuronBatchNorm.mqh  |
//|                                                     AnimateDread |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
//| CNeuronBatchNormOCL - batch normalization (Ioffe & Szegedy 2015). |
//| Needs CNeuronBaseOCL (AI\Network.mqh) already declared; included  |
//| from there, not standalone.                                       |
//|                                                                   |
//| WHY THIS EXISTS                                                   |
//| Every hidden stage in this project is unbounded (PRELU dense,     |
//| PRELU conv) and the only bounded stage in the whole forward path  |
//| was the SIGMOID classification head. That is a network with no    |
//| internal scale control at all, and it showed: across four         |
//| topologies on SP500 H1 the models peaked early then decayed       |
//| monotonically into all-Neutral, and the decay ordered exactly by  |
//| DEPTH - the shallow perceptron held ~52% balanced accuracy while  |
//| the deepest (conv+pool+LSTM+dense taper) sat at the 33.3%         |
//| one-class floor. That depth ordering is the signature of internal |
//| covariate shift, which is precisely what this layer addresses.    |
//|                                                                   |
//| Two further consequences worth spelling out, because they are the |
//| actual mechanism by which this is expected to help here:          |
//|                                                                   |
//|  1. It decouples weight decay from the learned function. With a   |
//|     normalized layer downstream, scaling the weights that feed it |
//|     leaves the output unchanged - so WEIGHT_DECAY can no longer   |
//|     grind the discriminative signal away, it only rescales the    |
//|     effective learning rate (van Laarhoven 2017). The observed    |
//|     failure was exactly a slow monotonic shrink of the per-bar    |
//|     logit spread (0.45 -> 0.38 over ~200 eras) until the evidence |
//|     tilt fell under the class-prior tilt and argmax degenerated   |
//|     to constant-Neutral. See WEIGHT_DECAY's comment in            |
//|     AI\Network.mqh, which describes that mechanism first-hand.    |
//|  2. It is what would make an UNBOUNDED logit head viable. The     |
//|     2026-07-27 attempt to run the head at NONE blew up (IS error  |
//|     5.6e15) specifically because nothing upstream constrained     |
//|     scale. Do not retry that without a normalization layer        |
//|     immediately before the head - and then also drop              |
//|     CLASS_LOGIT_SCALE to 1.0 and BIAS_MAGNITUDE to ~0.5.          |
//|                                                                   |
//| IMPLEMENTATION SHAPE - host-side reference, OpenCL kernels on top |
//| The math is elementwise O(n) and originally ran host-side on all  |
//| four tiers, against the host mirrors of the device buffers, so    |
//| the backends could not drift. The predicted cost - "on a real GPU |
//| it is a PCIe hop; if that ever matters, port these methods to     |
//| kernels - the math here is the specification" - came due on       |
//| 2026-08-09: the blocking syncs (four per BN layer per sample,     |
//| forward AND backward) were a large part of why an RX 580 lost to  |
//| a CPU thread pool, and Market builds forbid DLL imports, so the   |
//| OpenCL tier is what paying clients actually run.                  |
//| The port did exactly what that sentence said: Network.cl's        |
//| BatchNorm* kernels are transcriptions of NormalizeHost /          |
//| HiddenGradHost / StepGammaBeta, the host methods REMAIN the       |
//| runtime for the DLL and pure-MQL5 tiers and the reference the     |
//| kernels must match, and each kernel is verified against its host  |
//| twin on first use (SelfCheckBn*) - a disagreement latches the     |
//| kernels off process-wide (g_bnKernelUsable) after resyncing from  |
//| the good copy, so a transcription bug costs a warning and some    |
//| speed, never a poisoned .nnw. Edit host and kernel together.      |
//|                                                                   |
//| STATISTICS: exponential moving, not a stored mini-batch.          |
//| Training here is pure online SGD - one weight update per sample   |
//| (CNet::backProp per bar), never a batched pass - so there is no   |
//| mini-batch to average over. iBatchSize is therefore an EMA WINDOW |
//| LENGTH, not a buffer size: mean and variance are updated per      |
//| sample toward the last ~iBatchSize samples. Same formulation as   |
//| the NeuroNet_DNG reference's BatchFeedForward kernel; the book    |
//| (ch. 6.1.1) explicitly endorses the exponential form to avoid     |
//| storing per-neuron history.                                       |
//|                                                                   |
//| The statistics keep adapting on EVERY forward pass, including     |
//| out-of-sample scoring and live inference - they are never frozen  |
//| the way classic batch norm freezes them at inference time. That   |
//| is deliberate and matches the rest of this system (see            |
//| OnlineLearnStep: the deployed model is designed to keep tracking  |
//| the market). Two consequences worth knowing: OOS scoring lets the |
//| statistics see OOS activations - unsupervised, no label           |
//| information, but not a hermetic holdout - and a restored          |
//| checkpoint rewinds the statistics along with gamma/beta, since    |
//| getWeightsBN/setWeightsBN carry both.                             |
//+------------------------------------------------------------------+
#ifndef WARRIOR_NEURON_BATCHNORM_MQH
#define WARRIOR_NEURON_BATCHNORM_MQH
//--- Per-neuron slot layout inside BatchOptions. A FIXED stride of 9 on every optimizer, unlike the
//--- reference's 7-or-9 split: the two unused doubles per neuron are noise next to a weight matrix,
//--- and a stride that changes with a persisted enum is an indexing bug waiting for the first time
//--- somebody switches TrainingOptimizer on an existing model.
#define BN_OPT_STRIDE   9
#define BN_OPT_MEAN     0   // running mean
#define BN_OPT_VAR      1   // running variance
#define BN_OPT_NX       2   // normalized input, cached from the forward pass for the backward pass
#define BN_OPT_GAMMA    3   // learned scale, init 1
#define BN_OPT_BETA     4   // learned shift, init 0
#define BN_OPT_MG       5   // gamma: Adam first momentum, or SGD previous delta
#define BN_OPT_MB       6   // beta:  Adam first momentum, or SGD previous delta
#define BN_OPT_VG       7   // gamma: Adam second momentum (stored already square-rooted, as the
#define BN_OPT_VB       8   // beta:  ... UpdateWeightsAdam kernels in this engine also do)
//--- Variance floor. Serves the same purpose as MIN_ACTIVATION_DERIVATIVE: a neuron whose input
//--- happens to be near-constant over the EMA window has a near-zero variance, and dividing by its
//--- square root turns rounding noise into an arbitrarily large activation that then propagates.
//--- Applied to the standard deviation (not the variance) so it reads as "no unit is amplified by
//--- more than 1e4", which is the property that actually matters.
#define BN_EPSILON      1.0e-10
#define BN_MIN_STD      1.0e-4
//--- Sanity ceiling on a single input value, applied before it can touch the running statistics below.
//--- Not a normalization choice - every legitimate feature in this codebase is clamped to single digits
//--- long before it gets here (see BufferTempDataCompute). This is purely the bound that keeps the
//--- statistics arithmetic inside double range, so a garbage value degrades one unit on one bar instead
//--- of permanently latching the layer. See NormalizeHost() for the failure this exists to stop.
#define BN_MAX_INPUT    1.0e6
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
class CNeuronBatchNormOCL : public CNeuronBaseOCL
  {
protected:
   int               iBatchSize;    // EMA window length; <=1 disables normalization entirely
   //--- When true the running statistics are USED but not UPDATED, i.e. classic batch-norm inference
   //--- semantics. Off by default (see the note on adaptation in the class header). Exists because a
   //--- forward pass is otherwise not a pure function of its input: two evaluations of the same bar
   //--- return slightly different answers because the first one moved the statistics. Anything that
   //--- COMPARES two forward passes has to freeze them first or it is measuring its own side effect -
   //--- CExpertSignalAIBase::ValidateCpuInference() is exactly that, and would have silently failed
   //--- its DLL-free check on nothing but the one EMA step it took itself.
   bool              bStatsFrozen;
   //--- Number of samples seen. Used only to ramp the effective window up from 1 to iBatchSize over
   //--- the first iBatchSize samples (standard EMA bias correction). Without it the running mean
   //--- starts at 0 and approaches the true mean over ~iBatchSize samples, during which the layer
   //--- emits near-zero for everything - a cold-start dead zone indistinguishable from the very
   //--- collapse this layer exists to prevent.
   int               iSamplesSeen;
   //--- Persisted through Save/Load like any other parameter - gamma/beta are learned, and the running
   //--- statistics ARE the layer's inference behaviour, so a model that loses them is not the model
   //--- that was trained.
   //--- A CBufferDouble since 2026-08-09 so the OpenCL kernels can own it device-side; on every other
   //--- tier it behaves exactly as the CArrayDouble it was (no device buffer is ever created). WHO IS
   //--- AUTHORITATIVE is tracked by m_bnDeviceAuthoritative below, and every host read/write of this
   //--- buffer goes through the sync helpers - an unsynced BufferRead would clobber good host state
   //--- with stale device state, and an unsynced host write would be silently overwritten by the next
   //--- kernel. The 9-slot stride is baked into persisted .nnw files and into Network.cl's BN_OPT_*
   //--- copies; the two must stay identical.
   CBufferDouble    *BatchOptions;
   //--- Mini-batch running sums of dL/dgamma and dL/dbeta, one entry per unit. Host-only and NOT
   //--- persisted: transient within a batch, and every save point flushes first. See
   //--- accumulateInputWeightGrads for why they are not extra BatchOptions slots.
   double            m_accGamma[];
   double            m_accBeta[];
   //--- PER-SAMPLE TRANSFER CACHES. This layer's math is host-side while its neighbours are device
   //--- resident, so every value it touches crosses the bus - and each crossing is a BLOCKING sync,
   //--- which is what makes it expensive rather than the bytes. Per sample it used to do four reads:
   //---   feedForward          -> previous layer's Output
   //---   calcInputGradients   -> own Gradient, and the previous layer's Output AGAIN
   //---   update/accumulate    -> own Gradient AGAIN
   //--- The two repeats are exact duplicates. Nothing writes the previous layer's Output between the
   //--- forward pass and the backward pass, and nothing writes this layer's Gradient between the
   //--- gradient pass and the weight-update pass - CNet::backPropOCL runs those as two separate
   //--- top-to-bottom loops, and only the first one writes gradients.
   //---
   //--- Caching them is therefore bit-exact, not an approximation: the same values, read once.
   //--- Each cache is armed by its producer and DISARMED by feedForward, so a consumer that runs
   //--- without its producer having run this sample falls back to reading the buffer rather than
   //--- silently using the previous sample's data. That matters concretely: a batch-norm at layer 1
   //--- never gets calcInputGradients called at all (backPropOCL's loop stops at layerNum > 0 - the
   //--- same asymmetry documented there for a layer-1 LSTM), so its gradient cache is never armed.
   double            m_fwdInputCache[];
   bool              m_fwdInputCached;
   double            m_gradCache[];
   bool              m_gradCached;
   //--- DEVICE PATH (OpenCL only, 2026-08-09). m_bnAcc holds the mini-batch gamma/beta gradient sums
   //--- on the device (2 floats per unit: gamma then beta), the kernel twin of m_accGamma/m_accBeta.
   //--- m_bnDeviceAuthoritative says the DEVICE copy of BatchOptions is the truth (kernels have
   //--- written it since the last host sync); the m_bnChecked* flags latch each kernel's one-time
   //--- self-check against its host twin. The checks are the whole safety story for shipping kernels
   //--- that could not be built on the dev machine: a transcription or dispatch-binding error is
   //--- caught on its first use, the layer resyncs from the good copy, latches the kernels off
   //--- process-wide, and training continues host-side - a warning and some speed, never a poisoned
   //--- .nnw.
   CBufferDouble    *m_bnAcc;
   bool              m_bnDeviceAuthoritative;
   bool              m_bnCheckedFwd;
   bool              m_bnCheckedGrad;
   bool              m_bnCheckedAccum;
   bool              m_bnCheckedApply;
   //--- eligibility + buffer management for the kernel path
   bool              BnDeviceEligible(void);
   bool              EnsureBnDeviceBuffers(void);
   //--- read-only pull of the device statistics into the host mirror (checkpoints/saves mid-training);
   //--- device stays authoritative
   void              SyncOptionsToHost(void);
   //--- full handover to the host path: pull statistics, drain the device accumulator into
   //--- m_accGamma/m_accBeta so a mid-batch handover loses nothing, clear the flag
   void              EnsureHostAuthoritative(void);
   //--- one-way process-wide latch + this layer's handover, with the reason printed once
   void              LatchBnKernelsOff(const string reason);
   //--- kernel dispatches (return false on any SetArgument/Execute failure, no logging - the caller
   //--- decides between latching and falling back)
   bool              DispatchBnForward(CNeuronBaseOCL *NeuronOCL, double w);
   bool              DispatchBnHiddenGrad(CNeuronBaseOCL *NeuronOCL);
   bool              DispatchBnAccum(void);
   bool              DispatchBnApply(double scale, double lt);
   //--- one-time kernel-vs-host comparisons; each returns the OPERATION's result (true = the work got
   //--- done correctly, by whichever path survived), never "the kernel matched"
   bool              SelfCheckBnForward(CNeuronBaseOCL *NeuronOCL);
   bool              SelfCheckBnHiddenGrad(CNeuronBaseOCL *NeuronOCL);
   bool              SelfCheckBnAccum(void);
   bool              SelfCheckBnApply(double scale, double lt);
   //--- normalized disagreement: |got-ref| / (1e-3 * max(1,|ref|)), <=1 passes. DBL_MAX when exactly
   //--- one side is non-finite. The 1e-3 relative band is ~4 decades above fp32-vs-fp64 noise and ~3
   //--- below any real transcription error (wrong slot, wrong sign, wrong buffer), so it cannot
   //--- confuse the two.
   double            BnDiffScore(double ref, double got);
   //--- host twin of the backward elementwise math, factored out of calcInputGradients so the
   //--- self-check compares against literally the same code the host path runs
   void              HiddenGradHost(const double &grad[], const double &prevOut[],
                                    ENUM_ACTIVATION act, double &ig[], int n);
   //---
   virtual bool      feedForward(CNeuronBaseOCL *NeuronOCL);
   virtual bool      feedForwardCPU(CNeuronBaseOCL *NeuronOCL);
   virtual bool      updateInputWeights(CNeuronBaseOCL *NeuronOCL);
   virtual bool      accumulateInputWeightGrads(CNeuronBaseOCL *NeuronOCL);
   //--- one unit's gamma/beta step, shared by the per-sample and per-batch paths
   bool              StepGammaBeta(int shift, double gGamma, double gBeta, double lt);

public:
   virtual bool      BeginGradAccum(void);
   virtual bool      ApplyAccumulatedGradients(double scale);
protected:
   //--- shared by feedForward/feedForwardCPU: the whole forward transform for one already-read input
   //--- vector, writing straight into the host mirror of Output.
   bool              NormalizeHost(const double &inputs[], int count);
   //--- ensures BatchOptions exists and is sized/seeded for `neurons` units
   bool              InitOptions(int neurons);

public:
                     CNeuronBatchNormOCL(void);
                    ~CNeuronBatchNormOCL(void);
   //---
   virtual bool      Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type);
   virtual bool      Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type);
   virtual bool      calcInputGradients(CNeuronBaseOCL *NeuronOCL);
   //--- see bStatsFrozen. Deliberately NOT persisted: it is a transient evaluation mode, not model state.
   void              SetStatsFrozen(bool v)  {  bStatsFrozen = v;  }
   //--- Checkpoint/blend support. CNet::CaptureWeights/RestoreWeights/BlendWeightsFrom snapshot one
   //--- flat array per neuron object, so gamma/beta/mean/variance are appended AFTER the outgoing
   //--- dense weight matrix and split back out on restore. Without this the plateau ladder's
   //--- "restore best checkpoint" would put the dense weights back while leaving this layer's
   //--- parameters at whatever the diverged era left behind - a silently mismatched pair.
   virtual int       getWeightsBN(double &values[]);
   virtual bool      setWeightsBN(double &values[]);
   //--- How many of the trailing entries in getWeightsBN's array are BatchOptions rather than the
   //--- outgoing dense matrix. Published so the health report can split that packed block into its
   //--- four very different parts (weights / gamma-beta / running statistics / Adam moments) instead
   //--- of quoting one norm over all of them, which cannot say which is moving.
   int               BatchOptionsTotal(void) const
     {
      return (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
     }
   //--- Zero ONLY the gamma/beta moment slots (BN_OPT_MG/MB/VG/VB) plus the base class's buffers for
   //--- the outgoing dense matrix. The running mean/variance and the learned gamma/beta are MODEL
   //--- state, not optimizer state - they are exactly what getWeightsBN checkpoints and what a restore
   //--- puts back, so wiping them here would undo the restore this reset exists to complete. See
   //--- CNet::ResetOptimizerState.
   virtual bool      ResetOptimizerState(void)
     {
      bool ok = CNeuronBaseOCL::ResetOptimizerState();
      if(CheckPointer(BatchOptions) != POINTER_INVALID)
        {
         //--- Kernel-mode discipline: this zeroes SOME slots of a block whose truth may live on the
         //--- device, so pull first (or the untouched slots would be written back stale), zero, push.
         SyncOptionsToHost();
         int totalSlots = BatchOptions.Total();
         for(int shift = 0; shift + BN_OPT_VB < totalSlots; shift += BN_OPT_STRIDE)
           {
            ok = BatchOptions.Update(shift + BN_OPT_MG, 0.0) && ok;
            ok = BatchOptions.Update(shift + BN_OPT_MB, 0.0) && ok;
            ok = BatchOptions.Update(shift + BN_OPT_VG, 0.0) && ok;
            ok = BatchOptions.Update(shift + BN_OPT_VB, 0.0) && ok;
           }
         if(BatchOptions.GetIndex() >= 0)
            ok = BatchOptions.BufferWrite() && ok;
        }
      return ok;
     }
   //---
   virtual bool      Save(int const file_handle);
   virtual bool      Load(int const file_handle);
   //---
   virtual int       Type(void) const  {  return defNeuronBatchNormOCL;  }
  };
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
CNeuronBatchNormOCL::CNeuronBatchNormOCL(void) : iBatchSize(1), bStatsFrozen(false), iSamplesSeen(0),
   m_fwdInputCached(false), m_gradCached(false), m_bnDeviceAuthoritative(false),
   m_bnCheckedFwd(false), m_bnCheckedGrad(false), m_bnCheckedAccum(false), m_bnCheckedApply(false)
  {
   BatchOptions = NULL;
   m_bnAcc = NULL;
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
CNeuronBatchNormOCL::~CNeuronBatchNormOCL(void)
  {
   if(CheckPointer(BatchOptions) != POINTER_INVALID)
      delete BatchOptions;
   if(CheckPointer(m_bnAcc) != POINTER_INVALID)
      delete m_bnAcc;
  }
//+------------------------------------------------------------------+
//| gamma=1 / beta=0 / zeroed statistics for every unit.              |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::InitOptions(int neurons)
  {
   if(neurons <= 0)
      return false;
   if(CheckPointer(BatchOptions) == POINTER_INVALID)
     {
      BatchOptions = new CBufferDouble();
      if(CheckPointer(BatchOptions) == POINTER_INVALID)
         return false;
     }
   BatchOptions.Clear();
   if(!BatchOptions.Reserve(neurons * BN_OPT_STRIDE))
      return false;
   for(int n = 0; n < neurons; n++)
      for(int s = 0; s < BN_OPT_STRIDE; s++)
         if(!BatchOptions.Add(s == BN_OPT_GAMMA ? 1.0 : 0.0))
            return false;
   iSamplesSeen = 0;
   //--- Full host overwrite: if a device copy exists, push it so the two cannot disagree. The host is
   //--- authoritative after a re-init by definition.
   if(BatchOptions.GetIndex() >= 0)
     {
      if(!BatchOptions.BufferWrite())
         return false;
      m_bnDeviceAuthoritative = false;
     }
   return true;
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type)
  {
   if(!CNeuronBaseOCL::Init(numOutputs, myIndex, open_cl, numNeurons, optimization_type))
      return false;
   //--- Identity forward transform. The activation belongs to the layer AFTER this one; normalizing
   //--- and then squashing in the same step would defeat the point (Ioffe & Szegedy place the
   //--- normalization immediately BEFORE the non-linearity, not around it).
   activation = NONE;
   iBatchSize = (int)batchSize;
   return InitOptions((int)numNeurons);
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type)
  {
   if(!CNeuronBaseOCL::Init(numOutputs, myIndex, direct_ml, numNeurons, optimization_type))
      return false;
   activation = NONE;
   iBatchSize = (int)batchSize;
   return InitOptions((int)numNeurons);
  }
//+------------------------------------------------------------------+
//| The forward transform, host-side. Mirrors the NeuroNet_DNG        |
//| BatchFeedForward kernel, plus the bias-corrected warm-up window.  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::NormalizeHost(const double &inputs[], int count)
  {
   if(CheckPointer(Output) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID)
      return false;
   if(count > Output.Total())
      count = Output.Total();
   if(BatchOptions.Total() < count * BN_OPT_STRIDE)
      return false;
   //--- A window of 1 makes mean==x and variance==0 for every sample, i.e. a constant-zero output.
   //--- Treat it as "normalization off" and pass the input through untouched rather than emit zeros.
   if(iBatchSize <= 1)
     {
      for(int i = 0; i < count; i++)
         if(!Output.Update(i, inputs[i]))
            return false;
      return true;
     }
   if(!bStatsFrozen && iSamplesSeen < iBatchSize)
      iSamplesSeen++;
   //--- Effective window: ramps 1,2,3... up to iBatchSize, so the first samples produce an honest
   //--- running mean instead of one biased toward the zero initializer.
   double w = (double)MathMax(1, iSamplesSeen);
   for(int i = 0; i < count; i++)
     {
      int shift = i * BN_OPT_STRIDE;
      double x = inputs[i];
      //--- The running mean/variance below are PERSISTENT: they live in BatchOptions, are carried into
      //--- the .nnw by getWeightsBN, and every later sample normalizes against them. That makes them a
      //--- LATCH. One non-finite or astronomically large x poisons them permanently, and from then on
      //--- this layer emits NaN on every bar for the rest of the run - the arithmetic gets there in
      //--- three steps: (x-mean)^2 overflows variance to +inf, then mean*(w-1) overflows mean to +inf,
      //--- then delta/sd is inf/inf = NaN. Note it does NOT need x to be non-finite to start: a merely
      //--- huge finite x (an unguarded EMPTY_VALUE feature divided by ATR is ~1e307) passes every
      //--- upstream check and still overflows the square.
      //--- 2026-08-02: this presented as 13,776 identical "BufferWrite failed for buffer 3" lines and
      //--- nothing else. Buffer 3 is the first batch-norm layer's Output - the one sitting on the raw
      //--- input vector - and the CPU DLL's isfinite() boundary check was correctly refusing to store
      //--- the NaN, so the layer's device-side output silently froze at its last good value while
      //--- training carried on against it for a full era.
      //--- A value this large is meaningless whatever produced it, so clamp rather than propagate.
      if(!MathIsValidNumber(x))
         x = 0.0;
      x = MathMax(-BN_MAX_INPUT, MathMin(BN_MAX_INPUT, x));
      double mean = BatchOptions.At(shift + BN_OPT_MEAN);
      double variance = BatchOptions.At(shift + BN_OPT_VAR);
      //--- Self-heal statistics that were already poisoned before this guard existed. Without it an
      //--- affected .nnw stays dead across restarts, because Load faithfully restores the NaN.
      if(!MathIsValidNumber(mean))
         mean = x;
      if(!MathIsValidNumber(variance) || variance < 0.0)
         variance = 0.0;
      if(!bStatsFrozen)
        {
         mean = (mean * (w - 1.0) + x) / w;
         variance = (variance * (w - 1.0) + (x - mean) * (x - mean)) / w;
        }
      double delta = x - mean;
      double sd = MathMax(MathSqrt(variance + BN_EPSILON), BN_MIN_STD);
      double nx = delta / sd;
      //--- gamma/beta cannot go non-finite going forward (updateInputWeights validates and clamps every
      //--- step), but a model SAVED before that guard existed can carry NaN in here through Load. Same
      //--- self-heal as the statistics above: fall back to the identity transform for that unit.
      double gamma = BatchOptions.At(shift + BN_OPT_GAMMA);
      double beta = BatchOptions.At(shift + BN_OPT_BETA);
      if(!MathIsValidNumber(gamma))
         gamma = 1.0;
      if(!MathIsValidNumber(beta))
         beta = 0.0;
      double y = gamma * nx + beta;
      //--- nx is still cached when frozen: it costs nothing and keeps the buffer consistent with the
      //--- output just produced. There is no backward pass while frozen, so nothing reads it.
      if(!bStatsFrozen &&
         (!BatchOptions.Update(shift + BN_OPT_MEAN, mean) ||
          !BatchOptions.Update(shift + BN_OPT_VAR, variance)))
         return false;
      if(!BatchOptions.Update(shift + BN_OPT_NX, nx))
         return false;
      if(!Output.Update(i, y))
         return false;
     }
   return true;
  }
//+------------------------------------------------------------------+
//| Forward. On OpenCL this is a KERNEL since 2026-08-09 (verified    |
//| against NormalizeHost on first use, see SelfCheckBnForward); on   |
//| every other backend it pulls the previous layer's output to the   |
//| host, runs the transform, and pushes its output back.             |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::feedForward(CNeuronBaseOCL *NeuronOCL)
  {
   if(CheckPointer(NeuronOCL) == POINTER_INVALID)
      return false;
   //--- A new sample starts here, so last sample's gradient is no longer valid to reuse. Disarmed
   //--- BEFORE anything can fail below, so an aborted forward pass cannot leave a consumer holding a
   //--- cache that looks current - see the declaration comment.
   m_gradCached = false;
   m_fwdInputCached = false;
   //--- DEVICE PATH: nothing crosses the bus at all. The self-check runs the first time through and
   //--- proves the kernel against NormalizeHost before the device is trusted with the persisted
   //--- statistics; after that it is one dispatch per sample.
   if(BnDeviceEligible() && NeuronOCL.getOutputIndex() >= 0 && getOutputIndex() >= 0 &&
      EnsureBnDeviceBuffers())
     {
      if(!m_bnCheckedFwd)
         return SelfCheckBnForward(NeuronOCL);
      //--- Ramp bookkeeping stays host-side with iSamplesSeen; committed only on success so a failed
      //--- dispatch that falls through to NormalizeHost (which increments itself) cannot double-count.
      int seen = iSamplesSeen;
      if(!bStatsFrozen && seen < iBatchSize)
         seen++;
      if(DispatchBnForward(NeuronOCL, (double)MathMax(1, seen)))
        {
         iSamplesSeen = seen;
         return true;
        }
      LatchBnKernelsOff("BatchNormForward dispatch failed (error " +
                        IntegerToString(GetLastError()) + ")");
     }
   //--- HOST PATH (every non-OpenCL tier, and OpenCL after a latch). If the device had been
   //--- authoritative, pull its state first - NormalizeHost below must advance the REAL statistics,
   //--- not a stale mirror.
   EnsureHostAuthoritative();
   double inputs[];
   int count = NeuronOCL.getOutputVal(inputs);
   if(count <= 0)
      return false;
   if(!NormalizeHost(inputs, count))
      return false;
   //--- Armed for calcInputGradients, which needs exactly these values and would otherwise read the
   //--- same buffer back a second time this sample.
   if(ArrayCopy(m_fwdInputCache, inputs, 0, 0, count) == count)
      m_fwdInputCached = true;
   //--- Unlike the dense/conv kernels this layer's Output does NOT stay device-resident by itself -
   //--- it was just written host-side, so it has to be pushed before the next layer's kernel reads
   //--- it through getOutputIndex().
   return Output.BufferWrite();
  }
//+------------------------------------------------------------------+
//| Pure-MQL5 inference path (no backend at all): the previous        |
//| layer's values live only in its host mirror, so read them there.  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::feedForwardCPU(CNeuronBaseOCL *NeuronOCL)
  {
   if(CheckPointer(NeuronOCL) == POINTER_INVALID)
      return false;
   int count = NeuronOCL.Neurons();
   if(count <= 0)
      return false;
   double inputs[];
   if(ArrayResize(inputs, count) != count)
      return false;
   for(int i = 0; i < count; i++)
      inputs[i] = NeuronOCL.OutputHost(i);
   //--- No BufferWrite: there is no device buffer on this path, and Output's host mirror is what
   //--- the next layer's feedForwardCPU (and GetOutputsCPU) read.
   return NormalizeHost(inputs, count);
  }
//+------------------------------------------------------------------+
//| DEVICE PATH plumbing - see the m_bnAcc declaration comment for    |
//| the design. Everything below is OpenCL-only.                      |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::BnDeviceEligible(void)
  {
   return g_bnKernelUsable && iBatchSize > 1 &&
          CheckPointer(OpenCL) != POINTER_INVALID &&
          CheckPointer(BatchOptions) != POINTER_INVALID;
  }
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::EnsureBnDeviceBuffers(void)
  {
   //--- BufferCreate pushes the current host contents, so a freshly-created device copy is exactly
   //--- the state the host was authoritative over - the invariant every self-check relies on.
   if(BatchOptions.GetIndex() < 0 && !BatchOptions.BufferCreate(OpenCL))
      return false;
   if(CheckPointer(m_bnAcc) == POINTER_INVALID)
     {
      m_bnAcc = new CBufferDouble();
      if(CheckPointer(m_bnAcc) == POINTER_INVALID)
         return false;
      if(!m_bnAcc.BufferInit(Neurons() * 2, 0.0))
         return false;
     }
   if(m_bnAcc.GetIndex() < 0 && !m_bnAcc.BufferCreate(OpenCL))
      return false;
   return true;
  }
//+------------------------------------------------------------------+
void CNeuronBatchNormOCL::SyncOptionsToHost(void)
  {
   //--- Read-only pull for checkpoints/saves taken mid-training; the device REMAINS authoritative.
   //--- Guarded by the flag, because an unguarded BufferRead on a host-authoritative layer would
   //--- clobber good host state with whatever stale bytes the device still holds.
   if(m_bnDeviceAuthoritative && CheckPointer(BatchOptions) != POINTER_INVALID &&
      BatchOptions.GetIndex() >= 0)
      BatchOptions.BufferRead();
  }
//+------------------------------------------------------------------+
void CNeuronBatchNormOCL::EnsureHostAuthoritative(void)
  {
   if(!m_bnDeviceAuthoritative)
      return;
   if(CheckPointer(BatchOptions) != POINTER_INVALID && BatchOptions.GetIndex() >= 0)
      BatchOptions.BufferRead();
   //--- Drain the device-side batch accumulator into the host arrays, so a handover in the MIDDLE of
   //--- a batch keeps the samples the kernels already accumulated - without this, latching off after
   //--- sample 3 of 8 would silently drop three samples' gradients from the batch.
   if(CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0 && m_bnAcc.BufferRead())
     {
      int units = Neurons();
      if(ArraySize(m_accGamma) != units || ArraySize(m_accBeta) != units)
        {
         ArrayResize(m_accGamma, units);
         ArrayResize(m_accBeta, units);
         ArrayInitialize(m_accGamma, 0.0);
         ArrayInitialize(m_accBeta, 0.0);
        }
      for(int i = 0; i < units && 2 * i + 1 < m_bnAcc.Total(); i++)
        {
         m_accGamma[i] += m_bnAcc.At(2 * i);
         m_accBeta[i]  += m_bnAcc.At(2 * i + 1);
        }
      ZeroOptimizerBuffer(m_bnAcc);
     }
   m_bnDeviceAuthoritative = false;
  }
//+------------------------------------------------------------------+
void CNeuronBatchNormOCL::LatchBnKernelsOff(const string reason)
  {
   if(g_bnKernelUsable)
     {
      g_bnKernelUsable = false;
      Print(__FUNCTION__ + ": BATCH-NORM KERNELS DISABLED for the rest of this run - " + reason +
            ". Every batch-norm layer falls back to the host implementation: results stay correct "
            "(the host path is the reference the kernels were transcribed from), each layer just pays "
            "its device round-trips again. State was resynced from the good copy before anything "
            "could persist.");
     }
   EnsureHostAuthoritative();
  }
//+------------------------------------------------------------------+
double CNeuronBatchNormOCL::BnDiffScore(double ref, double got)
  {
   bool fRef = MathIsValidNumber(ref), fGot = MathIsValidNumber(got);
   if(!fRef || !fGot)
      return (fRef == fGot) ? 0.0 : DBL_MAX; // both poisoned the same way is agreement
   return MathAbs(got - ref) / (1.0e-3 * MathMax(1.0, MathAbs(ref)));
  }
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::DispatchBnForward(CNeuronBaseOCL *NeuronOCL, double w)
  {
   uint offset[1] = {0};
   uint size[1];
   size[0] = (uint)Neurons();
   if(!OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_matrix_i, NeuronOCL.getOutputIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_matrix_o, getOutputIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormForward, def_k_bnf_options, BatchOptions.GetIndex()) ||
      !OpenCL.SetArgument(def_k_BatchNormForward, def_k_bnf_w, (float)w) ||
      !OpenCL.SetArgument(def_k_BatchNormForward, def_k_bnf_frozen, bStatsFrozen ? 1 : 0))
      return false;
   ResetLastError();
   if(!OpenCL.Execute(def_k_BatchNormForward, 1, offset, size))
      return false;
   //--- The kernel just wrote the running statistics on the device: it is now the authority.
   m_bnDeviceAuthoritative = true;
   return true;
  }
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::DispatchBnHiddenGrad(CNeuronBaseOCL *NeuronOCL)
  {
   uint offset[1] = {0};
   uint size[1];
   size[0] = (uint)Neurons();
   if(!OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_matrix_g, getGradientIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_prev_o, NeuronOCL.getOutputIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_prev_g, NeuronOCL.getGradientIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormHiddenGrad, def_k_bnh_options, BatchOptions.GetIndex()) ||
      !OpenCL.SetArgument(def_k_BatchNormHiddenGrad, def_k_bnh_activation, NativeActivationCode(NeuronOCL.Activation())))
      return false;
   ResetLastError();
   return OpenCL.Execute(def_k_BatchNormHiddenGrad, 1, offset, size);
  }
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::DispatchBnAccum(void)
  {
   uint offset[1] = {0};
   uint size[1];
   size[0] = (uint)Neurons();
   if(!OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_matrix_g, getGradientIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_options, BatchOptions.GetIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormAccumGammaBeta, def_k_bna_acc, m_bnAcc.GetIndex()))
      return false;
   ResetLastError();
   return OpenCL.Execute(def_k_BatchNormAccumGammaBeta, 1, offset, size);
  }
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::DispatchBnApply(double scale, double lt)
  {
   uint offset[1] = {0};
   uint size[1];
   size[0] = (uint)Neurons();
   if(!OpenCL.SetArgumentBuffer(def_k_BatchNormApplyGammaBeta, def_k_bnp_options, BatchOptions.GetIndex()) ||
      !OpenCL.SetArgumentBuffer(def_k_BatchNormApplyGammaBeta, def_k_bnp_acc, m_bnAcc.GetIndex()) ||
      !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_scale, (float)scale) ||
      !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_lt, (float)lt) ||
      !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_b1, (float)b1) ||
      !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_b2, (float)b2) ||
      !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_lr, (float)eta) ||
      !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_momentum, (float)alpha) ||
      !OpenCL.SetArgument(def_k_BatchNormApplyGammaBeta, def_k_bnp_optimizer, (optimization == SGD) ? 0 : 1))
      return false;
   ResetLastError();
   if(!OpenCL.Execute(def_k_BatchNormApplyGammaBeta, 1, offset, size))
      return false;
   m_bnDeviceAuthoritative = true;
   return true;
  }
//+------------------------------------------------------------------+
//| SELF-CHECK: forward. Runs the HOST transform first (on the same   |
//| pre-state the device holds), dispatches the kernel, then compares |
//| output + statistics elementwise. Pass -> the device is verified   |
//| and becomes authoritative. Fail -> the host result is restored to |
//| both copies and the kernels latch off. Cost: one extra input read |
//| and one output+options read, ONCE per layer per process.          |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::SelfCheckBnForward(CNeuronBaseOCL *NeuronOCL)
  {
   int units = Neurons();
   double inputs[];
   int count = NeuronOCL.getOutputVal(inputs);
   if(count <= 0)
      return false;
   //--- Host reference. NormalizeHost mutates the host mirrors (Output, BatchOptions, iSamplesSeen);
   //--- the device still holds the untouched pre-state, which is exactly what the kernel must see.
   if(!NormalizeHost(inputs, count))
      return false;
   int oTotal = BatchOptions.Total();
   double refY[], refOpt[];
   ArrayResize(refY, units);
   ArrayResize(refOpt, oTotal);
   for(int i = 0; i < units; i++)
      refY[i] = Output.At(i);
   for(int i = 0; i < oTotal; i++)
      refOpt[i] = BatchOptions.At(i);
   //--- Kernel on the same sample. NormalizeHost already advanced iSamplesSeen, so the ramped window
   //--- it used is exactly MathMax(1, iSamplesSeen) now.
   bool ok = DispatchBnForward(NeuronOCL, (double)MathMax(1, iSamplesSeen));
   if(ok)
      ok = Output.BufferRead() && BatchOptions.BufferRead();
   double worst = 0.0;
   int worstAt = -1;
   if(ok)
     {
      for(int i = 0; i < units; i++)
        {
         double s = BnDiffScore(refY[i], Output.At(i));
         if(s > worst) { worst = s; worstAt = i; }
        }
      for(int i = 0; i < oTotal; i++)
        {
         double s = BnDiffScore(refOpt[i], BatchOptions.At(i));
         if(s > worst) { worst = s; worstAt = units + i; }
        }
     }
   if(ok && worst <= 1.0)
     {
      m_bnCheckedFwd = true;
      m_bnDeviceAuthoritative = true;
      Print(__FUNCTION__ + StringFormat(": BatchNormForward kernel VERIFIED against the host math on "
                                        "%d units (worst normalized diff %.2e) - this layer's forward "
                                        "pass now runs device-side.", units, worst));
      return true;
     }
   //--- Kernel wrong or unreachable: the host result is the answer. Put it back in both copies.
   for(int i = 0; i < units; i++)
      Output.Update(i, refY[i]);
   for(int i = 0; i < oTotal; i++)
      BatchOptions.Update(i, refOpt[i]);
   if(BatchOptions.GetIndex() >= 0)
      BatchOptions.BufferWrite();
   m_bnDeviceAuthoritative = false;
   LatchBnKernelsOff(ok ? StringFormat("BatchNormForward disagrees with the host math (worst "
                                       "normalized diff %.2e at element %d)", worst, worstAt)
                        : "BatchNormForward dispatch/readback failed (error " +
                          IntegerToString(GetLastError()) + ")");
   return Output.BufferWrite();
  }
//+------------------------------------------------------------------+
//| SELF-CHECK: backward. Same pattern; the reference is              |
//| HiddenGradHost, which IS the host path's own loop.                |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::SelfCheckBnHiddenGrad(CNeuronBaseOCL *NeuronOCL)
  {
   int units = Neurons();
   SyncOptionsToHost(); // the host reference must see the statistics the kernels have been updating
   double grad[];
   if(getGradient(grad) < units)
      return false;
   double prevOut[];
   if(NeuronOCL.getOutputVal(prevOut) < units)
      return false;
   double ig[];
   if(ArrayResize(ig, units) != units)
      return false;
   HiddenGradHost(grad, prevOut, NeuronOCL.Activation(), ig, units);
   bool ok = DispatchBnHiddenGrad(NeuronOCL);
   double got[];
   if(ok)
      ok = (NeuronOCL.getGradient(got) >= units);
   double worst = 0.0;
   int worstAt = -1;
   if(ok)
      for(int i = 0; i < units; i++)
        {
         double s = BnDiffScore(ig[i], got[i]);
         if(s > worst) { worst = s; worstAt = i; }
        }
   if(ok && worst <= 1.0)
     {
      m_bnCheckedGrad = true;
      Print(__FUNCTION__ + StringFormat(": BatchNormHiddenGrad kernel VERIFIED against the host math "
                                        "on %d units (worst normalized diff %.2e).", units, worst));
      return true;
     }
   LatchBnKernelsOff(ok ? StringFormat("BatchNormHiddenGrad disagrees with the host math (worst "
                                       "normalized diff %.2e at unit %d)", worst, worstAt)
                        : "BatchNormHiddenGrad dispatch/readback failed (error " +
                          IntegerToString(GetLastError()) + ")");
   //--- The host result is the answer either way; setGradient writes host and device copies.
   return NeuronOCL.setGradient(ig);
  }
//+------------------------------------------------------------------+
//| SELF-CHECK: gamma/beta accumulate. First use ever, so the device  |
//| accumulator holds the zeros it was created with.                  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::SelfCheckBnAccum(void)
  {
   int units = Neurons();
   SyncOptionsToHost(); // NX is device-fresh in kernel mode
   double grad[];
   if(getGradient(grad) < units)
      return false;
   double preAcc[];
   if(m_bnAcc.GetIndex() >= 0 && !m_bnAcc.BufferRead())
     {
      //--- Cannot even read the accumulator: latch (which drains whatever earlier samples the kernels
      //--- put there) before reporting the failure, so the batch is not silently truncated.
      LatchBnKernelsOff("BatchNormAccumGammaBeta pre-check read failed (error " +
                        IntegerToString(GetLastError()) + ")");
      return false;
     }
   ArrayResize(preAcc, m_bnAcc.Total());
   for(int i = 0; i < m_bnAcc.Total(); i++)
      preAcc[i] = m_bnAcc.At(i);
   bool ok = DispatchBnAccum();
   if(ok)
      ok = m_bnAcc.BufferRead();
   double worst = 0.0;
   int worstAt = -1;
   if(ok)
      for(int i = 0; i < units; i++)
        {
         double expG = preAcc[2 * i] + grad[i] * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX);
         double expB = preAcc[2 * i + 1] + grad[i];
         double s = MathMax(BnDiffScore(expG, m_bnAcc.At(2 * i)), BnDiffScore(expB, m_bnAcc.At(2 * i + 1)));
         if(s > worst) { worst = s; worstAt = i; }
        }
   if(ok && worst <= 1.0)
     {
      m_bnCheckedAccum = true;
      Print(__FUNCTION__ + StringFormat(": BatchNormAccumGammaBeta kernel VERIFIED against the host "
                                        "math on %d units (worst normalized diff %.2e).", units, worst));
      return true;
     }
   //--- Restore the truth (pre-state plus this sample's host-computed contribution) into the device
   //--- accumulator BEFORE latching, so the drain inside the latch hands the host arrays exactly the
   //--- right sums.
   for(int i = 0; i < units; i++)
     {
      m_bnAcc.Update(2 * i, preAcc[2 * i] + grad[i] * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX));
      m_bnAcc.Update(2 * i + 1, preAcc[2 * i + 1] + grad[i]);
     }
   if(m_bnAcc.GetIndex() >= 0)
      m_bnAcc.BufferWrite();
   LatchBnKernelsOff(ok ? StringFormat("BatchNormAccumGammaBeta disagrees with the host math (worst "
                                       "normalized diff %.2e at unit %d)", worst, worstAt)
                        : "BatchNormAccumGammaBeta dispatch/readback failed (error " +
                          IntegerToString(GetLastError()) + ")");
   return true;
  }
//+------------------------------------------------------------------+
//| SELF-CHECK: gamma/beta apply. The host reference is StepGammaBeta |
//| itself, run on the synced host mirror - literally the code the    |
//| host path executes, so the comparison cannot drift from it.       |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::SelfCheckBnApply(double scale, double lt)
  {
   int units = Neurons();
   SyncOptionsToHost();
   if(m_bnAcc.GetIndex() >= 0 && !m_bnAcc.BufferRead())
     {
      //--- Same reasoning as the accumulate pre-check: latch drains the device sums into the host
      //--- arrays, so the caller's host fallback steps the REAL batch rather than an empty one.
      LatchBnKernelsOff("BatchNormApplyGammaBeta pre-check read failed (error " +
                        IntegerToString(GetLastError()) + ")");
      return false;
     }
   //--- Host reference: run the real host step on the host mirror (currently the pre-state).
   double accG[], accB[];
   ArrayResize(accG, units);
   ArrayResize(accB, units);
   for(int i = 0; i < units; i++)
     {
      accG[i] = m_bnAcc.At(2 * i);
      accB[i] = m_bnAcc.At(2 * i + 1);
      if(!StepGammaBeta(i * BN_OPT_STRIDE, accG[i] * scale, accB[i] * scale, lt))
         return false;
     }
   int oTotal = BatchOptions.Total();
   double refOpt[];
   ArrayResize(refOpt, oTotal);
   for(int i = 0; i < oTotal; i++)
      refOpt[i] = BatchOptions.At(i);
   bool ok = DispatchBnApply(scale, lt);
   if(ok)
      ok = BatchOptions.BufferRead() && m_bnAcc.BufferRead();
   double worst = 0.0;
   int worstAt = -1;
   if(ok)
     {
      for(int i = 0; i < oTotal; i++)
        {
         double s = BnDiffScore(refOpt[i], BatchOptions.At(i));
         if(s > worst) { worst = s; worstAt = i; }
        }
      //--- and the kernel must have zeroed the accumulator
      for(int i = 0; i < m_bnAcc.Total(); i++)
         if(MathAbs(m_bnAcc.At(i)) > 1.0e-12)
           { worst = DBL_MAX; worstAt = oTotal + i; break; }
     }
   if(ok && worst <= 1.0)
     {
      m_bnCheckedApply = true;
      m_bnDeviceAuthoritative = true;
      Print(__FUNCTION__ + StringFormat(": BatchNormApplyGammaBeta kernel VERIFIED against the host "
                                        "math on %d units (worst normalized diff %.2e).", units, worst));
      return true;
     }
   //--- The host step already produced the correct post-state in the host mirror - push it, zero the
   //--- accumulator everywhere, and latch.
   for(int i = 0; i < oTotal; i++)
      BatchOptions.Update(i, refOpt[i]);
   if(BatchOptions.GetIndex() >= 0)
      BatchOptions.BufferWrite();
   ZeroOptimizerBuffer(m_bnAcc);
   m_bnDeviceAuthoritative = false;
   LatchBnKernelsOff(ok ? StringFormat("BatchNormApplyGammaBeta disagrees with the host math (worst "
                                       "normalized diff %.2e at slot %d)", worst, worstAt)
                        : "BatchNormApplyGammaBeta dispatch/readback failed (error " +
                          IntegerToString(GetLastError()) + ")");
   return true;
  }
//+------------------------------------------------------------------+
//| Backward. Called by the layer BELOW this one (inverted-call       |
//| convention, same as Conv/Pool/LSTM): consumes this layer's own    |
//| Gradient (dL/dy, already filled by calcHiddenGradients against    |
//| the layer above) and writes dL/dx into NeuronOCL's Gradient.      |
//|                                                                   |
//| dL/dx = gamma/sd * dL/dy.                                         |
//|                                                                   |
//| The mean/variance are treated as CONSTANTS with respect to the    |
//| current sample. That is exact for the running-statistics form     |
//| this layer implements, and it is also what batch norm does at     |
//| inference time. The reference's kernel additionally carries two   |
//| correction terms for the current sample's own contribution to the |
//| statistics; both are O(1/batch) and vanish at the window lengths  |
//| used here, and its variance term (mean*x / 2*var^1.5) does not    |
//| follow from the derivative in the paper - so they are deliberately|
//| NOT copied. Numerical robustness matters far more in this engine  |
//| than a correction that is already below the noise floor.          |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::calcInputGradients(CNeuronBaseOCL *NeuronOCL)
  {
   if(CheckPointer(NeuronOCL) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID)
      return false;
   int units = Neurons();
   int inputsCount = NeuronOCL.Neurons();
   if(units <= 0 || inputsCount != units)
      return false; // batch norm is elementwise - a size mismatch means the topology is wrong
   //--- DEVICE PATH: the gradient, the previous output and the statistics are all already on the
   //--- device; the result lands in the previous layer's device gradient where its own backward
   //--- kernels read it. Zero transfers.
   if(BnDeviceEligible() && NeuronOCL.getOutputIndex() >= 0 && NeuronOCL.getGradientIndex() >= 0 &&
      getGradientIndex() >= 0 && EnsureBnDeviceBuffers())
     {
      if(!m_bnCheckedGrad)
         return SelfCheckBnHiddenGrad(NeuronOCL);
      if(DispatchBnHiddenGrad(NeuronOCL))
         return true;
      LatchBnKernelsOff("BatchNormHiddenGrad dispatch failed (error " +
                        IntegerToString(GetLastError()) + ")");
     }
   //--- HOST PATH.
   EnsureHostAuthoritative();
   double grad[];
   int count = getGradient(grad);
   //--- Gradient is allocated with one slot more than Neurons() (see CNeuronBaseOCL::Init); only the
   //--- real units carry a value.
   if(count < units)
      return false;
   //--- Armed for the weight-update pass, which reads this same Gradient again - see the declaration.
   if(ArrayCopy(m_gradCache, grad, 0, 0, count) == count)
      m_gradCached = true;
   //--- The previous layer's Output, from this sample's forward pass rather than a second read of the
   //--- same device buffer. Falls back to reading whenever the cache is not armed or does not match.
   double prevOut[];
   if(m_fwdInputCached && ArraySize(m_fwdInputCache) >= inputsCount)
     {
      if(ArrayCopy(prevOut, m_fwdInputCache, 0, 0, inputsCount) != inputsCount)
         return false;
     }
   else
      if(NeuronOCL.getOutputVal(prevOut) < inputsCount)
         return false;
   double ig[];
   if(ArrayResize(ig, inputsCount) != inputsCount)
      return false;
   HiddenGradHost(grad, prevOut, NeuronOCL.Activation(), ig, inputsCount);
   return NeuronOCL.setGradient(ig);
  }
//+------------------------------------------------------------------+
//| The elementwise backward math, factored out of calcInputGradients |
//| (2026-08-09) so the kernel self-check compares against LITERALLY  |
//| the code the host path runs, not a second copy that could drift.  |
//+------------------------------------------------------------------+
void CNeuronBatchNormOCL::HiddenGradHost(const double &grad[], const double &prevOut[],
                                         ENUM_ACTIVATION act, double &ig[], int n)
  {
   for(int i = 0; i < n; i++)
     {
      int shift = i * BN_OPT_STRIDE;
      double g = grad[i];
      if(iBatchSize > 1)
        {
         double sd = MathMax(MathSqrt(BatchOptions.At(shift + BN_OPT_VAR) + BN_EPSILON), BN_MIN_STD);
         g = g * BatchOptions.At(shift + BN_OPT_GAMMA) / sd;
        }
      //--- Then the previous layer's own activation derivative, byte-for-byte the same treatment
      //--- Network.cl's CaclHiddenGradient applies - including the clamp-to-range "implied target"
      //--- reformulation - so that from the previous layer's point of view a batch-norm layer is
      //--- indistinguishable from any other. NONE falls through unscaled.
      double out = prevOut[i];
      switch(act)
        {
         case TANH:
            g = MathMax(-1.0, MathMin(1.0, g + out)) - out;
            g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, 1.0 - out * out);
            break;
         case SIGMOID:
            g = MathMax(0.0, MathMin(1.0, g + out)) - out;
            g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, out * (1.0 - out));
            break;
         case PRELU:
            g = g * (out >= 0 ? 1.0 : 0.01);
            break;
         default:
            break;
        }
      ig[i] = g;
     }
  }
//+------------------------------------------------------------------+
//| gamma/beta update. NeuronOCL is the previous layer and is         |
//| deliberately unused: batch norm consumes its output elementwise,  |
//| so there is no incoming weight matrix to update (the CNet         |
//| constructor gives the previous layer 0 outgoing weights when this |
//| layer follows it).                                                |
//|                                                                   |
//| NO WEIGHT DECAY on gamma/beta - unlike every other parameter in   |
//| this engine, and on purpose. Decaying gamma toward zero shrinks   |
//| the layer's output toward the constant beta, which is the exact   |
//| pathology this layer was added to stop; excluding normalization   |
//| parameters from weight decay is standard practice for the same    |
//| reason.                                                           |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::updateInputWeights(CNeuronBaseOCL *NeuronOCL)
  {
   if(CheckPointer(BatchOptions) == POINTER_INVALID)
      return false;
   if(iBatchSize <= 1)
      return true; // normalization off - gamma/beta are not in the graph
   int units = Neurons();
   //--- This per-sample path stays HOST-side even in kernel mode, deliberately: it only runs when the
   //--- net's train batch is 1 (online learning on a deployed model - once per confirmed bar, not per
   //--- training sample), so kernelizing it buys nothing, and the host step is the reference
   //--- implementation. What kernel mode DOES require is the sync bracket: pull the device statistics
   //--- before stepping (NX and the running stats are device-fresh), push the stepped gamma/beta back
   //--- after, so the next forward kernel sees them.
   EnsureHostAuthoritative();
   //--- Gradient from calcInputGradients' read this same sample, not a second device round trip.
   double grad[];
   int count = m_gradCached ? ArrayCopy(grad, m_gradCache) : getGradient(grad);
   if(count < units || units <= 0)
      return false;
   //--- Same bias-corrected step size the dense Adam path computes, so gamma/beta move on the same
   //--- schedule as the weights around them.
   double lt = (optimization == SGD) ? 0.0 : eta * sqrt(1 - pow(b2, t)) / (1 - pow(b1, t));
   for(int i = 0; i < units; i++)
     {
      int shift = i * BN_OPT_STRIDE;
      double g = grad[i];
      //--- dL/dgamma = dL/dy * nx  |  dL/dbeta = dL/dy
      double gGamma = g * BatchOptions.At(shift + BN_OPT_NX);
      double gBeta  = g;
      if(!StepGammaBeta(shift, gGamma, gBeta, lt))
         return false;
     }
   if(optimization != SGD && t < INT_MAX)
      t++;
   //--- Keep the device copy coherent so a later kernel forward reads the stepped gamma/beta.
   if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite())
      return false;
   return true;
  }
//+------------------------------------------------------------------+
//| ONE unit's gamma/beta optimizer step, factored out of             |
//| updateInputWeights so the mini-batch path                         |
//| (ApplyAccumulatedGradients) takes the identical step on the batch |
//| mean instead of carrying a second copy of this arithmetic.        |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::StepGammaBeta(int shift, double gGamma, double gBeta, double lt)
  {
     {
      double gamma  = BatchOptions.At(shift + BN_OPT_GAMMA);
      double beta   = BatchOptions.At(shift + BN_OPT_BETA);
      //--- Matches NormalizeHost's self-heal, and this is the copy that makes it STICK: the clamped
      //--- write at the end of this loop persists the repaired value, whereas the forward pass only
      //--- substitutes one locally. Without this a model that loaded a NaN gamma would normalize
      //--- correctly but never train that unit's scale again, since NaN + anything stays NaN.
      if(!MathIsValidNumber(gamma))
         gamma = 1.0;
      if(!MathIsValidNumber(beta))
         beta = 0.0;
      double dGamma = 0.0, dBeta = 0.0;
      if(optimization == SGD)
        {
         dGamma = eta * gGamma + alpha * BatchOptions.At(shift + BN_OPT_MG);
         dBeta  = eta * gBeta  + alpha * BatchOptions.At(shift + BN_OPT_MB);
         if(!BatchOptions.Update(shift + BN_OPT_MG, dGamma) ||
            !BatchOptions.Update(shift + BN_OPT_MB, dBeta))
            return false;
        }
      else
        {
         //--- Second momentum is stored ALREADY square-rooted so it can be used as the denominator
         //--- directly, the same convention this engine's UpdateWeightsAdam kernels use
         //--- (matrix_v[wi] = sqrt(...)).
         //--- This layer squares the stored value back before re-entering the recursion, which is the
         //--- actual Adam second moment. RECONCILED 2026-08-09: the weight kernels used to feed the
         //--- stored sqrt straight back in as `v` - mixing a standard deviation with a variance - and
         //--- that recursion has a fixed point at v ~= b2 for any |g| below 1, so its denominator
         //--- stopped tracking the gradient scale and Adam degraded to plain SGD. It was left alone
         //--- here on the grounds that fixing it would change every existing model on all four
         //--- backends; it changed them for the better, and the reason this layer kept training while
         //--- the conv/LSTM stages behind it froze was precisely that gamma/beta got it right. All
         //--- four tiers now match this form - see AI\Network.cl's UpdateWeightsAdam.
         double mg = b1 * BatchOptions.At(shift + BN_OPT_MG) + (1 - b1) * gGamma;
         double mb = b1 * BatchOptions.At(shift + BN_OPT_MB) + (1 - b1) * gBeta;
         double vg = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VG), 2) + (1 - b2) * gGamma * gGamma);
         double vb = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VB), 2) + (1 - b2) * gBeta * gBeta);
         dGamma = lt * mg / (vg > 0 ? vg : lt * 10);
         dBeta  = lt * mb / (vb > 0 ? vb : lt * 10);
         if(!BatchOptions.Update(shift + BN_OPT_MG, mg) ||
            !BatchOptions.Update(shift + BN_OPT_MB, mb) ||
            !BatchOptions.Update(shift + BN_OPT_VG, vg) ||
            !BatchOptions.Update(shift + BN_OPT_VB, vb))
            return false;
        }
      dGamma = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dGamma));
      dBeta  = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dBeta));
      //--- `return true` and not `continue`: this is one unit's step now that the loop lives in the
      //--- caller, and a non-finite delta means SKIP this unit, never fail the layer.
      if(!MathIsValidNumber(dGamma) || !MathIsValidNumber(dBeta))
         return true;
      if(!BatchOptions.Update(shift + BN_OPT_GAMMA, MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, gamma + dGamma))) ||
         !BatchOptions.Update(shift + BN_OPT_BETA,  MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, beta + dBeta))))
         return false;
     }
   return true;
  }
//+------------------------------------------------------------------+
//| MINI-BATCH (2026-08-09 audit, F4). gamma/beta are host-side       |
//| parameters, so their accumulation is a plain host sum - no kernel |
//| and no extra device buffer. The per-sample normalized input       |
//| (BN_OPT_NX) is overwritten by each forward pass, which is exactly  |
//| why dL/dgamma has to be formed HERE, per sample, rather than      |
//| reconstructed at the end of the batch.                            |
//| Held in plain host arrays rather than new BatchOptions slots on    |
//| purpose: BN_OPT_STRIDE is baked into every persisted .nnw through  |
//| getWeightsBN, so widening it would invalidate every saved model.   |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::accumulateInputWeightGrads(CNeuronBaseOCL *NeuronOCL)
  {
   //--- The outgoing dense matrix is accumulated by the layer above, exactly as for a plain dense
   //--- neuron; this adds only the normalization parameters' own gradients.
   if(CheckPointer(BatchOptions) == POINTER_INVALID || iBatchSize <= 1)
      return true;   // normalization off - gamma/beta are not in the graph
   int units = Neurons();
   //--- DEVICE PATH: the gradient and NX are both device-fresh, so the accumulation happens where
   //--- they already live. One dispatch, no transfers.
   if(BnDeviceEligible() && getGradientIndex() >= 0 && EnsureBnDeviceBuffers())
     {
      if(!m_bnCheckedAccum)
         return SelfCheckBnAccum();
      if(DispatchBnAccum())
         return true;
      LatchBnKernelsOff("BatchNormAccumGammaBeta dispatch failed (error " +
                        IntegerToString(GetLastError()) + ")");
      //--- The latch drained whatever the kernels had accumulated this batch into m_accGamma/
      //--- m_accBeta; the host code below adds THIS sample on top, so nothing is lost or doubled.
     }
   EnsureHostAuthoritative();
   //--- Gradient from calcInputGradients' read this same sample - see the declaration comment. This is
   //--- the batched twin of updateInputWeights above and takes the value from the same place.
   double grad[];
   int count = m_gradCached ? ArrayCopy(grad, m_gradCache) : getGradient(grad);
   if(count < units || units <= 0)
      return false;
   if(ArraySize(m_accGamma) != units || ArraySize(m_accBeta) != units)
     {
      ArrayResize(m_accGamma, units);
      ArrayResize(m_accBeta, units);
      ArrayInitialize(m_accGamma, 0.0);
      ArrayInitialize(m_accBeta, 0.0);
     }
   for(int i = 0; i < units; i++)
     {
      double g = grad[i];
      m_accGamma[i] += g * BatchOptions.At(i * BN_OPT_STRIDE + BN_OPT_NX);
      m_accBeta[i]  += g;
     }
   return true;
  }
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::BeginGradAccum(void)
  {
   bool ok = CNeuronBaseOCL::BeginGradAccum();
   ArrayInitialize(m_accGamma, 0.0);
   ArrayInitialize(m_accBeta, 0.0);
   //--- Belt and braces for the device accumulator: the apply kernel zeroes it itself, but a batch
   //--- abandoned mid-way (era boundary, restore) must not leak its partial sums into the next one.
   if(CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0)
      ok = ZeroOptimizerBuffer(m_bnAcc) && ok;
   return ok;
  }
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::ApplyAccumulatedGradients(double scale)
  {
   //--- Outgoing dense matrix first, through the shared block optimizer.
   bool ok = ApplyAccumToBlock(Weights, GradAccum, FirstMomentum, SecondMomentum, DeltaWeights, scale);
   if(CheckPointer(BatchOptions) != POINTER_INVALID && iBatchSize > 1)
     {
      double lt = (optimization == SGD) ? 0.0 : eta * sqrt(1 - pow(b2, t)) / (1 - pow(b1, t));
      //--- DEVICE PATH: the batch was accumulated by the kernels, so it is stepped by the kernel too.
      //--- A dispatch failure latches (which drains the device sums into m_accGamma/m_accBeta) and
      //--- drops to the host loop below, so the batch is stepped exactly once either way.
      bool stepped = false;
      if(BnDeviceEligible() && CheckPointer(m_bnAcc) != POINTER_INVALID && m_bnAcc.GetIndex() >= 0)
        {
         if(!m_bnCheckedApply)
            stepped = SelfCheckBnApply(scale, lt);
         else
            if(DispatchBnApply(scale, lt))
               stepped = true;
            else
               LatchBnKernelsOff("BatchNormApplyGammaBeta dispatch failed (error " +
                                 IntegerToString(GetLastError()) + ")");
        }
      if(!stepped)
        {
         EnsureHostAuthoritative();
         int units = MathMin(Neurons(), MathMin(ArraySize(m_accGamma), ArraySize(m_accBeta)));
         for(int i = 0; i < units; i++)
            if(!StepGammaBeta(i * BN_OPT_STRIDE, m_accGamma[i] * scale, m_accBeta[i] * scale, lt))
              {
               ok = false;
               break;
              }
         ArrayInitialize(m_accGamma, 0.0);
         ArrayInitialize(m_accBeta, 0.0);
         //--- Keep a created-but-idle device copy coherent with the host step.
         if(BatchOptions.GetIndex() >= 0)
            ok = BatchOptions.BufferWrite() && ok;
        }
     }
   //--- One step, so t advances once - matching every other layer's batched apply.
   if(optimization != SGD && t < INT_MAX)
      t++;
   return ok;
  }
//+------------------------------------------------------------------+
//| Outgoing dense weight matrix followed by the whole BatchOptions   |
//| block, as one flat array - see the declaration comment.           |
//+------------------------------------------------------------------+
int CNeuronBatchNormOCL::getWeightsBN(double &values[])
  {
   //--- Checkpoints and the health report call this mid-training; in kernel mode the statistics live
   //--- on the device, so pull them first. Read-only - the device stays authoritative.
   SyncOptionsToHost();
   double w[];
   int wCount = getWeights(w);
   if(wCount < 0)
      wCount = 0;
   int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
   if(ArrayResize(values, wCount + oCount) != wCount + oCount)
      return 0;
   for(int i = 0; i < wCount; i++)
      values[i] = w[i];
   for(int i = 0; i < oCount; i++)
      values[wCount + i] = BatchOptions.At(i);
   return wCount + oCount;
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::setWeightsBN(double &values[])
  {
   int total = ArraySize(values);
   int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
   int wCount = total - oCount;
   if(wCount < 0)
      return false; // snapshot predates this layer's parameters - refuse rather than half-restore
   if(wCount > 0)
     {
      double w[];
      if(ArrayResize(w, wCount) != wCount)
         return false;
      for(int i = 0; i < wCount; i++)
         w[i] = values[i];
      if(!setWeights(w))
         return false;
     }
   for(int i = 0; i < oCount; i++)
      if(!BatchOptions.Update(i, values[wCount + i]))
         return false;
   //--- Full host overwrite of every slot: push it so a kernel-mode net keeps computing on the
   //--- RESTORED statistics rather than the diverged ones the device still holds. After this the two
   //--- copies are identical, so whichever side was authoritative remains consistent.
   if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite())
      return false;
   return true;
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Save(const int file_handle)
  {
   //--- The statistics ARE the layer's inference behaviour; in kernel mode their current values live
   //--- on the device, and a .nnw written from the stale host mirror would be a different model.
   SyncOptionsToHost();
   if(!CNeuronBaseOCL::Save(file_handle))
      return false;
   if(FileWriteInteger(file_handle, iBatchSize, INT_VALUE) < INT_VALUE)
      return false;
   if(FileWriteInteger(file_handle, iSamplesSeen, INT_VALUE) < INT_VALUE)
      return false;
   if(CheckPointer(BatchOptions) == POINTER_INVALID)
      return false;
   return BatchOptions.Save(file_handle);
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Load(const int file_handle)
  {
   if(!CNeuronBaseOCL::Load(file_handle))
      return false;
   iBatchSize = FileReadInteger(file_handle, INT_VALUE);
   iSamplesSeen = FileReadInteger(file_handle, INT_VALUE);
   if(CheckPointer(BatchOptions) == POINTER_INVALID)
     {
      BatchOptions = new CBufferDouble();
      if(CheckPointer(BatchOptions) == POINTER_INVALID)
         return false;
     }
   if(!BatchOptions.Load(file_handle))
      return false;
   //--- Loaded state is a full host overwrite - push it if a device copy already exists (a reload into
   //--- a live net), and the host is authoritative until the first kernel forward.
   if(BatchOptions.GetIndex() >= 0 && !BatchOptions.BufferWrite())
      return false;
   m_bnDeviceAuthoritative = false;
   //--- A model whose statistics block does not match its own width is not usable for inference -
   //--- fail loudly here rather than index past the end on the first forward pass.
   if(BatchOptions.Total() != Neurons() * BN_OPT_STRIDE)
     {
      Print(__FUNCTION__ + ": batch-norm parameter block is " + IntegerToString(BatchOptions.Total()) +
            " values but this layer has " + IntegerToString(Neurons()) + " units (expected " +
            IntegerToString(Neurons() * BN_OPT_STRIDE) + ") - file does not match the topology");
      return false;
     }
   return true;
  }
#endif // WARRIOR_NEURON_BATCHNORM_MQH
//+------------------------------------------------------------------+