4088850/Warrior_EA
Watch
1
0
Fork
You've already forked Warrior_EA
 0
forked from animatedread/Warrior_EA
Code Pull requests Activity
Warrior_EA/AI/NeuronBatchNorm.mqh
AnimateDread 65dc1bddb4 fix(ai): freeze batch-norm statistics when comparing two forward passes 
With normalization enabled a forward pass is not a pure function of its
input - it also advances the running mean/variance. ValidateCpuInference
compares the live backend net against a throwaway pure-MQL5 clone loaded
from the just-saved .nnw, so its own reference pass left the live model one
EMA step ahead of the file the clone reads. The check would then have been
measuring its own side effect, and a marginal result decides whether
buyers' backtests are allowed to run DLL-free.

Adds CNet::SetBatchNormFrozen / CNeuronBatchNormOCL::SetStatsFrozen -
classic batch-norm inference semantics, statistics used but not updated -
and freezes both sides for the duration of the comparison. Not persisted:
it is a transient evaluation mode, not model state. Default stays
adaptive, which is what the rest of the system (online continual learning)
is built around.

Compiles 0 errors, 0 warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:37:50 -04:00

560 lines
29 KiB
MQL5
Raw Permalink Blame History

//+------------------------------------------------------------------+
//| NeuronBatchNorm.mqh |
//| AnimateDread |
//| https://www.mql5.com |
//+------------------------------------------------------------------+
//| CNeuronBatchNormOCL - batch normalization (Ioffe & Szegedy 2015). |
//| Needs CNeuronBaseOCL (AI\Network.mqh) already declared; included |
//| from there, not standalone. |
//| |
//| WHY THIS EXISTS |
//| Every hidden stage in this project is unbounded (PRELU dense, |
//| PRELU conv) and the only bounded stage in the whole forward path |
//| was the SIGMOID classification head. That is a network with no |
//| internal scale control at all, and it showed: across four |
//| topologies on SP500 H1 the models peaked early then decayed |
//| monotonically into all-Neutral, and the decay ordered exactly by |
//| DEPTH - the shallow perceptron held ~52% balanced accuracy while |
//| the deepest (conv+pool+LSTM+dense taper) sat at the 33.3% |
//| one-class floor. That depth ordering is the signature of internal |
//| covariate shift, which is precisely what this layer addresses. |
//| |
//| Two further consequences worth spelling out, because they are the |
//| actual mechanism by which this is expected to help here: |
//| |
//| 1. It decouples weight decay from the learned function. With a |
//| normalized layer downstream, scaling the weights that feed it |
//| leaves the output unchanged - so WEIGHT_DECAY can no longer |
//| grind the discriminative signal away, it only rescales the |
//| effective learning rate (van Laarhoven 2017). The observed |
//| failure was exactly a slow monotonic shrink of the per-bar |
//| logit spread (0.45 -> 0.38 over ~200 eras) until the evidence |
//| tilt fell under the class-prior tilt and argmax degenerated |
//| to constant-Neutral. See WEIGHT_DECAY's comment in |
//| AI\Network.mqh, which describes that mechanism first-hand. |
//| 2. It is what would make an UNBOUNDED logit head viable. The |
//| 2026-07-27 attempt to run the head at NONE blew up (IS error |
//| 5.6e15) specifically because nothing upstream constrained |
//| scale. Do not retry that without a normalization layer |
//| immediately before the head - and then also drop |
//| CLASS_LOGIT_SCALE to 1.0 and BIAS_MAGNITUDE to ~0.5. |
//| |
//| IMPLEMENTATION SHAPE - deliberately host-side |
//| The math is elementwise O(n), not O(n^2) like a dense layer, so |
//| it runs in MQL5 against the host mirrors of the device buffers |
//| rather than as a fourth copy of a kernel in Network.cl + |
//| WarriorCPU.cpp + WarriorDML.cpp. That is a deliberate trade: |
//| - it works identically on all four compute tiers with no DLL |
//| rebuild and no risk of the backends drifting out of sync, |
//| which has been a recurring source of bugs in this engine; |
//| - it follows the precedent already set by the softmax+CCE |
//| output gradient and the per-sample loss weighting, both of |
//| which are computed in MQL5 for the same reason (see |
//| CNet::backPropOCL); |
//| - the cost is one buffer round-trip per BN layer per sample. |
//| On the CPU-DLL tier those buffers are host memory anyway, so |
//| it is a memcpy. On a real GPU it is a PCIe hop; if that ever |
//| matters, port these three methods to kernels - the math here |
//| is the specification. |
//| |
//| STATISTICS: exponential moving, not a stored mini-batch. |
//| Training here is pure online SGD - one weight update per sample |
//| (CNet::backProp per bar), never a batched pass - so there is no |
//| mini-batch to average over. iBatchSize is therefore an EMA WINDOW |
//| LENGTH, not a buffer size: mean and variance are updated per |
//| sample toward the last ~iBatchSize samples. Same formulation as |
//| the NeuroNet_DNG reference's BatchFeedForward kernel; the book |
//| (ch. 6.1.1) explicitly endorses the exponential form to avoid |
//| storing per-neuron history. |
//| |
//| The statistics keep adapting on EVERY forward pass, including |
//| out-of-sample scoring and live inference - they are never frozen |
//| the way classic batch norm freezes them at inference time. That |
//| is deliberate and matches the rest of this system (see |
//| OnlineLearnStep: the deployed model is designed to keep tracking |
//| the market). Two consequences worth knowing: OOS scoring lets the |
//| statistics see OOS activations - unsupervised, no label |
//| information, but not a hermetic holdout - and a restored |
//| checkpoint rewinds the statistics along with gamma/beta, since |
//| getWeightsBN/setWeightsBN carry both. |
//+------------------------------------------------------------------+
#ifndef WARRIOR_NEURON_BATCHNORM_MQH
#define WARRIOR_NEURON_BATCHNORM_MQH
//--- Per-neuron slot layout inside BatchOptions. A FIXED stride of 9 on every optimizer, unlike the
//--- reference's 7-or-9 split: the two unused doubles per neuron are noise next to a weight matrix,
//--- and a stride that changes with a persisted enum is an indexing bug waiting for the first time
//--- somebody switches TrainingOptimizer on an existing model.
#define BN_OPT_STRIDE 9
#define BN_OPT_MEAN 0 // running mean
#define BN_OPT_VAR 1 // running variance
#define BN_OPT_NX 2 // normalized input, cached from the forward pass for the backward pass
#define BN_OPT_GAMMA 3 // learned scale, init 1
#define BN_OPT_BETA 4 // learned shift, init 0
#define BN_OPT_MG 5 // gamma: Adam first momentum, or SGD previous delta
#define BN_OPT_MB 6 // beta: Adam first momentum, or SGD previous delta
#define BN_OPT_VG 7 // gamma: Adam second momentum (stored already square-rooted, as the
#define BN_OPT_VB 8 // beta: ... UpdateWeightsAdam kernels in this engine also do)
//--- Variance floor. Serves the same purpose as MIN_ACTIVATION_DERIVATIVE: a neuron whose input
//--- happens to be near-constant over the EMA window has a near-zero variance, and dividing by its
//--- square root turns rounding noise into an arbitrarily large activation that then propagates.
//--- Applied to the standard deviation (not the variance) so it reads as "no unit is amplified by
//--- more than 1e4", which is the property that actually matters.
#define BN_EPSILON 1.0e-10
#define BN_MIN_STD 1.0e-4
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
class CNeuronBatchNormOCL : public CNeuronBaseOCL
{
protected:
int iBatchSize; // EMA window length; <=1 disables normalization entirely
//--- When true the running statistics are USED but not UPDATED, i.e. classic batch-norm inference
//--- semantics. Off by default (see the note on adaptation in the class header). Exists because a
//--- forward pass is otherwise not a pure function of its input: two evaluations of the same bar
//--- return slightly different answers because the first one moved the statistics. Anything that
//--- COMPARES two forward passes has to freeze them first or it is measuring its own side effect -
//--- CExpertSignalAIBase::ValidateCpuInference() is exactly that, and would have silently failed
//--- its DLL-free check on nothing but the one EMA step it took itself.
bool bStatsFrozen;
//--- Number of samples seen. Used only to ramp the effective window up from 1 to iBatchSize over
//--- the first iBatchSize samples (standard EMA bias correction). Without it the running mean
//--- starts at 0 and approaches the true mean over ~iBatchSize samples, during which the layer
//--- emits near-zero for everything - a cold-start dead zone indistinguishable from the very
//--- collapse this layer exists to prevent.
int iSamplesSeen;
//--- Host-only: no kernel ever reads it, so it deliberately gets no device buffer. Persisted
//--- through Save/Load like any other parameter - gamma/beta are learned, and the running
//--- statistics ARE the layer's inference behaviour, so a model that loses them is not the model
//--- that was trained.
CArrayDouble *BatchOptions;
//---
virtual bool feedForward(CNeuronBaseOCL *NeuronOCL);
virtual bool feedForwardCPU(CNeuronBaseOCL *NeuronOCL);
virtual bool updateInputWeights(CNeuronBaseOCL *NeuronOCL);
//--- shared by feedForward/feedForwardCPU: the whole forward transform for one already-read input
//--- vector, writing straight into the host mirror of Output.
bool NormalizeHost(const double &inputs[], int count);
//--- ensures BatchOptions exists and is sized/seeded for `neurons` units
bool InitOptions(int neurons);
public:
CNeuronBatchNormOCL(void);
~CNeuronBatchNormOCL(void);
//---
virtual bool Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type);
virtual bool Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type);
virtual bool calcInputGradients(CNeuronBaseOCL *NeuronOCL);
//--- see bStatsFrozen. Deliberately NOT persisted: it is a transient evaluation mode, not model state.
void SetStatsFrozen(bool v) { bStatsFrozen = v; }
//--- Checkpoint/blend support. CNet::CaptureWeights/RestoreWeights/BlendWeightsFrom snapshot one
//--- flat array per neuron object, so gamma/beta/mean/variance are appended AFTER the outgoing
//--- dense weight matrix and split back out on restore. Without this the plateau ladder's
//--- "restore best checkpoint" would put the dense weights back while leaving this layer's
//--- parameters at whatever the diverged era left behind - a silently mismatched pair.
virtual int getWeightsBN(double &values[]);
virtual bool setWeightsBN(double &values[]);
//---
virtual bool Save(int const file_handle);
virtual bool Load(int const file_handle);
//---
virtual int Type(void) const { return defNeuronBatchNormOCL; }
};
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
CNeuronBatchNormOCL::CNeuronBatchNormOCL(void) : iBatchSize(1), bStatsFrozen(false), iSamplesSeen(0)
{
BatchOptions = NULL;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
CNeuronBatchNormOCL::~CNeuronBatchNormOCL(void)
{
if(CheckPointer(BatchOptions) != POINTER_INVALID)
delete BatchOptions;
}
//+------------------------------------------------------------------+
//| gamma=1 / beta=0 / zeroed statistics for every unit. |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::InitOptions(int neurons)
{
if(neurons <= 0)
return false;
if(CheckPointer(BatchOptions) == POINTER_INVALID)
{
BatchOptions = new CArrayDouble();
if(CheckPointer(BatchOptions) == POINTER_INVALID)
return false;
}
BatchOptions.Clear();
if(!BatchOptions.Reserve(neurons * BN_OPT_STRIDE))
return false;
for(int n = 0; n < neurons; n++)
for(int s = 0; s < BN_OPT_STRIDE; s++)
if(!BatchOptions.Add(s == BN_OPT_GAMMA ? 1.0 : 0.0))
return false;
iSamplesSeen = 0;
return true;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type)
{
if(!CNeuronBaseOCL::Init(numOutputs, myIndex, open_cl, numNeurons, optimization_type))
return false;
//--- Identity forward transform. The activation belongs to the layer AFTER this one; normalizing
//--- and then squashing in the same step would defeat the point (Ioffe & Szegedy place the
//--- normalization immediately BEFORE the non-linearity, not around it).
activation = NONE;
iBatchSize = (int)batchSize;
return InitOptions((int)numNeurons);
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Init(uint numOutputs, uint myIndex, CDirectMLMy *direct_ml, uint numNeurons, uint batchSize, ENUM_OPTIMIZATION optimization_type)
{
if(!CNeuronBaseOCL::Init(numOutputs, myIndex, direct_ml, numNeurons, optimization_type))
return false;
activation = NONE;
iBatchSize = (int)batchSize;
return InitOptions((int)numNeurons);
}
//+------------------------------------------------------------------+
//| The forward transform, host-side. Mirrors the NeuroNet_DNG |
//| BatchFeedForward kernel, plus the bias-corrected warm-up window. |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::NormalizeHost(const double &inputs[], int count)
{
if(CheckPointer(Output) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID)
return false;
if(count > Output.Total())
count = Output.Total();
if(BatchOptions.Total() < count * BN_OPT_STRIDE)
return false;
//--- A window of 1 makes mean==x and variance==0 for every sample, i.e. a constant-zero output.
//--- Treat it as "normalization off" and pass the input through untouched rather than emit zeros.
if(iBatchSize <= 1)
{
for(int i = 0; i < count; i++)
if(!Output.Update(i, inputs[i]))
return false;
return true;
}
if(!bStatsFrozen && iSamplesSeen < iBatchSize)
iSamplesSeen++;
//--- Effective window: ramps 1,2,3... up to iBatchSize, so the first samples produce an honest
//--- running mean instead of one biased toward the zero initializer.
double w = (double)MathMax(1, iSamplesSeen);
for(int i = 0; i < count; i++)
{
int shift = i * BN_OPT_STRIDE;
double x = inputs[i];
double mean = BatchOptions.At(shift + BN_OPT_MEAN);
double variance = BatchOptions.At(shift + BN_OPT_VAR);
if(!bStatsFrozen)
{
mean = (mean * (w - 1.0) + x) / w;
variance = (variance * (w - 1.0) + (x - mean) * (x - mean)) / w;
}
double delta = x - mean;
double sd = MathMax(MathSqrt(variance + BN_EPSILON), BN_MIN_STD);
double nx = delta / sd;
double y = BatchOptions.At(shift + BN_OPT_GAMMA) * nx + BatchOptions.At(shift + BN_OPT_BETA);
//--- nx is still cached when frozen: it costs nothing and keeps the buffer consistent with the
//--- output just produced. There is no backward pass while frozen, so nothing reads it.
if(!bStatsFrozen &&
(!BatchOptions.Update(shift + BN_OPT_MEAN, mean) ||
!BatchOptions.Update(shift + BN_OPT_VAR, variance)))
return false;
if(!BatchOptions.Update(shift + BN_OPT_NX, nx))
return false;
if(!Output.Update(i, y))
return false;
}
return true;
}
//+------------------------------------------------------------------+
//| Device path: pull the previous layer's output to the host, run |
//| the transform, push our output back. See the file header for why |
//| this is not a kernel. |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::feedForward(CNeuronBaseOCL *NeuronOCL)
{
if(CheckPointer(NeuronOCL) == POINTER_INVALID)
return false;
double inputs[];
int count = NeuronOCL.getOutputVal(inputs);
if(count <= 0)
return false;
if(!NormalizeHost(inputs, count))
return false;
//--- Unlike the dense/conv kernels this layer's Output does NOT stay device-resident by itself -
//--- it was just written host-side, so it has to be pushed before the next layer's kernel reads
//--- it through getOutputIndex().
return Output.BufferWrite();
}
//+------------------------------------------------------------------+
//| Pure-MQL5 inference path (no backend at all): the previous |
//| layer's values live only in its host mirror, so read them there. |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::feedForwardCPU(CNeuronBaseOCL *NeuronOCL)
{
if(CheckPointer(NeuronOCL) == POINTER_INVALID)
return false;
int count = NeuronOCL.Neurons();
if(count <= 0)
return false;
double inputs[];
if(ArrayResize(inputs, count) != count)
return false;
for(int i = 0; i < count; i++)
inputs[i] = NeuronOCL.OutputHost(i);
//--- No BufferWrite: there is no device buffer on this path, and Output's host mirror is what
//--- the next layer's feedForwardCPU (and GetOutputsCPU) read.
return NormalizeHost(inputs, count);
}
//+------------------------------------------------------------------+
//| Backward. Called by the layer BELOW this one (inverted-call |
//| convention, same as Conv/Pool/LSTM): consumes this layer's own |
//| Gradient (dL/dy, already filled by calcHiddenGradients against |
//| the layer above) and writes dL/dx into NeuronOCL's Gradient. |
//| |
//| dL/dx = gamma/sd * dL/dy. |
//| |
//| The mean/variance are treated as CONSTANTS with respect to the |
//| current sample. That is exact for the running-statistics form |
//| this layer implements, and it is also what batch norm does at |
//| inference time. The reference's kernel additionally carries two |
//| correction terms for the current sample's own contribution to the |
//| statistics; both are O(1/batch) and vanish at the window lengths |
//| used here, and its variance term (mean*x / 2*var^1.5) does not |
//| follow from the derivative in the paper - so they are deliberately|
//| NOT copied. Numerical robustness matters far more in this engine |
//| than a correction that is already below the noise floor. |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::calcInputGradients(CNeuronBaseOCL *NeuronOCL)
{
if(CheckPointer(NeuronOCL) == POINTER_INVALID || CheckPointer(BatchOptions) == POINTER_INVALID)
return false;
double grad[];
int count = getGradient(grad);
//--- Gradient is allocated with one slot more than Neurons() (see CNeuronBaseOCL::Init); only the
//--- real units carry a value.
int units = Neurons();
if(count < units || units <= 0)
return false;
int inputsCount = NeuronOCL.Neurons();
if(inputsCount != units)
return false; // batch norm is elementwise - a size mismatch means the topology is wrong
double prevOut[];
if(NeuronOCL.getOutputVal(prevOut) < inputsCount)
return false;
double ig[];
if(ArrayResize(ig, inputsCount) != inputsCount)
return false;
for(int i = 0; i < inputsCount; i++)
{
int shift = i * BN_OPT_STRIDE;
double g = grad[i];
if(iBatchSize > 1)
{
double sd = MathMax(MathSqrt(BatchOptions.At(shift + BN_OPT_VAR) + BN_EPSILON), BN_MIN_STD);
g = g * BatchOptions.At(shift + BN_OPT_GAMMA) / sd;
}
//--- Then the previous layer's own activation derivative, byte-for-byte the same treatment
//--- Network.cl's CaclHiddenGradient applies - including the clamp-to-range "implied target"
//--- reformulation - so that from the previous layer's point of view a batch-norm layer is
//--- indistinguishable from any other. NONE falls through unscaled.
double out = prevOut[i];
switch(NeuronOCL.Activation())
{
case TANH:
g = MathMax(-1.0, MathMin(1.0, g + out)) - out;
g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, 1.0 - out * out);
break;
case SIGMOID:
g = MathMax(0.0, MathMin(1.0, g + out)) - out;
g = g * MathMax(MIN_ACTIVATION_DERIVATIVE, out * (1.0 - out));
break;
case PRELU:
g = g * (out >= 0 ? 1.0 : 0.01);
break;
default:
break;
}
ig[i] = g;
}
return NeuronOCL.setGradient(ig);
}
//+------------------------------------------------------------------+
//| gamma/beta update. NeuronOCL is the previous layer and is |
//| deliberately unused: batch norm consumes its output elementwise, |
//| so there is no incoming weight matrix to update (the CNet |
//| constructor gives the previous layer 0 outgoing weights when this |
//| layer follows it). |
//| |
//| NO WEIGHT DECAY on gamma/beta - unlike every other parameter in |
//| this engine, and on purpose. Decaying gamma toward zero shrinks |
//| the layer's output toward the constant beta, which is the exact |
//| pathology this layer was added to stop; excluding normalization |
//| parameters from weight decay is standard practice for the same |
//| reason. |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::updateInputWeights(CNeuronBaseOCL *NeuronOCL)
{
if(CheckPointer(BatchOptions) == POINTER_INVALID)
return false;
if(iBatchSize <= 1)
return true; // normalization off - gamma/beta are not in the graph
double grad[];
int count = getGradient(grad);
int units = Neurons();
if(count < units || units <= 0)
return false;
//--- Same bias-corrected step size the dense Adam path computes, so gamma/beta move on the same
//--- schedule as the weights around them.
double lt = (optimization == SGD) ? 0.0 : eta * sqrt(1 - pow(b2, t)) / (1 - pow(b1, t));
for(int i = 0; i < units; i++)
{
int shift = i * BN_OPT_STRIDE;
double g = grad[i];
//--- dL/dgamma = dL/dy * nx | dL/dbeta = dL/dy
double gGamma = g * BatchOptions.At(shift + BN_OPT_NX);
double gBeta = g;
double gamma = BatchOptions.At(shift + BN_OPT_GAMMA);
double beta = BatchOptions.At(shift + BN_OPT_BETA);
double dGamma = 0.0, dBeta = 0.0;
if(optimization == SGD)
{
dGamma = eta * gGamma + alpha * BatchOptions.At(shift + BN_OPT_MG);
dBeta = eta * gBeta + alpha * BatchOptions.At(shift + BN_OPT_MB);
if(!BatchOptions.Update(shift + BN_OPT_MG, dGamma) ||
!BatchOptions.Update(shift + BN_OPT_MB, dBeta))
return false;
}
else
{
//--- Second momentum is stored ALREADY square-rooted so it can be used as the denominator
//--- directly, the same convention this engine's UpdateWeightsAdam kernels use
//--- (matrix_v[wi] = sqrt(...)).
//--- DELIBERATE DIFFERENCE from those kernels: they feed the stored value straight back in
//--- as `v`, i.e. sqrt(b2*sqrt(v_prev) + (1-b2)*g^2), which mixes a standard deviation with a
//--- variance and is not the Adam recursion. Squaring it back first (below) is. The two are
//--- not reconciled here because changing the weight kernels would alter the behaviour of
//--- every existing model on all four backends at once; new code gets it right.
double mg = b1 * BatchOptions.At(shift + BN_OPT_MG) + (1 - b1) * gGamma;
double mb = b1 * BatchOptions.At(shift + BN_OPT_MB) + (1 - b1) * gBeta;
double vg = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VG), 2) + (1 - b2) * gGamma * gGamma);
double vb = sqrt(b2 * pow(BatchOptions.At(shift + BN_OPT_VB), 2) + (1 - b2) * gBeta * gBeta);
dGamma = lt * mg / (vg > 0 ? vg : lt * 10);
dBeta = lt * mb / (vb > 0 ? vb : lt * 10);
if(!BatchOptions.Update(shift + BN_OPT_MG, mg) ||
!BatchOptions.Update(shift + BN_OPT_MB, mb) ||
!BatchOptions.Update(shift + BN_OPT_VG, vg) ||
!BatchOptions.Update(shift + BN_OPT_VB, vb))
return false;
}
dGamma = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dGamma));
dBeta = MathMax(-MAX_WEIGHT_DELTA, MathMin(MAX_WEIGHT_DELTA, dBeta));
if(!MathIsValidNumber(dGamma) || !MathIsValidNumber(dBeta))
continue;
if(!BatchOptions.Update(shift + BN_OPT_GAMMA, MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, gamma + dGamma))) ||
!BatchOptions.Update(shift + BN_OPT_BETA, MathMax(-MAX_WEIGHT, MathMin(MAX_WEIGHT, beta + dBeta))))
return false;
}
if(optimization != SGD && t < INT_MAX)
t++;
return true;
}
//+------------------------------------------------------------------+
//| Outgoing dense weight matrix followed by the whole BatchOptions |
//| block, as one flat array - see the declaration comment. |
//+------------------------------------------------------------------+
int CNeuronBatchNormOCL::getWeightsBN(double &values[])
{
double w[];
int wCount = getWeights(w);
if(wCount < 0)
wCount = 0;
int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
if(ArrayResize(values, wCount + oCount) != wCount + oCount)
return 0;
for(int i = 0; i < wCount; i++)
values[i] = w[i];
for(int i = 0; i < oCount; i++)
values[wCount + i] = BatchOptions.At(i);
return wCount + oCount;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::setWeightsBN(double &values[])
{
int total = ArraySize(values);
int oCount = (CheckPointer(BatchOptions) == POINTER_INVALID) ? 0 : BatchOptions.Total();
int wCount = total - oCount;
if(wCount < 0)
return false; // snapshot predates this layer's parameters - refuse rather than half-restore
if(wCount > 0)
{
double w[];
if(ArrayResize(w, wCount) != wCount)
return false;
for(int i = 0; i < wCount; i++)
w[i] = values[i];
if(!setWeights(w))
return false;
}
for(int i = 0; i < oCount; i++)
if(!BatchOptions.Update(i, values[wCount + i]))
return false;
return true;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Save(const int file_handle)
{
if(!CNeuronBaseOCL::Save(file_handle))
return false;
if(FileWriteInteger(file_handle, iBatchSize, INT_VALUE) < INT_VALUE)
return false;
if(FileWriteInteger(file_handle, iSamplesSeen, INT_VALUE) < INT_VALUE)
return false;
if(CheckPointer(BatchOptions) == POINTER_INVALID)
return false;
return BatchOptions.Save(file_handle);
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CNeuronBatchNormOCL::Load(const int file_handle)
{
if(!CNeuronBaseOCL::Load(file_handle))
return false;
iBatchSize = FileReadInteger(file_handle, INT_VALUE);
iSamplesSeen = FileReadInteger(file_handle, INT_VALUE);
if(CheckPointer(BatchOptions) == POINTER_INVALID)
{
BatchOptions = new CArrayDouble();
if(CheckPointer(BatchOptions) == POINTER_INVALID)
return false;
}
if(!BatchOptions.Load(file_handle))
return false;
//--- A model whose statistics block does not match its own width is not usable for inference -
//--- fail loudly here rather than index past the end on the first forward pass.
if(BatchOptions.Total() != Neurons() * BN_OPT_STRIDE)
{
Print(__FUNCTION__ + ": batch-norm parameter block is " + IntegerToString(BatchOptions.Total()) +
" values but this layer has " + IntegerToString(Neurons()) + " units (expected " +
IntegerToString(Neurons() * BN_OPT_STRIDE) + ") - file does not match the topology");
return false;
}
return true;
}
#endif // WARRIOR_NEURON_BATCHNORM_MQH
//+------------------------------------------------------------------+
View git blame Copy permalink