Commit graph

2 commits

Author SHA1 Message Date
AnimateDread
6a687cda41 feat: add SGD+momentum optimizer and input-driven hyperparameters
Replace hardcoded lr and momentum with new input variables for Adam and
SGD+momentum. Add OpenCL kernel LSTM_UpdateWeightsMomentum alongside the
existing Adam kernel. Update comments and revert beta1 to book default 0.9.
2026-07-18 14:56:41 -04:00
AnimateDread
8c66a0fb89 fix(cl): stabilize network training with tighter limits, weight decay, and sign-agreement gate
- Tighten MAX_WEIGHT from 1.0e6 to 100.0 to prevent unbounded weight growth.
- Add MIN_ACTIVATION_DERIVATIVE (1e-4) to avoid zero gradients for saturated units.
- Introduce WEIGHT_DECAY (0.01) to decouple decay from Adam updates.
- Add MAX_WEIGHT_DELTA (0.1) to clamp per-step Adam updates and prevent overshoot.
- Apply sign-agreement gate on weight updates in Adam kernel to only apply steps aligned with current gradient direction.
- Fix tanh/sigmoid derivative calculations to use the new floor instead of hard-coded edge-case values.
2026-07-15 21:47:37 -04:00