From 2e188b42eef59b5e52c8deb70f4c54dc78a1a627 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 1 Oct 2026 05:26:48 +0000 Subject: [PATCH 1/2] feat(optimization): add stateful SGD/momentum and Adam step optimisers Introduce a StepOptimizer interface with Step(theta, gradient) and Reset(), then implement SGD (vanilla, momentum, Nesterov) and Adam (bias-corrected first and second moments) as concrete float-only, no-heap step optimisers for on-device NN training. Closes #339 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01N4pngRiBqeb1xiseKXNLG6 --- README.md | 2 +- doc/optimization/README.md | 1 + doc/optimization/StepOptimizer.md | 117 ++++++++++++++++++ numerical/optimization/Adam.cpp | 6 + numerical/optimization/Adam.hpp | 92 ++++++++++++++ numerical/optimization/CMakeLists.txt | 5 + numerical/optimization/Sgd.cpp | 6 + numerical/optimization/Sgd.hpp | 70 +++++++++++ numerical/optimization/StepOptimizer.hpp | 20 +++ numerical/optimization/test/CMakeLists.txt | 1 + .../optimization/test/TestStepOptimizer.cpp | 94 ++++++++++++++ 11 files changed, 413 insertions(+), 1 deletion(-) create mode 100644 doc/optimization/StepOptimizer.md create mode 100644 numerical/optimization/Adam.cpp create mode 100644 numerical/optimization/Adam.hpp create mode 100644 numerical/optimization/Sgd.cpp create mode 100644 numerical/optimization/Sgd.hpp create mode 100644 numerical/optimization/StepOptimizer.hpp create mode 100644 numerical/optimization/test/TestStepOptimizer.cpp diff --git a/README.md b/README.md index b08d4d71..7020cd48 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ Refer to the documentation to quickly integrate and utilize the library's signal | [Controllers](doc/controllers/README.md) | Bang-Bang/Hysteresis, Deadbeat Control, PID, LQR, LQI (Integral/Servo State Feedback), MPC, Saturation, Rate Limiter, Slew-Limited Saturation, Feedforward/2-DOF, Gain-Scheduled Controller, Lead-Lag Compensator, Luenberger Observer | | [Estimators](doc/estimators/README.md) | Linear Regression, Polynomial Fitting, Total Least Squares, Yule-Walker (offline), Recursive Least Squares, LMS / NLMS Adaptive Filter (online), Consistency Metrics / NEES / NIS | | [Filters](doc/filters/README.md) | Kalman, Extended Kalman, Unscented Kalman, Square-Root Kalman, Alpha-Beta/Alpha-Beta-Gamma, FIR, IIR, Exponential Moving Average, Moving Average, Complementary, Median Filter, CIC (Cascaded Integrator-Comb), Notch/Comb Filter, Savitzky-Golay Filter, Biquad/Second-Order-Section Cascade, IIR Filter Design (Butterworth/Chebyshev-I + Bilinear Transform), Madgwick/Mahony AHRS | -| [Optimization](doc/optimization/README.md) | Gradient Descent | +| [Optimization](doc/optimization/README.md) | Gradient Descent, SGD / Adam Step Optimisers | | [Regularization](doc/regularization/README.md) | L1 (Lasso), L2 (Ridge) | | [Math](doc/math/README.md) | CORDIC, Quaternion, MatrixNorms, Step Response Metrics, MatrixExponential | | [Solvers](doc/solvers/README.md) | Gaussian Elimination, Levinson-Durbin, Durand-Kerner, Cholesky, DARE, Runge-Kutta ODE Integrators (RK4 + Dormand-Prince), Spectral Radius & Discrete Stability Margin, QR Decomposition (Householder / Givens), LU Decomposition with Partial Pivoting, Singular Value Decomposition (Golub-Kahan) | diff --git a/doc/optimization/README.md b/doc/optimization/README.md index a0872f03..dd6e435a 100644 --- a/doc/optimization/README.md +++ b/doc/optimization/README.md @@ -8,3 +8,4 @@ General-purpose optimization algorithms for parameter fitting, training, and con |--------------------------------------------------|---------------------------------------------------------------------------------------------| | [Bayesian Optimization](BayesianOptimization.md) | Gradient-free global optimization using Gaussian Process surrogate and Expected Improvement | | [Gradient Descent](Optimizer.md) | Iterative batch optimization via gradient-based weight updates | +| [SGD / Adam Step Optimisers](StepOptimizer.md) | Stateful per-step SGD (with optional momentum/Nesterov) and Adam with bias correction | diff --git a/doc/optimization/StepOptimizer.md b/doc/optimization/StepOptimizer.md new file mode 100644 index 00000000..8d77e65b --- /dev/null +++ b/doc/optimization/StepOptimizer.md @@ -0,0 +1,117 @@ +# Stateful Step Optimisers (SGD / Adam) + +## Overview & Motivation + +On-device neural network training and online system identification require an optimiser that accepts one externally computed gradient per call and maintains its own state between calls. Unlike batch optimisers that own the full objective, a **step optimiser** separates the gradient source from the update rule, enabling training loops where back-propagation runs elsewhere. + +Two industry-standard rules are provided: **Stochastic Gradient Descent** (SGD) with optional momentum or Nesterov lookahead, and the **Adam** (Adaptive Moment Estimation) optimiser with bias correction. + +## Mathematical Theory + +### SGD + +Vanilla SGD applies the raw gradient: + +$$\theta_{t+1} = \theta_t - \eta \, g_t$$ + +where $g_t = \nabla_\theta \mathcal{L}(\theta_t)$ and $\eta$ is the learning rate. + +**Momentum** accumulates a velocity $v$, smoothing oscillations and accelerating progress in low-curvature directions: + +$$v_{t+1} = \beta \, v_t + g_t, \qquad \theta_{t+1} = \theta_t - \eta \, v_{t+1}$$ + +with $\beta \in [0, 1)$ the momentum coefficient (typically $0.9$). + +**Nesterov momentum** incorporates a lookahead correction, replacing the plain velocity step with: + +$$\theta_{t+1} = \theta_t - \eta \, \bigl(g_t + \beta \, v_{t+1}\bigr)$$ + +This form evaluates the effective update at the anticipated next position, yielding faster convergence on smooth convex objectives. + +### Adam + +Adam maintains exponential moving averages of the gradient (first moment $m$) and the squared gradient (second moment $v$): + +$$m_{t+1} = \beta_1 \, m_t + (1 - \beta_1) \, g_t$$ + +$$v_{t+1} = \beta_2 \, v_t + (1 - \beta_2) \, g_t^2 \quad (\text{element-wise})$$ + +Both estimates are biased toward zero at initialisation. **Bias correction** removes this bias: + +$$\hat{m}_{t+1} = \frac{m_{t+1}}{1 - \beta_1^{t+1}}, \qquad \hat{v}_{t+1} = \frac{v_{t+1}}{1 - \beta_2^{t+1}}$$ + +The parameter update normalises the corrected first moment by the square root of the corrected second moment, providing a per-parameter adaptive step size: + +$$\theta_{t+1} = \theta_t - \eta \, \frac{\hat{m}_{t+1}}{\sqrt{\hat{v}_{t+1}} + \varepsilon}$$ + +Typical hyper-parameters: $\eta = 10^{-3}$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\varepsilon = 10^{-8}$. + +## Complexity Analysis + +| Algorithm | Time per step | Extra state (floats) | Notes | +|-------------|---------------|----------------------|--------------------------------------------| +| SGD | $O(N)$ | $N$ | One velocity vector (equals the gradient when momentum is 0) | +| Adam | $O(N)$ | $2N + 2$ | Two moment vectors plus the running powers $\beta_1^t$, $\beta_2^t$ | + +$N$ is the number of parameters. All operations are in-place; no heap allocation is required. + +## Step-by-Step Walkthrough + +**SGD with momentum** on $\mathcal{L}(\theta) = \frac{1}{2}\|\theta\|^2$, $N=1$, $\theta_0 = 1$, $\eta = 0.1$, $\beta = 0.9$: + +| $t$ | $g_t = \theta_t$ | $v_t = 0.9 v_{t-1} + g_t$ | $\theta_{t+1} = \theta_t - 0.1 v_t$ | +|-----|------------------|---------------------------|--------------------------------------| +| 1 | 1.000 | 1.000 | 0.900 | +| 2 | 0.900 | 1.800 | 0.720 | +| 3 | 0.720 | 2.340 | 0.486 | + +**Adam** on the same objective, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\varepsilon = 10^{-8}$, $\theta_0 = 0$, $g_1 = 1$: + +| Quantity | Value | +|---------------|-----------------| +| $m_1$ | $0.1$ | +| $v_1$ | $0.001$ | +| $\hat{m}_1$ | $1.0$ | +| $\hat{v}_1$ | $1.0$ | +| $\theta_1$ | $-\eta \approx -0.001$ | + +Bias correction is the critical step: without it, $m_1/\sqrt{v_1} \approx 3.16$, giving a first step roughly $\sqrt{1000}$ times larger than the bias-corrected value. + +## Pitfalls & Edge Cases + +- **Learning rate too large.** SGD without momentum diverges for $\eta \geq 2/L$ on $L$-smooth losses. Momentum reduces the effective stability bound further; reduce $\eta$ or $\beta$ if oscillation is observed. +- **Adam with small $\varepsilon$.** Setting $\varepsilon$ too small causes division by near-zero when a parameter has zero gradient history, producing numerical instability. The default $10^{-8}$ is sufficient for `float`. +- **Momentum at reset.** When `Reset()` is called, the velocity (SGD) or moment estimates (Adam) return to zero. The first step after a reset behaves identically to starting from scratch, so the bias-correction denominator for Adam also restarts from $t=1$. +- **Nesterov with large $\beta$.** The lookahead correction adds $\beta \, v_{t+1}$ to the update, which can overshoot on sparse or noisy gradients. Prefer plain momentum when the gradient signal is noisy. +- **Float precision.** For Adam, the accumulated second moment $v$ can underflow toward zero for very small gradients and single-precision arithmetic. Increasing $\varepsilon$ mitigates this at the cost of less adaptivity. + +## Variants & Generalizations + +| Variant | Change from base | +|-----------------------|-----------------------------------------------------------------------| +| AdaGrad | Non-decaying sum of squared gradients (no $\beta_2$ decay); aggressive learning rate shrinkage | +| RMSProp | Adam without first-moment tracking; lacks bias correction | +| AdamW | Decoupled weight-decay applied directly to $\theta$ before the gradient step | +| AMSGrad | Replaces $\hat{v}$ with the running maximum to guarantee monotone effective step-size | + +## Applications + +- **On-device neural network training** — Updating weights after each batch of sensor data. +- **Online system identification** — Fitting model parameters in real time as measurements arrive. +- **Adaptive control** — Adjusting gain schedules or feed-forward maps without offline re-training. +- **Sensor calibration** — Minimising residual error by incrementally fitting a polynomial or affine model. + +## Connections to Other Algorithms + +| Component | Relationship | +|-------------------------------------------------|--------------------------------------------------------------------------------------| +| [Gradient Descent](Optimizer.md) | Batch counterpart; shares the learning-rate update rule but recomputes the objective each time | +| [LMS Adaptive Filter](../estimators/README.md) | Equivalent to online SGD for a linear regression model under MSE loss | +| [Regularization](../regularization/README.md) | Adds a penalty gradient to $g_t$; compatible with any step optimiser | + +## References & Further Reading + +- Ruder, S., "An overview of gradient descent optimization algorithms", *arXiv:1609.04747*, 2016. +- Kingma, D.P. and Ba, J., "Adam: A Method for Stochastic Optimization", *ICLR*, 2015. +- Nesterov, Y., "A method for solving the convex programming problem with convergence rate $O(1/k^2)$", *Soviet Mathematics Doklady*, 1983. +- Goodfellow, I., Bengio, Y., and Courville, A., *Deep Learning*, Chapter 8, MIT Press, 2016. diff --git a/numerical/optimization/Adam.cpp b/numerical/optimization/Adam.cpp new file mode 100644 index 00000000..9832e456 --- /dev/null +++ b/numerical/optimization/Adam.cpp @@ -0,0 +1,6 @@ +#include "numerical/optimization/Adam.hpp" + +namespace optimization +{ + template class Adam; +} diff --git a/numerical/optimization/Adam.hpp b/numerical/optimization/Adam.hpp new file mode 100644 index 00000000..5a676a35 --- /dev/null +++ b/numerical/optimization/Adam.hpp @@ -0,0 +1,92 @@ +#pragma once + +#if defined(__GNUC__) && !defined(__clang__) +#pragma GCC push_options +#pragma GCC optimize("O3", "fast-math") +#endif + +#include "numerical/math/CompilerOptimizations.hpp" +#include "numerical/math/Math.hpp" +#include "numerical/optimization/StepOptimizer.hpp" + +namespace optimization +{ + template + class Adam + : public StepOptimizer + { + static_assert(std::is_floating_point_v, "Adam supports floating-point types only"); + + public: + using Vector = typename StepOptimizer::Vector; + + struct Parameters + { + T learningRate; + T beta1{ T{ 0.9 } }; + T beta2{ T{ 0.999 } }; + T epsilon{ T{ 1e-8 } }; + }; + + explicit Adam(const Parameters& params); + + void Step(Vector& theta, const Vector& gradient) override; + void Reset() override; + + private: + Parameters parameters; + Vector firstMoment{}; + Vector secondMoment{}; + T beta1Power{ T{ 1 } }; + T beta2Power{ T{ 1 } }; + }; + + template + Adam::Adam(const Parameters& params) + : parameters{ params } + { + really_assert(params.learningRate > T{ 0 }); + really_assert(params.beta1 >= T{ 0 } && params.beta1 < T{ 1 }); + really_assert(params.beta2 >= T{ 0 } && params.beta2 < T{ 1 }); + really_assert(params.epsilon > T{ 0 }); + } + + template + OPTIMIZE_FOR_SPEED void Adam::Step(Vector& theta, const Vector& gradient) + { + beta1Power *= parameters.beta1; + beta2Power *= parameters.beta2; + + firstMoment = firstMoment * parameters.beta1 + gradient * (T{ 1 } - parameters.beta1); + + for (std::size_t i = 0; i < N; ++i) + secondMoment[i] = parameters.beta2 * secondMoment[i] + (T{ 1 } - parameters.beta2) * gradient[i] * gradient[i]; + + const T beta1Correction = T{ 1 } - beta1Power; + const T beta2Correction = T{ 1 } - beta2Power; + + for (std::size_t i = 0; i < N; ++i) + { + const T mHat = firstMoment[i] / beta1Correction; + const T vHat = secondMoment[i] / beta2Correction; + theta[i] -= parameters.learningRate * mHat / (math::Sqrt(vHat) + parameters.epsilon); + } + } + + template + void Adam::Reset() + { + firstMoment = Vector{}; + secondMoment = Vector{}; + beta1Power = T{ 1 }; + beta2Power = T{ 1 }; + } + +#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD + extern template class Adam; +#endif +} + +#if defined(__GNUC__) && !defined(__clang__) +#pragma GCC pop_options +#endif diff --git a/numerical/optimization/CMakeLists.txt b/numerical/optimization/CMakeLists.txt index d230cf1c..7335194d 100644 --- a/numerical/optimization/CMakeLists.txt +++ b/numerical/optimization/CMakeLists.txt @@ -12,15 +12,20 @@ target_link_libraries(numerical.optimization ${NUMERICAL_VISIBILITY} ) target_sources(numerical.optimization PRIVATE + Adam.hpp BayesianOptimization.hpp GradientDescent.hpp ObjectiveFunction.hpp Optimizer.hpp + Sgd.hpp + StepOptimizer.hpp ) numerical_add_coverage_sources(numerical.optimization + Adam.cpp BayesianOptimization.cpp GradientDescent.cpp + Sgd.cpp ) add_subdirectory(test) diff --git a/numerical/optimization/Sgd.cpp b/numerical/optimization/Sgd.cpp new file mode 100644 index 00000000..644baa99 --- /dev/null +++ b/numerical/optimization/Sgd.cpp @@ -0,0 +1,6 @@ +#include "numerical/optimization/Sgd.hpp" + +namespace optimization +{ + template class Sgd; +} diff --git a/numerical/optimization/Sgd.hpp b/numerical/optimization/Sgd.hpp new file mode 100644 index 00000000..9ff9df85 --- /dev/null +++ b/numerical/optimization/Sgd.hpp @@ -0,0 +1,70 @@ +#pragma once + +#if defined(__GNUC__) && !defined(__clang__) +#pragma GCC push_options +#pragma GCC optimize("O3", "fast-math") +#endif + +#include "numerical/math/CompilerOptimizations.hpp" +#include "numerical/optimization/StepOptimizer.hpp" + +namespace optimization +{ + template + class Sgd + : public StepOptimizer + { + static_assert(std::is_floating_point_v, "Sgd supports floating-point types only"); + + public: + using Vector = typename StepOptimizer::Vector; + + struct Parameters + { + T learningRate; + T momentum{ T{ 0 } }; + bool nesterov{ false }; + }; + + explicit Sgd(const Parameters& params); + + void Step(Vector& theta, const Vector& gradient) override; + void Reset() override; + + private: + Parameters parameters; + Vector velocity{}; + }; + + template + Sgd::Sgd(const Parameters& params) + : parameters{ params } + { + really_assert(params.learningRate > T{ 0 }); + really_assert(params.momentum >= T{ 0 } && params.momentum < T{ 1 }); + } + + template + OPTIMIZE_FOR_SPEED void Sgd::Step(Vector& theta, const Vector& gradient) + { + velocity = velocity * parameters.momentum + gradient; + if (parameters.nesterov) + theta = theta - (gradient + velocity * parameters.momentum) * parameters.learningRate; + else + theta = theta - velocity * parameters.learningRate; + } + + template + void Sgd::Reset() + { + velocity = Vector{}; + } + +#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD + extern template class Sgd; +#endif +} + +#if defined(__GNUC__) && !defined(__clang__) +#pragma GCC pop_options +#endif diff --git a/numerical/optimization/StepOptimizer.hpp b/numerical/optimization/StepOptimizer.hpp new file mode 100644 index 00000000..9859b4c7 --- /dev/null +++ b/numerical/optimization/StepOptimizer.hpp @@ -0,0 +1,20 @@ +#pragma once + +#include "numerical/math/Matrix.hpp" + +namespace optimization +{ + template + class StepOptimizer + { + static_assert(std::is_floating_point_v, "StepOptimizer supports floating-point types only"); + + public: + virtual ~StepOptimizer() = default; + + using Vector = math::Vector; + + virtual void Step(Vector& theta, const Vector& gradient) = 0; + virtual void Reset() = 0; + }; +} diff --git a/numerical/optimization/test/CMakeLists.txt b/numerical/optimization/test/CMakeLists.txt index 895273de..99a22e66 100644 --- a/numerical/optimization/test/CMakeLists.txt +++ b/numerical/optimization/test/CMakeLists.txt @@ -12,4 +12,5 @@ numerical_link_qemu_runtime(numerical.optimization_test) target_sources(numerical.optimization_test PRIVATE TestBayesianOptimization.cpp TestGradientDescent.cpp + TestStepOptimizer.cpp ) diff --git a/numerical/optimization/test/TestStepOptimizer.cpp b/numerical/optimization/test/TestStepOptimizer.cpp new file mode 100644 index 00000000..0fa4ee85 --- /dev/null +++ b/numerical/optimization/test/TestStepOptimizer.cpp @@ -0,0 +1,94 @@ +#include "numerical/math/Tolerance.hpp" +#include "numerical/optimization/Adam.hpp" +#include "numerical/optimization/Sgd.hpp" +#include +#include +#include + +namespace +{ + using Vector2f = math::Vector; + + Vector2f MakeVec(float x, float y) + { + Vector2f v{}; + v[0] = x; + v[1] = y; + return v; + } + + class TestStepOptimizer + : public ::testing::Test + {}; +} + +TEST_F(TestStepOptimizer, single_step_against_hand_computed_values) +{ + optimization::Sgd sgd{ { 0.1f, 0.0f, false } }; + auto theta = MakeVec(1.0f, 2.0f); + sgd.Step(theta, MakeVec(0.5f, 1.0f)); + EXPECT_NEAR(theta[0], 0.95f, 1e-5f); + EXPECT_NEAR(theta[1], 1.9f, 1e-5f); + + optimization::Sgd nsgd{ { 0.1f, 0.9f, true } }; + auto thetaN = MakeVec(1.0f, 2.0f); + nsgd.Step(thetaN, MakeVec(0.5f, 1.0f)); + EXPECT_NEAR(thetaN[0], 0.905f, 1e-5f); + EXPECT_NEAR(thetaN[1], 1.81f, 1e-5f); +} + +TEST_F(TestStepOptimizer, convergence_on_quadratic) +{ + optimization::Sgd sgd{ { 0.1f, 0.0f, false } }; + auto theta = MakeVec(1.0f, 1.0f); + + for (int i = 0; i < 100; ++i) + sgd.Step(theta, theta); + + EXPECT_NEAR(theta[0], 0.0f, math::Tolerance()); + EXPECT_NEAR(theta[1], 0.0f, math::Tolerance()); +} + +TEST_F(TestStepOptimizer, adam_bias_correction_at_t1) +{ + optimization::Adam adam{ { 0.001f, 0.9f, 0.999f, 1e-8f } }; + auto theta = MakeVec(0.0f, 0.0f); + adam.Step(theta, MakeVec(1.0f, 1.0f)); + + const float beta1Correction = 1.0f - 0.9f; + const float beta2Correction = 1.0f - 0.999f; + const float mHat = (0.1f) / beta1Correction; + const float vHat = (0.001f) / beta2Correction; + const float expected = -0.001f * mHat / (std::sqrt(vHat) + 1e-8f); + + EXPECT_NEAR(theta[0], expected, 1e-5f); + EXPECT_NEAR(theta[1], expected, 1e-5f); +} + +TEST_F(TestStepOptimizer, reset_restores_initial_state) +{ + optimization::Sgd sgd{ { 0.1f, 0.9f, false } }; + auto theta1 = MakeVec(1.0f, 2.0f); + sgd.Step(theta1, MakeVec(0.5f, 1.0f)); + + sgd.Reset(); + + auto theta2 = MakeVec(1.0f, 2.0f); + sgd.Step(theta2, MakeVec(0.5f, 1.0f)); + + EXPECT_NEAR(theta1[0], theta2[0], 1e-6f); + EXPECT_NEAR(theta1[1], theta2[1], 1e-6f); + + optimization::Adam adam{ { 0.01f } }; + auto theta3 = MakeVec(1.0f, 2.0f); + adam.Step(theta3, MakeVec(0.5f, 1.0f)); + adam.Step(theta3, MakeVec(0.5f, 1.0f)); + + adam.Reset(); + + auto theta4 = MakeVec(1.0f, 2.0f); + adam.Step(theta4, MakeVec(0.5f, 1.0f)); + + EXPECT_NEAR(theta4[0], 0.99f, 1e-5f); + EXPECT_NEAR(theta4[1], 1.99f, 1e-5f); +} From e9fb0350e32ce0b4549a2f25f3893a08887a7927 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 1 Oct 2026 05:56:32 +0000 Subject: [PATCH 2/2] docs: align markdown tables per MegaLinter Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01N4pngRiBqeb1xiseKXNLG6 --- doc/optimization/README.md | 2 +- doc/optimization/StepOptimizer.md | 52 +++++++++++++++---------------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/doc/optimization/README.md b/doc/optimization/README.md index dd6e435a..76a711b4 100644 --- a/doc/optimization/README.md +++ b/doc/optimization/README.md @@ -8,4 +8,4 @@ General-purpose optimization algorithms for parameter fitting, training, and con |--------------------------------------------------|---------------------------------------------------------------------------------------------| | [Bayesian Optimization](BayesianOptimization.md) | Gradient-free global optimization using Gaussian Process surrogate and Expected Improvement | | [Gradient Descent](Optimizer.md) | Iterative batch optimization via gradient-based weight updates | -| [SGD / Adam Step Optimisers](StepOptimizer.md) | Stateful per-step SGD (with optional momentum/Nesterov) and Adam with bias correction | +| [SGD / Adam Step Optimisers](StepOptimizer.md) | Stateful per-step SGD (with optional momentum/Nesterov) and Adam with bias correction | diff --git a/doc/optimization/StepOptimizer.md b/doc/optimization/StepOptimizer.md index 8d77e65b..c395082b 100644 --- a/doc/optimization/StepOptimizer.md +++ b/doc/optimization/StepOptimizer.md @@ -48,10 +48,10 @@ Typical hyper-parameters: $\eta = 10^{-3}$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, ## Complexity Analysis -| Algorithm | Time per step | Extra state (floats) | Notes | -|-------------|---------------|----------------------|--------------------------------------------| -| SGD | $O(N)$ | $N$ | One velocity vector (equals the gradient when momentum is 0) | -| Adam | $O(N)$ | $2N + 2$ | Two moment vectors plus the running powers $\beta_1^t$, $\beta_2^t$ | +| Algorithm | Time per step | Extra state (floats) | Notes | +|-----------|---------------|----------------------|---------------------------------------------------------------------| +| SGD | $O(N)$ | $N$ | One velocity vector (equals the gradient when momentum is 0) | +| Adam | $O(N)$ | $2N + 2$ | Two moment vectors plus the running powers $\beta_1^t$, $\beta_2^t$ | $N$ is the number of parameters. All operations are in-place; no heap allocation is required. @@ -60,20 +60,20 @@ $N$ is the number of parameters. All operations are in-place; no heap allocation **SGD with momentum** on $\mathcal{L}(\theta) = \frac{1}{2}\|\theta\|^2$, $N=1$, $\theta_0 = 1$, $\eta = 0.1$, $\beta = 0.9$: | $t$ | $g_t = \theta_t$ | $v_t = 0.9 v_{t-1} + g_t$ | $\theta_{t+1} = \theta_t - 0.1 v_t$ | -|-----|------------------|---------------------------|--------------------------------------| -| 1 | 1.000 | 1.000 | 0.900 | -| 2 | 0.900 | 1.800 | 0.720 | -| 3 | 0.720 | 2.340 | 0.486 | +|-----|------------------|---------------------------|-------------------------------------| +| 1 | 1.000 | 1.000 | 0.900 | +| 2 | 0.900 | 1.800 | 0.720 | +| 3 | 0.720 | 2.340 | 0.486 | **Adam** on the same objective, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\varepsilon = 10^{-8}$, $\theta_0 = 0$, $g_1 = 1$: -| Quantity | Value | -|---------------|-----------------| -| $m_1$ | $0.1$ | -| $v_1$ | $0.001$ | -| $\hat{m}_1$ | $1.0$ | -| $\hat{v}_1$ | $1.0$ | -| $\theta_1$ | $-\eta \approx -0.001$ | +| Quantity | Value | +|-------------|------------------------| +| $m_1$ | $0.1$ | +| $v_1$ | $0.001$ | +| $\hat{m}_1$ | $1.0$ | +| $\hat{v}_1$ | $1.0$ | +| $\theta_1$ | $-\eta \approx -0.001$ | Bias correction is the critical step: without it, $m_1/\sqrt{v_1} \approx 3.16$, giving a first step roughly $\sqrt{1000}$ times larger than the bias-corrected value. @@ -87,12 +87,12 @@ Bias correction is the critical step: without it, $m_1/\sqrt{v_1} \approx 3.16$, ## Variants & Generalizations -| Variant | Change from base | -|-----------------------|-----------------------------------------------------------------------| -| AdaGrad | Non-decaying sum of squared gradients (no $\beta_2$ decay); aggressive learning rate shrinkage | -| RMSProp | Adam without first-moment tracking; lacks bias correction | -| AdamW | Decoupled weight-decay applied directly to $\theta$ before the gradient step | -| AMSGrad | Replaces $\hat{v}$ with the running maximum to guarantee monotone effective step-size | +| Variant | Change from base | +|---------|------------------------------------------------------------------------------------------------| +| AdaGrad | Non-decaying sum of squared gradients (no $\beta_2$ decay); aggressive learning rate shrinkage | +| RMSProp | Adam without first-moment tracking; lacks bias correction | +| AdamW | Decoupled weight-decay applied directly to $\theta$ before the gradient step | +| AMSGrad | Replaces $\hat{v}$ with the running maximum to guarantee monotone effective step-size | ## Applications @@ -103,11 +103,11 @@ Bias correction is the critical step: without it, $m_1/\sqrt{v_1} \approx 3.16$, ## Connections to Other Algorithms -| Component | Relationship | -|-------------------------------------------------|--------------------------------------------------------------------------------------| -| [Gradient Descent](Optimizer.md) | Batch counterpart; shares the learning-rate update rule but recomputes the objective each time | -| [LMS Adaptive Filter](../estimators/README.md) | Equivalent to online SGD for a linear regression model under MSE loss | -| [Regularization](../regularization/README.md) | Adds a penalty gradient to $g_t$; compatible with any step optimiser | +| Component | Relationship | +|------------------------------------------------|------------------------------------------------------------------------------------------------| +| [Gradient Descent](Optimizer.md) | Batch counterpart; shares the learning-rate update rule but recomputes the objective each time | +| [LMS Adaptive Filter](../estimators/README.md) | Equivalent to online SGD for a linear regression model under MSE loss | +| [Regularization](../regularization/README.md) | Adds a penalty gradient to $g_t$; compatible with any step optimiser | ## References & Further Reading