09 — Training

Optimizers#

OptimizationMathematical 1 min read
◆ The PatternBeyond vanilla SGD — momentum, adaptive learning rates, Adam

Modern optimizers improve on vanilla gradient descent by adding momentum (using past gradients) and adaptive rates (different step size per parameter).

SGD+Momentum: v := βv − α·∇L    w := w + v
β = momentum (typically 0.9) — accumulates velocity across steps
RMSProp: s := ρs + (1−ρ)·(∇L)²    w := w − α·∇L/√(s+ε)
Divides by RMS of recent gradients — large gradients get smaller steps
Adam: m̂ = m/(1−β₁ᵗ)    v̂ = v/(1−β₂ᵗ)    w := w − α·m̂/√(v̂+ε)
Adam = momentum + RMSProp + bias correction. Default: β₁=0.9, β₂=0.999, ε=1e-8
// Optimizer comparison on a saddle point surface
AdamW is the default choice for most modern models. It decouples weight decay from the gradient update, fixing a subtle bug in Adam's L2 regularization.
Pattern bridge: Momentum in Adam is literally momentum in trading — using past velocity to guide the next step. Exponential moving averages smooth both gradient updates and price series identically.
Worked example

A narrow valley, 50 times steeper in one direction than the other, started from (10, 1). Plain gradient descent, with a step close to the largest the steep direction allows, needs 152 steps to get the loss under 0.001. Momentum needs 93, and Adam, which scales each direction separately, 80. The difference is how they cope with uneven curvature.

Where it misleads
  • Each optimizer was given hand-picked settings here; with other settings the ranking can change. Optimizer comparisons are only as fair as their tuning.
  • Adaptive methods can converge faster but generalise worse than well-tuned SGD on some tasks (Wilson et al. 2017).
  • Toy valleys are smooth and deterministic; real training adds noise from mini-batches.
In code
grad = lambda p: np.array([p[0], 50 * p[1]])        # f = 0.5 (x^2 + 50 y^2): a narrow valley
f = lambda p: 0.5 * (p[0] ** 2 + 50 * p[1] ** 2)

def steps_to(update, tol=1e-3, limit=10_000):
    p, state = np.array([10.0, 1.0]), {}
    for k in range(1, limit + 1):
        p = update(p, grad(p), state, k)
        if f(p) < tol: return k
    return limit

def sgd(p, g, s, k): return p - 0.035 * g
def momentum(p, g, s, k):
    s['v'] = 0.9 * s.get('v', 0) - 0.035 * g; return p + s['v']
def adam(p, g, s, k, lr=0.5, b1=0.9, b2=0.999):
    s['m'] = b1 * s.get('m', 0) + (1 - b1) * g; s['v'] = b2 * s.get('v', 0) + (1 - b2) * g * g
    m, v = s['m'] / (1 - b1 ** k), s['v'] / (1 - b2 ** k)
    return p - lr * m / (np.sqrt(v) + 1e-8)

result = {name: steps_to(u) for name, u in [('sgd', sgd), ('momentum', momentum), ('adam', adam)]}
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

The same idea with a library
# PyTorch optimizers
optim = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
optim = torch.optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999))
optim = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)  # default choice
Sources
  • D. P. Kingma & J. Ba, “Adam: A Method for Stochastic Optimization”, ICLR, 2015
  • B. T. Polyak, “Some Methods of Speeding Up the Convergence of Iteration Methods”, USSR Computational Mathematics and Mathematical Physics 4(5), 1964
  • A. C. Wilson et al., “The Marginal Value of Adaptive Gradient Methods in Machine Learning”, NeurIPS, 2017

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 11
← Previous
Backpropagation
Open in the full reader, with the topic sidebar