Optimization✓ Mathematical
◆ The PatternBeyond vanilla SGD — momentum, adaptive learning rates, Adam
Modern optimizers improve on vanilla gradient descent by adding momentum (using past gradients) and adaptive rates (different step size per parameter).
SGD+Momentum: v := βv − α·∇L w := w + v
β = momentum (typically 0.9) — accumulates velocity across steps
RMSProp: s := ρs + (1−ρ)·(∇L)² w := w − α·∇L/√(s+ε)
Divides by RMS of recent gradients — large gradients get smaller steps
Adam: m̂ = m/(1−β₁ᵗ) v̂ = v/(1−β₂ᵗ) w := w − α·m̂/√(v̂+ε)
Adam = momentum + RMSProp + bias correction. Default: β₁=0.9, β₂=0.999, ε=1e-8
// Optimizer comparison on a saddle point surface
# PyTorch optimizers optim = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9) optim = torch.optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999)) optim = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01) # default choice
AdamW is the default choice for most modern models. It decouples weight decay from the gradient update, fixing a subtle bug in Adam's L2 regularization.
Pattern bridge: Momentum in Adam is literally momentum in trading — using past velocity to guide the next step. Exponential moving averages smooth both gradient updates and price series identically.