Optimization✓ Mathematical
1 min read
◆ The PatternRolling downhill on the loss landscape to find optimal weights
Gradient Descent is how models learn. We start at random weights and take small steps in the direction of steepest descent — opposite the gradient — to reach minimum loss.
w := w − α · ∂L/∂w
α = learning rate | ∂L/∂w = gradient (slope of loss w.r.t. weight)
| Variant | Samples/step | Pros | Cons |
|---|---|---|---|
| Batch GD | All n | Stable, exact gradient | Slow per step |
| Stochastic GD | 1 | Fast, noisy escapes minima | High variance |
| Mini-batch GD | 32–256 | Best of both worlds | Batch size is a hyperparameter |
// Animated loss landscape — adjust learning rate and run
Learning Rate α0.10
Steps0
Learning rate: Too large → overshoot and diverge. Too small → extremely slow convergence. Learning rate warmup + decay schedulers (topic 12) solve this in practice.
Pattern bridge: Rolling downhill on a loss surface is the same intuition behind Rate of Change in markets — both measure slope to decide direction.
How to use this in practice
- Start with Adam optimizer (lr=1e-3) — it handles most cases well out of the box
- If training loss plateaus, try reducing lr by 10x or use a cosine annealing schedule
- If training loss oscillates wildly → lr is too high — halve it until stable
- Use gradient clipping (
max_norm=1.0) for RNNs and transformers to prevent exploding gradients - Monitor both train loss and val loss — divergence means you're overfitting (see Learning Curves)
When gradient descent fails: Non-convex loss landscapes have local minima and saddle points. SGD with momentum helps escape saddle points; Adam helps with sparse gradients. For very deep networks, poor initialization can cause vanishing gradients — use He or Xavier init.
Performance in practice
- Adam converges 2-5x faster than plain SGD on most tasks, but SGD+momentum often finds flatter minima → better generalization
- GPT-3 training used Adam with β1=0.9, β2=0.95, lr warmup over 375M tokens then cosine decay
- Batch size matters: larger batches = more stable gradients but worse generalization. Most papers use 32-256 for CV, 8-64 for NLP
- Mixed-precision training (FP16) cuts memory 2x and speeds up training 30-50% on modern GPUs with negligible accuracy loss
When to use this
✓ Use when: Training any neural network. Fine-tuning pre-trained models. Logistic regression on large datasets. Any differentiable loss function.
✗ Skip when: Tree models (XGBoost, Random Forest) — they use different optimization. Small linear models where closed-form solutions exist (OLS). Problems where derivative-free optimization (genetic algorithms, Bayesian optimization) is more appropriate.
Use this pattern on real data
Pattern Portal Case: Housing Regression
Train a baseline model, monitor train/validation loss, and compare learning-rate choices against MAE/RMSE.
Quick start — copy to notebook
pip install torch scikit-learn
import torch
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
for epoch in range(20):
model.train()
optimizer.zero_grad()
pred = model(X_train)
loss = loss_fn(pred, y_train)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()