Gradient Descent

04 — Foundations

Gradient Descent#

OptimizationMathematical 1 min read

◆ The PatternRolling downhill on the loss landscape to find optimal weights

Gradient Descent is how models learn. We start at random weights and take small steps in the direction of steepest descent — opposite the gradient — to reach minimum loss.

w := w − α · ∂L/∂w

α = learning rate | ∂L/∂w = gradient (slope of loss w.r.t. weight)

Variant	Samples/step	Pros	Cons
Batch GD	All n	Stable, exact gradient	Slow per step
Stochastic GD	1	Fast, noisy escapes minima	High variance
Mini-batch GD	32–256	Best of both worlds	Batch size is a hyperparameter

// Animated loss landscape — adjust learning rate and run

Learning Rate α0.10

Steps0

Learning rate: Too large → overshoot and diverge. Too small → extremely slow convergence. Learning rate warmup + decay schedulers (topic 12) solve this in practice.

Pattern bridge: Rolling downhill on a loss surface is the same intuition behind Rate of Change in markets — both measure slope to decide direction.

How to use this in practice

Start with Adam optimizer (lr=1e-3) — it handles most cases well out of the box
If training loss plateaus, try reducing lr by 10x or use a cosine annealing schedule
If training loss oscillates wildly → lr is too high — halve it until stable
Use gradient clipping (max_norm=1.0) for RNNs and transformers to prevent exploding gradients
Monitor both train loss and val loss — divergence means you're overfitting (see Learning Curves)

When gradient descent fails: Non-convex loss landscapes have local minima and saddle points. SGD with momentum helps escape saddle points; Adam helps with sparse gradients. For very deep networks, poor initialization can cause vanishing gradients — use He or Xavier init.

Performance in practice

Adam usually converges faster and with less tuning than plain SGD; well-tuned SGD with momentum often generalises as well or better on vision tasks (Wilson et al. 2017)
GPT-3 was trained with Adam (β₁ = 0.9, β₂ = 0.95), a linear warmup over the first 375M tokens, then cosine decay (Brown et al. 2020)
Very large batches can generalise worse unless the learning rate and warmup are scaled with them (Keskar et al. 2017; Goyal et al. 2017)
Mixed-precision training (FP16/BF16) roughly halves activation memory and speeds training considerably on GPUs with tensor cores, usually without loss of accuracy (Micikevicius et al. 2018)

When to use this

✓ Use when: Training any neural network. Fine-tuning pre-trained models. Logistic regression on large datasets. Any differentiable loss function.

✗ Skip when: Tree models (XGBoost, Random Forest) — they use different optimization. Small linear models where closed-form solutions exist (OLS). Problems where derivative-free optimization (genetic algorithms, Bayesian optimization) is more appropriate.

Use this pattern on real data

Case study: California housing

Train a baseline model, monitor train/validation loss, and compare learning-rate choices against MAE/RMSE.

Quick start — copy to notebook

pip install torch scikit-learn
import torch

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
for epoch in range(20):
    model.train()
    optimizer.zero_grad()
    pred = model(X_train)
    loss = loss_fn(pred, y_train)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()

Worked example

Minimising (w − 3)² from w = 0. With a learning rate of 0.1, ten steps reach 2.68, closing in on 3. With 1.1 each step overshoots further than the last and ten steps end at −15.6. For this curve any rate above 1 diverges: the step size is not a detail, it decides whether learning happens at all.

Where it misleads

Real loss surfaces are not one bowl; gradient descent finds a nearby low point, not the lowest.
A good learning rate depends on the curvature, which differs between directions (see Optimizers).
Stochastic gradients from mini-batches are noisy; the loss does not fall every step.

In code

def descend(lr, steps=10, w=0.0):
    for _ in range(steps):
        w -= lr * 2 * (w - 3)               # gradient of (w - 3)^2
    return w

good = descend(0.1)                          # converging towards 3
too_big = descend(1.1)                       # each step overshoots further

Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources

Deep Learning, I. Goodfellow, Y. Bengio & A. Courville, MIT Press, 2016 — chapter 4
Convex Optimization, S. Boyd & L. Vandenberghe, Cambridge University Press, 2004 — chapter 9

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. The loss jumps around and grows during training. What is the most likely cause?

Show answer

C. Too large a step crosses the valley and lands higher on the other side; repeated, the loss diverges.

2. In which direction does the gradient of the loss point?

Show answer

A. The negative gradient is the locally steepest way down — not necessarily the shortest way to the minimum.

3. Why scale features before gradient descent?

Show answer

C. With similar scales the contours are rounder, and one learning rate suits every direction.

Linked from 6

← Previous

Logistic Regression

Activation Functions

Open in the full reader, with the topic sidebar