Stabilization✓ Mathematical
◆ The PatternPreventing exploding gradients by capping their magnitude
In RNNs and deep networks, gradients can explode exponentially during backprop. Gradient clipping caps the gradient norm before the optimizer step, keeping training stable.
Clip by norm: if ||g|| > max_norm → g := g · max_norm / ||g||
Rescales the entire gradient vector to have norm ≤ max_norm. Preserves direction.
Clip by value: gᵢ := clamp(gᵢ, −clip_val, +clip_val)
Clips each element independently. Faster but can change gradient direction.
// Gradient norm over training — with and without clipping
Max Norm1.0
# Standard training loop with gradient clipping loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # clip by norm optimizer.step()
When to use: Almost always for RNNs/LSTMs. Common in transformer training too (GPT uses max_norm=1.0). The max_norm value of 1.0 is a good default.
Pattern bridge: Capping gradient magnitude is a guardrail — like ATR-based stops that cap how much a position can move against you.