StabilizationMathematical
◆ The PatternPreventing exploding gradients by capping their magnitude
In RNNs and deep networks, gradients can explode exponentially during backprop. Gradient clipping caps the gradient norm before the optimizer step, keeping training stable.
Clip by norm: if ||g|| > max_norm → g := g · max_norm / ||g||
Rescales the entire gradient vector to have norm ≤ max_norm. Preserves direction.
Clip by value: gᵢ := clamp(gᵢ, −clip_val, +clip_val)
Clips each element independently. Faster but can change gradient direction.
// Gradient norm over training — with and without clipping
Max Norm1.0
When to use: Almost always for RNNs/LSTMs. Common in transformer training too (GPT uses max_norm=1.0). The max_norm value of 1.0 is a good default.
Pattern bridge: Capping gradient magnitude is a guardrail — like ATR-based stops that cap how much a position can move against you.
Worked example
A gradient of length 50. Clipping by norm to 1 gives (0.6, −0.8, 0.01): one fiftieth the length, exactly the same direction (cosine 1.0). Clipping each value to [−1, 1] gives (1, −1, 0.5): the small third component is now as big as the others, and the direction has turned (cosine 0.94).
Where it misleads
- A clipping threshold that triggers on every step is just a smaller learning rate; log how often it fires.
- Clipping treats the symptom; frequent explosions often point to a bad learning rate, initialization or data bug.
- Clipping by value distorts the direction; prefer clipping by norm.
In code
g = np.array([30.0, -40.0, 0.5]) # an exploding gradient, norm about 50 max_norm = 1.0 by_norm = g * min(1, max_norm / np.linalg.norm(g)) by_value = np.clip(g, -1, 1) cos = lambda u, v: u @ v / (np.linalg.norm(u) * np.linalg.norm(v))
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
# Standard training loop with gradient clipping loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # clip by norm optimizer.step()
Sources
- R. Pascanu, T. Mikolov & Y. Bengio, “On the Difficulty of Training Recurrent Neural Networks”, ICML, 2013
- J. Zhang, T. He, S. Sra & A. Jadbabaie, “Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity”, ICLR, 2020
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 2