StabilizationMathematical
◆ The PatternHow you start determines if you converge — Xavier, He, and why they matter
Bad initialization → activations explode or vanish → gradients die → training fails. The goal: keep variance of activations stable across layers.
Xavier/Glorot: W ~ N(0, 2/(fan_in + fan_out))
Designed for tanh/sigmoid — keeps variance ≈1 going forward and backward
He/Kaiming: W ~ N(0, 2/fan_in)
Designed for ReLU — compensates for ReLU killing half the activations
// Activation variance through 10 layers with different initializations
Modern practice: PyTorch's nn.Linear uses Kaiming uniform by default. Transformers typically use small normal init (std ≈ 0.02) + special scaling for residual paths.
Pattern bridge: Xavier initialization sets variance to 1/fan_in — the same principle behind variance scaling.
Worked example
Twenty ReLU layers, 256 wide. With weights drawn from N(0, 1), activations grow to a standard deviation of 1.5×10²¹. With weights of standard deviation 0.01 they shrink to 1.2×10⁻¹⁹. He initialization, variance 2/fan-in, keeps them at 0.73. The right scale makes each layer pass on a signal of the same size.
Where it misleads
- The rules assume particular activations: Xavier for tanh-like, He for ReLU. Mismatch them and the drift returns.
- Normalization layers and residual connections make networks far less sensitive to initialization.
- Good initialization makes training possible; it does not make it good.
In code
rng = np.random.default_rng(4)
def final_std(scale, layers=20, width=256):
h = rng.normal(size=(width, 100))
for _ in range(layers):
W = rng.normal(0, scale, (width, width))
h = np.maximum(W @ h, 0) # ReLU layers
return h.std()
naive = final_std(1.0) # N(0, 1) weights
he = final_std(np.sqrt(2 / 256)) # He: variance 2 / fan_in
tiny = final_std(0.01)Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
# PyTorch initialization nn.init.xavier_uniform_(layer.weight) # for tanh/sigmoid nn.init.kaiming_normal_(layer.weight) # for ReLU (default) nn.init.zeros_(layer.bias) # biases → 0
Sources
- X. Glorot & Y. Bengio, “Understanding the Difficulty of Training Deep Feedforward Neural Networks”, AISTATS, 2010
- K. He, X. Zhang, S. Ren & J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, ICCV, 2015
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1