Stabilization✓ Mathematical
◆ The PatternHow you start determines if you converge — Xavier, He, and why they matter
Bad initialization → activations explode or vanish → gradients die → training fails. The goal: keep variance of activations stable across layers.
Xavier/Glorot: W ~ N(0, 2/(fan_in + fan_out))
Designed for tanh/sigmoid — keeps variance ≈1 going forward and backward
He/Kaiming: W ~ N(0, 2/fan_in)
Designed for ReLU — compensates for ReLU killing half the activations
// Activation variance through 10 layers with different initializations
# PyTorch initialization nn.init.xavier_uniform_(layer.weight) # for tanh/sigmoid nn.init.kaiming_normal_(layer.weight) # for ReLU (default) nn.init.zeros_(layer.bias) # biases → 0
Modern practice: PyTorch's nn.Linear uses Kaiming uniform by default. Transformers typically use small normal init (std ≈ 0.02) + special scaling for residual paths.
Pattern bridge: Xavier initialization sets variance to 1/fan_in — the same principle behind variance scaling.