11 — Training

Batch Normalization#

StabilizationMathematical 1 min read
◆ The PatternNormalizing activations to keep training stable and fast

Batch Normalization normalises each layer's activations to have zero mean and unit variance across the mini-batch, then scales/shifts with learned parameters γ and β.

μ_B = (1/m)·Σxᵢ    σ²_B = (1/m)·Σ(xᵢ−μ_B)²
Compute batch mean and variance
x̂ᵢ = (xᵢ − μ_B) / √(σ²_B + ε)
Normalise: ε = small constant for numerical stability (1e-5)
yᵢ = γ · x̂ᵢ + β
γ = learned scale  |  β = learned shift — restores representational power
// Effect of batch norm on activation distributions across layers
Layer Norm vs Batch Norm: BatchNorm normalises over the batch dimension — problematic for small batches and transformers. LayerNorm normalises over the feature dimension and is the standard in transformers. See topic 27 for all variants.
Pattern bridge: Normalizing activations to zero mean and unit variance is exactly z-score standardization from statistics. Markets use the same logic: Bollinger Bands normalize price relative to its rolling mean and standard deviation. For transformers and small batches, see the other normalization variants.
Worked example

A batch of 64 activations with mean 5 and standard deviation 3, normalised: every unit now has mean 0 and standard deviation 1. But the statistics come from the batch. With 64 examples the batch mean is off by about 0.38 on average; with 2 examples by 2.15 — noise as large as the signal. That is why batch norm struggles with small batches.

Where it misleads
  • Training uses batch statistics and inference uses running averages; when the two differ, train and test behave differently.
  • For small batches or sequences, LayerNorm or GroupNorm avoid the dependence on the batch (see Normalization).
  • Why it helps is still debated; Santurkar et al. (2018) argue it smooths the loss landscape rather than fixing “internal covariate shift”.
In code
rng = np.random.default_rng(3)
acts = rng.normal(5, 3, size=(64, 4))              # a batch of 64, four units
xhat = (acts - acts.mean(0)) / np.sqrt(acts.var(0) + 1e-5)
after = (xhat.mean(0).round(6).tolist(), xhat.std(0).round(3).tolist())
# how noisy the batch mean is as an estimate of the true mean (5)
noise = {m: np.std([rng.normal(5, 3, m).mean() for _ in range(2000)]).round(3) for m in (2, 64)}
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources
  • S. Ioffe & C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, ICML, 2015
  • S. Santurkar, D. Tsipras, A. Ilyas & A. Madry, “How Does Batch Normalization Help Optimization?”, NeurIPS, 2018
  • Y. Wu & K. He, “Group Normalization”, ECCV, 2018

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 8
← Previous
Regularization
Open in the full reader, with the topic sidebar