ArchitectureMathematical
1 min read
◆ The PatternLayerNorm, RMSNorm, GroupNorm — which normalisation for which architecture
Different architectures need different normalisation. BatchNorm works for CNNs, LayerNorm for transformers, RMSNorm for modern LLMs (faster, no mean subtraction).
LayerNorm: x̂ = (x − μ) / √(σ² + ε) · γ + β
Normalises over features (last dim) — independent of batch size. Standard for transformers.
RMSNorm: x̂ = x / RMS(x) · γ RMS(x) = √(Σxᵢ²/n)
Skips mean subtraction — faster than LayerNorm (7–64% in its authors’ tests). Used in LLaMA, Mistral, Gemma.
GroupNorm: split channels into G groups, normalise each
Works with any batch size. G=32 is common. Used in diffusion models (U-Net).
| Method | Normalises Over | Batch-Dependent | Used In |
|---|---|---|---|
| BatchNorm | Batch dim | Yes | CNNs, ResNets |
| LayerNorm | Feature dim | No | BERT, GPT-2, ViT |
| RMSNorm | Feature dim (no mean) | No | LLaMA, Mistral, Gemma |
| GroupNorm | Channel groups | No | U-Net, diffusion |
| InstanceNorm | Single sample, per-channel | No | Style transfer |
Pattern bridge: LayerNorm in transformers standardizes activations per sample — the same operation as z-scoring. It’s why Bollinger Bands work: normalizing price by its own volatility reveals the signal beneath.
Worked example
The vector (2, 4, 6, 8). LayerNorm gives (−1.34, −0.45, 0.45, 1.34); RMSNorm, which skips the mean, gives (0.37, 0.73, 1.10, 1.46). Add 100 to every feature and LayerNorm’s output is unchanged (True), while RMSNorm’s becomes (0.97, 0.99, 1.01, 1.03). RMSNorm is cheaper; LayerNorm is shift-invariant.
Where it misleads
- The learned scale and shift (γ, β) can undo the normalization; it changes the parametrisation, not what the network can represent.
- Which norm works best is empirical and architecture-specific; results do not transfer automatically.
- Where the norm sits (before or after the residual) matters as much as which norm it is.
In code
x = np.array([2.0, 4.0, 6.0, 8.0]) layer = (x - x.mean()) / np.sqrt(x.var() + 1e-5) # LayerNorm, before gamma and beta rms = x / np.sqrt((x ** 2).mean() + 1e-5) # RMSNorm: no mean subtraction shifted = x + 100 # add the same constant to every feature layer_shift = (shifted - shifted.mean()) / np.sqrt(shifted.var() + 1e-5) rms_shift = shifted / np.sqrt((shifted ** 2).mean() + 1e-5)
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
nn.LayerNorm(512) # standard transformers # RMSNorm (not in PyTorch by default) class RMSNorm(nn.Module): def __init__(self, d, eps=1e-6): super().__init__() self.w = nn.Parameter(torch.ones(d)) self.eps = eps def forward(self, x): return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.w
Sources
- J. L. Ba, J. R. Kiros & G. E. Hinton, “Layer Normalization”, arXiv:1607.06450, 2016
- B. Zhang & R. Sennrich, “Root Mean Square Layer Normalization”, NeurIPS, 2019
- Y. Wu & K. He, “Group Normalization”, ECCV, 2018
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1