Architecture✓ Mathematical
◆ The PatternLayerNorm, RMSNorm, GroupNorm — which normalisation for which architecture
Different architectures need different normalisation. BatchNorm works for CNNs, LayerNorm for transformers, RMSNorm for modern LLMs (faster, no mean subtraction).
LayerNorm: x̂ = (x − μ) / √(σ² + ε) · γ + β
Normalises over features (last dim) — independent of batch size. Standard for transformers.
RMSNorm: x̂ = x / RMS(x) · γ RMS(x) = √(Σxᵢ²/n)
Skips mean subtraction — 15% faster than LayerNorm. Used in LLaMA, Mistral, Gemma.
GroupNorm: split channels into G groups, normalise each
Works with any batch size. G=32 is common. Used in diffusion models (U-Net).
| Method | Normalises Over | Batch-Dependent | Used In |
|---|---|---|---|
| BatchNorm | Batch dim | Yes | CNNs, ResNets |
| LayerNorm | Feature dim | No | BERT, GPT-2, ViT |
| RMSNorm | Feature dim (no mean) | No | LLaMA, Mistral, Gemma |
| GroupNorm | Channel groups | No | U-Net, diffusion |
| InstanceNorm | Single sample, per-channel | No | Style transfer |
nn.LayerNorm(512) # standard transformers # RMSNorm (not in PyTorch by default) class RMSNorm(nn.Module): def __init__(self, d, eps=1e-6): super().__init__() self.w = nn.Parameter(torch.ones(d)) self.eps = eps def forward(self, x): return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.w
Pattern bridge: LayerNorm in transformers standardizes activations per sample — the same operation as z-scoring. It’s why Bollinger Bands work: normalizing price by its own volatility reveals the signal beneath.