Stabilization✓ Mathematical
◆ The PatternNormalizing activations to keep training stable and fast
Batch Normalization normalises each layer's activations to have zero mean and unit variance across the mini-batch, then scales/shifts with learned parameters γ and β.
μ_B = (1/m)·Σxᵢ σ²_B = (1/m)·Σ(xᵢ−μ_B)²
Compute batch mean and variance
x̂ᵢ = (xᵢ − μ_B) / √(σ²_B + ε)
Normalise: ε = small constant for numerical stability (1e-5)
yᵢ = γ · x̂ᵢ + β
γ = learned scale | β = learned shift — restores representational power
// Effect of batch norm on activation distributions across layers
Layer Norm vs Batch Norm: BatchNorm normalises over the batch dimension — problematic for small batches and transformers. LayerNorm normalises over the feature dimension and is the standard in transformers. See topic 27 for all variants.
Pattern bridge: Normalizing activations to zero mean and unit variance is exactly z-score standardization from statistics. Markets use the same logic: Bollinger Bands normalize price relative to its rolling mean and standard deviation.