Pre-Norm · Residuals✓ Mathematical
◆ The PatternThe fundamental repeating unit of every LLM
A transformer block combines attention and FFN with residual connections and layer normalization. Modern LLMs use Pre-Norm (normalize before each sublayer) rather than Post-Norm, which stabilizes training at scale.
Pre-Norm block: x → x + Attn(LN(x)) → x + FFN(LN(x))
LayerNorm before sublayer, residual after. Gradients flow cleanly through the skip connection.
RMSNorm(x) = x / RMS(x) · γ where RMS(x) = √(mean(x²))
Root Mean Square normalization — no mean subtraction. Faster, used in LLaMA, Mistral.
A 70B model stacks 80 blocks. Each block adds ~875M parameters. The residual stream acts as a highway — early layers write features, later layers read and refine them. This is the residual stream mental model.
The residual connection is why deep transformers work at all. Without it, gradients vanish through 80+ layers. With it, there's always a direct path from output to any layer.
Interactive — data flow through a transformer block
Python — transformer block#
import torch.nn as nn
class TransformerBlock(nn.Module):
def __init__(self, d_model, n_heads, n_kv_heads):
super().__init__()
self.ln1 = nn.RMSNorm(d_model)
self.attn = GQA(d_model, n_heads, n_kv_heads)
self.ln2 = nn.RMSNorm(d_model)
self.ffn = SwiGLU_FFN(d_model)
def forward(self, x):
x = x + self.attn(self.ln1(x)) # attention + residual
x = x + self.ffn(self.ln2(x)) # FFN + residual
return xPattern bridge: The repeated unit: attention → add+norm → FFN → add+norm. In ML math, the residual connections prevent gradient vanishing.