ArchitectureMathematical
◆ The PatternThe architecture behind BERT, GPT, and all modern LLMs
The Transformer combines multi-head self-attention with position-wise feed-forward networks, residual connections, and layer normalisation.
PE(pos, 2i) = sin(pos/10000^(2i/d))
Positional Encoding — sine/cosine waves encode token position
FFN(x) = max(0, xW₁+b₁)W₂+b₂
Position-wise feed-forward: two linear layers with ReLU/GELU
x = LayerNorm(x + Sublayer(x))
Residual + LayerNorm — applied around every sublayer
// Transformer block diagram
Pattern bridge: The transformer block — attention + feed-forward + residual — is now the backbone of every modern LLM. GPT-2, counted in the worked example, is a decoder-only model: the original transformer without its encoder.
Worked example
GPT-2 small: width 768, feed-forward 3,072, 12 layers, 50,257 tokens. Each layer holds 7,087,872 parameters — a third in attention, two thirds in the feed-forward block. With token and position embeddings the total is 124,439,808, the familiar “124M”. Most of a transformer is its MLPs.
Where it misleads
- Parameter counts say little about compute: attention cost grows with sequence length, which is not in the count.
- Small details — where the LayerNorm sits, the positional scheme — matter a lot for training stability (Xiong et al. 2020).
- Shared input and output embeddings, as in GPT-2, are counted once here; other models count them twice.
In code
d, ff, layers, vocab = 768, 3072, 12, 50_257 # GPT-2 small attn = 4 * d * d + 4 * d # Q, K, V and output projections mlp = 2 * d * ff + ff + d norms = 2 * 2 * d per_layer = attn + mlp + norms embeddings = vocab * d + 1024 * d # tokens + 1,024 learned positions total = layers * per_layer + embeddings + 2 * d # + final LayerNorm
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
encoder_layer = nn.TransformerEncoderLayer(
d_model=512, nhead=8, dim_feedforward=2048,
dropout=0.1, norm_first=True # Pre-LN (modern default)
)
transformer = nn.TransformerEncoder(encoder_layer, num_layers=6)Sources
- A. Vaswani et al., “Attention Is All You Need”, NeurIPS, 2017
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei & I. Sutskever, “Language Models Are Unsupervised Multitask Learners”, OpenAI, 2019
- R. Xiong et al., “On Layer Normalization in the Transformer Architecture”, ICML, 2020
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1