Architecture✓ Mathematical
◆ The PatternThe architecture behind BERT, GPT, and all modern LLMs
The Transformer combines multi-head self-attention with position-wise feed-forward networks, residual connections, and layer normalisation.
PE(pos, 2i) = sin(pos/10000^(2i/d))
Positional Encoding — sine/cosine waves encode token position
FFN(x) = max(0, xW₁+b₁)W₂+b₂
Position-wise feed-forward: two linear layers with ReLU/GELU
x = LayerNorm(x + Sublayer(x))
Residual + LayerNorm — applied around every sublayer
// Transformer block diagram
encoder_layer = nn.TransformerEncoderLayer(
d_model=512, nhead=8, dim_feedforward=2048,
dropout=0.1, norm_first=True # Pre-LN (modern default)
)
transformer = nn.TransformerEncoder(encoder_layer, num_layers=6)Pattern bridge: The transformer block — attention + feed-forward + residual — is now the backbone of every modern LLM.