07 — Architecture

Transformer Block#

Pre-Norm · ResidualsMathematical 1 min read
◆ The PatternThe fundamental repeating unit of every LLM

A transformer block combines attention and FFN with residual connections and layer normalization. Modern LLMs use Pre-Norm (normalize before each sublayer) rather than Post-Norm, which stabilizes training at scale.

Pre-Norm block: x → x + Attn(LN(x)) → x + FFN(LN(x))
LayerNorm before sublayer, residual after. Gradients flow cleanly through the skip connection.
RMSNorm(x) = x / RMS(x) · γ where RMS(x) = √(mean(x²))
Root Mean Square normalization — no mean subtraction. Faster, used in LLaMA, Mistral.

A 70B model stacks 80 blocks. Each block adds about 856M parameters (Llama 2 70B). The residual stream acts as a highway — early layers write features, later layers read and refine them. This is the residual stream mental model.

The residual connection is why deep transformers work at all. Without it, gradients vanish through 80+ layers. With it, there's always a direct path from output to any layer.
Interactive — data flow through a transformer block

Python — transformer block#

Pattern bridge: The repeated unit: attention → add+norm → FFN → add+norm. In ML math, the residual connections prevent gradient vanishing.
Worked example

Counting one block of Llama 2 70B: attention 151.0M parameters (grouped keys and values make it lighter), the SwiGLU feed-forward 704.6M, two norms almost nothing — 855.7M per block. Eighty blocks plus the two embedding tables give 68,976,648,192: the “70B”. Four fifths of each block is the feed-forward layer.

Where it misleads
  • Parameter count is not compute per token for mixture-of-experts models, where only some blocks’ experts run.
  • Pre-norm trains more stably than post-norm (Xiong et al. 2020), but very deep pre-norm models can under-use their later layers.
  • The “residual stream” picture is a useful lens for interpretability, not a description of what each layer does.
In code
d, kv_dim, ff, layers, vocab = 8_192, 1_024, 28_672, 80, 32_000   # Llama-2-70B
attn = d * d + 2 * d * kv_dim + d * d          # Q, K and V (grouped), output
ffn = 3 * d * ff                                # SwiGLU
block = attn + ffn + 2 * d                      # + two RMSNorm scales
total = layers * block + 2 * vocab * d + d      # + input and output embeddings, final norm
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, d_model, n_heads, n_kv_heads):
        super().__init__()
        self.ln1 = nn.RMSNorm(d_model)
        self.attn = GQA(d_model, n_heads, n_kv_heads)
        self.ln2 = nn.RMSNorm(d_model)
        self.ffn = SwiGLU_FFN(d_model)

    def forward(self, x):
        x = x + self.attn(self.ln1(x))   # attention + residual
        x = x + self.ffn(self.ln2(x))    # FFN + residual
        return x
Sources
  • H. Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models”, arXiv:2307.09288, 2023
  • R. Xiong et al., “On Layer Normalization in the Transformer Architecture”, ICML, 2020
  • N. Elhage et al., “A Mathematical Framework for Transformer Circuits”, Anthropic, transformer-circuits.pub, 2021

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 1
← Previous
Feed-Forward Networks
Open in the full reader, with the topic sidebar