GPT · LLaMA · Causal LM✓ Mathematical
◆ The PatternThe dominant architecture behind GPT, LLaMA, Mistral, and most modern LLMs
Decoder-only models use causal (left-to-right) masking so each token can only attend to itself and previous tokens — never the future. This enables autoregressive generation: predict next token, append, repeat.
Causal mask: M_ij = 0 if j ≤ i, else −∞
Upper triangle set to −∞ before softmax → zeroes out future attention weights.
P(text) = ∏ P(token_t | token_1, ..., token_{t-1})
Autoregressive factorization — the probability of text as a product of conditional probabilities.
Why decoder-only won: (1) simpler than encoder-decoder, (2) scales better with compute, (3) naturally handles both understanding and generation in a single architecture. The "decoder" name comes from the original Transformer paper where this half decoded outputs.
Every GPT, LLaMA, Mistral, Gemma, Claude, and most modern LLMs are decoder-only. The encoder-decoder style (T5, BART) is now mainly used for specialized tasks like translation.
Interactive — causal mask & autoregressive generation
Python — causal attention mask#
import torch
def causal_mask(seq_len):
"""Lower-triangular mask for causal (left-to-right) attention"""
mask = torch.triu(torch.ones(seq_len, seq_len), diagonal=1)
return mask.masked_fill(mask == 1, float('-inf'))
# Usage in attention
scores = Q @ K.T / d_k**0.5
scores = scores + causal_mask(seq_len) # mask future
weights = torch.softmax(scores, dim=-1)
output = weights @ VPattern bridge: Autoregressive generation — each token conditioned only on the past. In markets, recency bias makes traders decode only from recent history.