Sinusoidal · RoPE · ALiBi✓ Mathematical
◆ The PatternGiving transformers a sense of token order
Self-attention is permutation-equivariant — it treats "the cat sat" identically to "sat cat the" without positional information. We need to inject position somehow.
Sinusoidal: PE(pos,2i) = sin(pos / 10000^(2i/d)) PE(pos,2i+1) = cos(pos / 10000^(2i/d))
Original Transformer (2017). Fixed, not learned. Each dimension has a different frequency.
RoPE: q'ₘ = Rθ,m · qₘ k'ₙ = Rθ,n · kₙ → q'ₘᵀk'ₙ depends on (m−n)
Rotary Position Embedding. Rotates Q,K vectors — attention depends on relative distance. Used in LLaMA, Mistral.
ALiBi: attention(i,j) = qᵢᵀkⱼ − m · |i − j|
Attention with Linear Biases. No learned parameters — just subtract a slope × distance. Used in BLOOM.
RoPE dominates modern LLMs because it encodes relative position and can be extrapolated beyond training length via NTK-aware scaling or YaRN.
Learned absolute position embeddings (GPT-2) can't extrapolate beyond training length. Relative methods (RoPE, ALiBi) handle longer sequences naturally.
Interactive — positional encoding patterns
Python — sinusoidal positional encoding#
import torch, math
def sinusoidal_pe(max_len, d_model):
pe = torch.zeros(max_len, d_model)
pos = torch.arange(max_len).unsqueeze(1).float()
div = torch.exp(torch.arange(0, d_model, 2).float()
* (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(pos * div)
pe[:, 1::2] = torch.cos(pos * div)
return pe # shape: (max_len, d_model)
pe = sinusoidal_pe(512, 256)
# pe[pos] gives the encoding vector for position posPattern bridge: Injecting position via sinusoids so the model knows word order. In markets, Aroon encodes "time since" as position information.