Long Context · RoPE Scaling✓ Mathematical
◆ The PatternHow much text can a model see at once — and how to push the limits
The context window is the maximum number of tokens a model can process in one forward pass. GPT-3 had 2K, GPT-4 Turbo has 128K, Gemini 1.5 has 1M+. Expanding context is critical for document analysis, code understanding, and long conversations.
Attention cost: O(n²) time, O(n) KV-cache memory
Quadratic compute + linear memory = context length is the fundamental bottleneck.
RoPE scaling: θ' = θ · α where α = target_len / train_len
NTK-aware interpolation stretches RoPE frequencies to extrapolate beyond training length.
Approaches to longer context: (1) RoPE scaling (NTK, YaRN) — cheapest, (2) Sliding window attention (Mistral) — each layer sees a local window, (3) Sparse attention patterns — attend to subset, (4) Ring attention — distribute across GPUs, (5) Simply train on more context.
Doubling context length quadruples attention compute but only doubles KV-cache memory. Long-context models primarily fight the compute cost, not the memory cost.
Interactive — attention patterns at different context lengths
Python — sliding window attention#
def sliding_window_mask(seq_len, window_size):
"""Causal + sliding window: attend to last W tokens only"""
causal = torch.triu(torch.ones(seq_len, seq_len), diagonal=1)
window = torch.tril(torch.ones(seq_len, seq_len),
diagonal=-window_size)
mask = causal + window
return mask.masked_fill(mask >= 1, float('-inf'))
# Mistral uses window_size=4096: each token attends to
# the previous 4096 tokens only, regardless of context lengthPattern bridge: The finite span of tokens the model can see at once. In statistics, sample size is the context window of inference — more data, better estimates.