PagedAttention · vLLM✓ Mathematical
◆ The PatternEliminating the memory fragmentation that limits batch size
Standard KV-cache allocates a contiguous buffer for the maximum sequence length per request — this wastes memory (most sequences are shorter). PagedAttention (vLLM) allocates KV-cache in fixed-size blocks (like OS virtual memory pages), eliminating fragmentation.
PagedAttention: KV-cache = non-contiguous blocks of size B tokens each
Physical blocks allocated on demand. Block table maps logical → physical. No wasted pre-allocation.
Prefix caching: shared prompts → shared KV blocks (copy-on-write)
If 100 requests share a system prompt, cache its KV once. Saves ~50% memory for chat workloads.
Impact: vLLM achieves 2–4× higher throughput than naive serving by fitting more requests in the same GPU memory. Additional optimizations: KV-cache quantization (FP8 per KV), sliding window eviction, and radix tree prefix sharing.
PagedAttention is the single most impactful inference optimization for serving LLMs at scale. It's why vLLM, TGI, and SGLang are the standard serving frameworks.
Interactive — paged vs contiguous KV-cache allocation
Python — vLLM serving#
from vllm import LLM, SamplingParams
# vLLM handles PagedAttention automatically
llm = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
tensor_parallel_size=1,
gpu_memory_utilization=0.9, # use 90% of GPU for KV-cache
enable_prefix_caching=True, # share KV for common prefixes
)
params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)
outputs = llm.generate(["Explain PagedAttention"], params)
print(outputs[0].outputs[0].text)Pattern bridge: Grouped-query and multi-query attention reduce memory by sharing keys/values. The same rank reduction principle behind SVD and LoRA.