23 — Inference

KV-Cache Optimization#

PagedAttention · vLLMMathematical 1 min read
◆ The PatternEliminating the memory fragmentation that limits batch size

Standard KV-cache allocates a contiguous buffer for the maximum sequence length per request — this wastes memory (most sequences are shorter). PagedAttention (vLLM) allocates KV-cache in fixed-size blocks (like OS virtual memory pages), eliminating fragmentation.

PagedAttention: KV-cache = non-contiguous blocks of size B tokens each
Physical blocks allocated on demand. Block table maps logical → physical. No wasted pre-allocation.
Prefix caching: shared prompts → shared KV blocks (copy-on-write)
If 100 requests share a system prompt, its KV is stored once instead of 100 times; the saving grows with the prompt’s share of each request.

Impact: vLLM achieves 2–4× higher throughput than naive serving by fitting more requests in the same GPU memory. Additional optimizations: KV-cache quantization (FP8 per KV), sliding window eviction, and radix tree prefix sharing.

PagedAttention is one of the most important serving optimisations of recent years; vLLM introduced it, and other serving frameworks have adopted similar paged KV-caches.
Interactive — paged vs contiguous KV-cache allocation

Python — vLLM serving#

Pattern bridge: Grouped-query and multi-query attention reduce memory by sharing keys/values. The same rank reduction principle behind SVD and LoRA.
Worked example

64 live requests whose lengths vary around 400 tokens hold 36,363 tokens of cache. Reserving a buffer for the 4,096-token maximum per request uses only 13.9% of the reserved memory; allocating 16-token pages as needed uses 98.6%. That difference is how PagedAttention fits several times more requests on the same GPU.

Where it misleads
  • Paging removes fragmentation, not the cache itself; long contexts still need the memory.
  • Evicting or compressing cache entries to save memory can drop information the model needed later.
  • Prefix sharing only helps when many requests start with exactly the same tokens.
In code
rng = np.random.default_rng(2)
lengths = np.clip(rng.lognormal(np.log(400), 0.8, 64).astype(int), 16, 4_096)   # 64 live requests
reserved_contiguous = 64 * 4_096                         # a buffer for the maximum length each
reserved_paged = (np.ceil(lengths / 16) * 16).sum()      # 16-token blocks, allocated as needed
used = lengths.sum()
util = (used / reserved_contiguous, used / reserved_paged)
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
from vllm import LLM, SamplingParams

# vLLM handles PagedAttention automatically
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.9,  # use 90% of GPU for KV-cache
    enable_prefix_caching=True,  # share KV for common prefixes
)

params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)
outputs = llm.generate(["Explain PagedAttention"], params)
print(outputs[0].outputs[0].text)
Sources
  • W. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP, 2023
  • L. Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs”, arXiv:2312.07104, 2024 — prefix caching

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 1
← Previous
Quantization
Open in the full reader, with the topic sidebar