Continuous Batching · Prefill vs Decode✓ Mathematical
◆ The PatternServing hundreds of concurrent requests efficiently
LLM inference has two distinct phases: prefill (process the full prompt — compute-bound, fast per token) and decode (generate one token — memory-bound, slow per token). Continuous batching dynamically adds/removes requests from a batch as they finish, rather than waiting for the longest sequence.
Static batching: all requests padded to same length, wait for slowest
Wastes GPU cycles on padding. Throughput = 1/longest_sequence.
Continuous batching: new requests join mid-batch, finished ones leave
GPU is always busy. Throughput improves 10-20× compared to static batching.
Key metrics: TTFT (time to first token — mainly prefill), TPS (tokens per second — decode speed), throughput (total tokens/sec across all requests). Disaggregating prefill and decode to separate GPU pools (prefill cluster + decode cluster) is the latest frontier.
The fundamental LLM serving insight: prefill is compute-bound, decode is memory-bound. They have opposite optimization strategies. Modern serving engines schedule them separately.
Interactive — static vs continuous batching timeline
Python — throughput benchmark#
import time, asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8000/v1")
async def single_request(prompt):
start = time.monotonic()
resp = await client.completions.create(
model="llama-3.1-8b", prompt=prompt,
max_tokens=256, temperature=0.7)
elapsed = time.monotonic() - start
tokens = resp.usage.completion_tokens
return tokens, elapsed
async def benchmark(n_concurrent=32):
prompts = ["Explain transformers in detail."] * n_concurrent
tasks = [single_request(p) for p in prompts]
results = await asyncio.gather(*tasks)
total_tokens = sum(r[0] for r in results)
wall_time = max(r[1] for r in results)
print(f"Throughput: {total_tokens/wall_time:.0f} tok/s")Pattern bridge: Processing multiple requests simultaneously for throughput. In mini-batch gradient descent, the same principle trades per-sample accuracy for throughput.