24 — Inference

Batching & Throughput#

Continuous Batching · Prefill vs Decode✓ Mathematical
◆ The PatternServing hundreds of concurrent requests efficiently

LLM inference has two distinct phases: prefill (process the full prompt — compute-bound, fast per token) and decode (generate one token — memory-bound, slow per token). Continuous batching dynamically adds/removes requests from a batch as they finish, rather than waiting for the longest sequence.

Static batching: all requests padded to same length, wait for slowest
Wastes GPU cycles on padding. Throughput = 1/longest_sequence.
Continuous batching: new requests join mid-batch, finished ones leave
GPU is always busy. Throughput improves 10-20× compared to static batching.

Key metrics: TTFT (time to first token — mainly prefill), TPS (tokens per second — decode speed), throughput (total tokens/sec across all requests). Disaggregating prefill and decode to separate GPU pools (prefill cluster + decode cluster) is the latest frontier.

The fundamental LLM serving insight: prefill is compute-bound, decode is memory-bound. They have opposite optimization strategies. Modern serving engines schedule them separately.
Interactive — static vs continuous batching timeline

Python — throughput benchmark#

import time, asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1")

async def single_request(prompt):
    start = time.monotonic()
    resp = await client.completions.create(
        model="llama-3.1-8b", prompt=prompt,
        max_tokens=256, temperature=0.7)
    elapsed = time.monotonic() - start
    tokens = resp.usage.completion_tokens
    return tokens, elapsed

async def benchmark(n_concurrent=32):
    prompts = ["Explain transformers in detail."] * n_concurrent
    tasks = [single_request(p) for p in prompts]
    results = await asyncio.gather(*tasks)
    total_tokens = sum(r[0] for r in results)
    wall_time = max(r[1] for r in results)
    print(f"Throughput: {total_tokens/wall_time:.0f} tok/s")
Pattern bridge: Processing multiple requests simultaneously for throughput. In mini-batch gradient descent, the same principle trades per-sample accuracy for throughput.
← Previous
KV-Cache Optimization
Open in the full reader, with the topic sidebar →