24 — Inference

Batching & Throughput#

Continuous Batching · Prefill vs DecodePractice 1 min read
◆ The PatternServing hundreds of concurrent requests efficiently

LLM inference has two distinct phases: prefill (process the full prompt — compute-bound, fast per token) and decode (generate one token — memory-bound, slow per token). Continuous batching dynamically adds/removes requests from a batch as they finish, rather than waiting for the longest sequence.

Static batching: all requests padded to same length, wait for slowest
Wastes GPU cycles on padding. Throughput = 1/longest_sequence.
Continuous batching: new requests join mid-batch, finished ones leave
GPU is always busy. Throughput can rise several-fold over static batching (Yu et al. 2022).

Key metrics: TTFT (time to first token — mainly prefill), TPS (tokens per second — decode speed), throughput (total tokens/sec across all requests). Disaggregating prefill and decode to separate GPU pools (prefill cluster + decode cluster) is the latest frontier.

The fundamental LLM serving insight: prefill is compute-bound, decode is memory-bound. They have opposite optimization strategies. Modern serving engines schedule them separately.
Interactive — static vs continuous batching timeline

Python — throughput benchmark#

Pattern bridge: Processing multiple requests simultaneously for throughput. In mini-batch gradient descent, the same principle trades per-sample accuracy for throughput.
Worked example

25 batches of 8 requests with output lengths between 20 and 600 tokens. With static batching each batch runs until its longest request finishes, and on average only 59% of the slots are doing useful work. Continuous batching refills a slot as soon as a request ends, approaching 100%.

Where it misleads
  • Higher throughput can mean worse latency for each user; batch size is a trade-off, not a free gain.
  • Prefill (reading the prompt) and decode compete for the same GPU; long prompts stall everyone’s generation unless the two are separated.
  • Benchmarks with uniform request lengths overstate real throughput.
In code
rng = np.random.default_rng(3)
out = rng.integers(20, 600, size=(25, 8))              # output lengths: 25 batches of 8 requests
static_util = (out.sum(1) / (8 * out.max(1))).mean()    # each batch runs until its longest request ends
continuous_util = 1.0                                   # finished slots refilled at once (the ideal)
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
import time, asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1")

async def single_request(prompt):
    start = time.monotonic()
    resp = await client.completions.create(
        model="llama-3.1-8b", prompt=prompt,
        max_tokens=256, temperature=0.7)
    elapsed = time.monotonic() - start
    tokens = resp.usage.completion_tokens
    return tokens, elapsed

async def benchmark(n_concurrent=32):
    prompts = ["Explain transformers in detail."] * n_concurrent
    tasks = [single_request(p) for p in prompts]
    results = await asyncio.gather(*tasks)
    total_tokens = sum(r[0] for r in results)
    wall_time = max(r[1] for r in results)
    print(f"Throughput: {total_tokens/wall_time:.0f} tok/s")
Sources
  • G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim & B.-G. Chun, “Orca: A Distributed Serving System for Transformer-Based Generative Models”, OSDI, 2022
  • W. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP, 2023
  • Y. Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”, OSDI, 2024

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 1
← Previous
KV-Cache Optimization
Open in the full reader, with the topic sidebar