Performance✓ Mathematical
◆ The Patternp50, p99, batching, and the tradeoffs that define your SLA
Latency is the time for a single prediction. Throughput is predictions per second. You can often trade one for the other — dynamic batching increases throughput but adds latency. p50 tells you the typical experience; p99 tells you the worst.
Throughput = Batch Size / Latency
Batching amortises fixed overhead (model loading, context switching) across multiple inputs, improving throughput at the cost of per-request latency.
// Interactive — latency distribution with batch size
Batch Size1
p50—
p99—
Pattern bridge: Latency percentiles work exactly like distribution shape analysis — the tail matters more than the mean. In markets, the equivalent is tail risk in volatility.