Latency is the time for a single prediction. Throughput is predictions per second. You can often trade one for the other — dynamic batching increases throughput but adds latency. p50 tells you the typical experience; p99 tells you the worst.
100,000 simulated requests have a median of 40 ms and a 99th percentile of 162 ms. If each request waits on ten backend calls that are each slow 1% of the time, 9.6% of requests hit at least one slow call — the tail of the parts becomes the body of the whole. And by Little’s law, 200 requests a second at 50 ms means 10 in flight at any moment.
- Averages hide the tail; report percentiles, and know which percentile the SLO is about.
- Percentiles cannot be averaged across servers or time windows; aggregate the raw histograms instead.
- Load tests with a fixed number of clients under-report latency when the system slows (coordinated omission).
rng = np.random.default_rng(0) lat = rng.lognormal(mean=np.log(40), sigma=0.6, size=100_000) # simulated latencies in ms p50, p99 = np.percentile(lat, [50, 99]) slow_one = 0.01 # each backend call is slow 1% of the time slow_fanout = 1 - (1 - slow_one) ** 10 # a request that waits on 10 such calls in_flight = 200 * 0.050 # Little: 200 requests/s x 50 ms = requests in progress
Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
- J. Dean & L. A. Barroso, “The Tail at Scale”, Communications of the ACM 56(2), 2013
- J. D. C. Little, “A Proof for the Queuing Formula: L = λW”, Operations Research 9(3), 1961
- Site Reliability Engineering: How Google Runs Production Systems, B. Beyer, C. Jones, J. Petoff & N. R. Murphy (eds.), O’Reilly, 2016
Last reviewed October 2026 · How this site checks its content · Report a mistake