In The Goal, a 1984 novel about a failing factory, Eliyahu Goldratt has his plant manager learn the lesson on a scout hike. The column of boys can only move as fast as the slowest one; hurrying the others just stretches the line. In the factory it is a single machine, and every improvement made anywhere else only grows the pile of parts waiting in front of it.
Two facts from queueing theory make this sharper. John Little showed in 1961 that, in any stable system, the number of things inside it equals the rate at which they arrive times how long each one stays. Queues and waiting times therefore rise and fall together. And waiting does not grow evenly with load. In the simplest queue, a server busy 50% of the time keeps each job waiting about as long as it takes to serve it; at 90% busy the time in the system is ten times the service time, at 95% twenty. The last few percent of utilisation are paid for in waiting.
Serving a model
The tension between latency and throughput is the bottleneck in miniature. Batching raises throughput by making requests wait for company; auto-scaling adds capacity at the constraint before the queue explodes; caching skips the slow step entirely for answers already known. Which serving pattern fits — online, batch or streaming — depends on where the slow step sits and who is waiting for it, and GPU inference is mostly the art of finding out whether the chip is waiting on arithmetic or on memory.
Generating text
For a large language model producing one token at a time, the bottleneck is usually not computation but moving the weights from memory for every token. That is why the fixes all target bytes and steps. Quantization and model compression shrink what has to be read; a mixture of experts reads only a few of the experts for each token; speculative decoding lets a small model draft several tokens that the large one checks in a single pass, so the slow step runs fewer times.
Pipelines and funnels
An ML pipeline finishes when its slowest chain of dependent steps finishes; orchestration can run everything else in parallel, but not shorten that chain. A feature store exists largely so the same expensive features are not recomputed by every model that needs them. Outside engineering the shape is identical: in a conversion funnel the narrowest step caps everything after it, and doubling the traffic at the top only doubles the crowd that drops out there.
Bottlenecks also move. Widen the slowest step and the second slowest becomes the constraint, often somewhere nobody was watching. That is the quiet logic behind thresholds and feedback loops in systems that seem to run smoothly right up until they do not.
Last reviewed October 2026 · How this site checks its content · Report a mistake