GPUs accelerate inference through massive parallelism, but they're expensive and tricky to manage. TensorRT optimises models for NVIDIA GPUs with layer fusion and kernel auto-tuning. Triton Inference Server handles multi-model scheduling and dynamic batching on GPU.
| Approach | Speedup (typical reported; model-dependent) | Tradeoff |
|---|---|---|
| TensorRT FP16 | 2–4× | Slight accuracy loss, compile time |
| TensorRT INT8 | 3–6× | Needs calibration dataset |
| Multi-model GPU | Better utilisation | Memory contention |
| CPU (small models) | Baseline | Often sufficient for <10M params |
A 7-billion-parameter model in fp16 is 14 GB of weights. Generating a token reads every weight once, so with about 2 TB/s of memory bandwidth one sequence gets at most about 143 tokens/s, however fast the arithmetic is. Batching 16 sequences reuses each read, for up to about 2,286 tokens/s — until the GPU becomes compute-bound or runs out of memory for the KV-cache.
- These are ceilings, not measurements; kernels, attention and the KV-cache take their share. Benchmark on your own traffic.
- Bigger batches raise throughput but also latency per request; the right batch size depends on the SLO.
- For small models on tabular data a CPU is often cheaper per prediction; the GPU pays off only when it is kept busy.
params = 7e9 # a 7B-parameter model bytes_per = 2 # fp16 weights_gb = params * bytes_per / 1e9 bandwidth_gb_s = 2000 # memory bandwidth of a modern data-centre GPU, ~2 TB/s tokens_s_batch1 = bandwidth_gb_s / weights_gb # each new token reads every weight once tokens_s_batch16 = 16 * tokens_s_batch1 # one read serves 16 sequences (until compute-bound)
Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
- S. Williams, A. Waterman & D. Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures”, Communications of the ACM 52(4), 2009
- R. Pope et al., “Efficiently Scaling Transformer Inference”, Proceedings of Machine Learning and Systems 5, 2023
Last reviewed October 2026 · How this site checks its content · Report a mistake