Quantization reduces weight precision from 32-bit floats to 16-bit or 8-bit integers. Post-training quantization (PTQ) is the quick path — just convert. Quantization-aware training (QAT) simulates low precision during training for better accuracy at INT8.
| Method | Accuracy impact (indicative) | Speed gain (indicative) |
|---|---|---|
| FP16 (half precision) | Negligible | ~2× |
| PTQ INT8 | 0.5–2% drop | ~3× |
| QAT INT8 | <0.5% drop | ~3× |
| INT4 (experimental) | Variable | ~5× |
10,000 weights rounded to 8-bit integers with one scale for the tensor: 4× less memory, and an average error of 0.9% of a weight’s size. Add a single outlier 50 standard deviations out and the scale stretches to fit it; the average error jumps to 12%. Outliers, not the bit-width itself, are what breaks naive quantization — which is why LLM.int8() treats them separately.
- Per-tensor scales are the crude version; per-channel or per-group scales handle uneven weights much better.
- Activations are often harder to quantize than weights.
- Small average errors can still flip close decisions; evaluate the task metric, not the reconstruction error.
rng = np.random.default_rng(5)
w = rng.normal(0, 0.02, 10_000).astype(np.float32)
def int8_error(w): # symmetric per-tensor INT8
scale = np.abs(w).max() / 127
q = np.clip(np.round(w / scale), -127, 127)
return np.abs(q * scale - w).mean() / np.abs(w).mean()
err = int8_error(w)
w_out = w.copy(); w_out[0] = 1.0 # one outlier weight, 50 standard deviations out
err_outlier = int8_error(w_out)
memory_saving = 32 / 8Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
- B. Jacob et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference”, CVPR, 2018
- T. Dettmers, M. Lewis, Y. Belkada & L. Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, NeurIPS, 2022
Last reviewed October 2026 · How this site checks its content · Report a mistake