INT8 · INT4 · GPTQ · AWQ✓ Mathematical
1 min read
◆ The PatternShrink models 2–4× with minimal quality loss
Quantization reduces the precision of model weights from 16-bit floats to 8-bit or 4-bit integers. This halves (or quarters) memory usage and speeds up memory-bound inference. The challenge: preserving output quality.
Linear quantization: q = round((x − zero) / scale) x ≈ q · scale + zero
Map continuous weights to discrete integer grid. Scale and zero-point define the mapping.
Model sizes: FP16=2B/param → INT8=1B/param → INT4=0.5B/param
A 70B model: 140GB (FP16) → 70GB (INT8) → 35GB (INT4). Fits on 2× A100 vs 4× A100.
Methods: (1) GPTQ — weight-only, layer-by-layer with Hessian info, (2) AWQ — activation-aware, protects salient weights, (3) GGUF — CPU-friendly mixed-precision, (4) bitsandbytes — NF4 datatype for QLoRA. Weight-only quantization (activations stay in fp16) is most common for LLMs.
INT8 quantization has virtually no quality loss for most tasks. INT4 shows small degradation but is the sweet spot for serving — 4× memory savings are too compelling to ignore.
Interactive — precision comparison
Python — quantize with bitsandbytes#
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
# 4-bit quantization (NF4 for QLoRA)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, # quantize the quantization constants
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-70B",
quantization_config=bnb_config,
device_map="auto"
)
# 70B model now fits in ~35GB VRAMPattern bridge: Reducing precision from float32 to int8/int4 is binning applied to weights — discrete approximation of continuous values.
Performance in practice
- GPTQ 4-bit loses <1% accuracy on most benchmarks vs FP16. GGUF format (llama.cpp) runs 70B models on a MacBook with 64GB RAM
- Memory savings: FP32→FP16 = 2x, FP16→INT8 = 2x, INT8→INT4 = 2x. A 70B model goes from 280GB → 35GB
- Inference speed: INT4 on GPU is 2-3x faster than FP16 because memory bandwidth is the bottleneck, not compute
- AWQ (Activation-aware Weight Quantization) preserves salient weights at higher precision — better quality than naive rounding
When to use this
✓ Use when: Deploying models locally or on limited hardware. Reducing inference costs in production. Running large models (30B+) on consumer GPUs. Edge deployment on phones/laptops.
✗ Skip when: Using cloud APIs (already optimized). Tasks requiring maximum precision (scientific computation). Small models (<1B) that already fit in memory. Training — quantize for inference only.