LLM Engineering

A 70B model needs 140 GB in fp16, 70 GB in int8 and 35 GB in int4. How it is quantized matters as much as the bit width: with a few outlier weights, one int4 scale for the whole tensor gives an average error of 87% of a weight’s size; a separate scale per group of 128 weights cuts it to 30%. Group-wise scales are why 4-bit models work.

• Average reconstruction error is not task quality; evaluate on the tasks you care about, including long-context and maths, which degrade first.
• Weight-only quantization saves memory and bandwidth; it speeds up compute-bound batches much less.
• Below 4 bits quality usually falls quickly (Dettmers & Zettlemoyer 2023).

sizes = {bits: 70e9 * bits / 8 / 1e9 for bits in (16, 8, 4)}   # a 70B model, GB
rng = np.random.default_rng(1)
w = rng.normal(0, 0.02, 4_096)
w[::512] = 0.5                                    # a few outlier weights

def int4_error(w, group):
    g = w.reshape(-1, group)
    scale = np.abs(g).max(axis=1, keepdims=True) / 7
    return np.abs(np.clip(np.round(g / scale), -7, 7) * scale - g).mean() / np.abs(w).mean()

per_tensor, per_group = int4_error(w, 4_096), int4_error(w, 128)

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# 4-bit quantization (NF4 for QLoRA)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,  # quantize the quantization constants
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B",
    quantization_config=bnb_config,
    device_map="auto"
)
# 70B model now fits in ~35GB VRAMCopy