LoRA & QLoRA — LLM Engineering

15 — Training

LoRA & QLoRA#

Parameter-EfficientMathematical 1 min read

◆ The PatternFine-tune a 70B model on a single GPU

LoRA (Low-Rank Adaptation) freezes all pre-trained weights and injects small trainable rank-decomposition matrices. Instead of updating W ∈ ℝ^(d×d), we learn W' = W + BA where B ∈ ℝ^(d×r), A ∈ ℝ^(r×d) with rank r ≪ d (typically 8–64).

W' = W₀ + (α/r) · B · A where B ∈ ℝ^(d×r), A ∈ ℝ^(r×d)

Only B and A are trained. W₀ stays frozen. α/r scales the adapter contribution.

Trainable params: 2 · d · r per adapter ≪ d²

For d=4096, r=16: 131K params per adapter vs 16.7M for the full matrix — 128× reduction.

QLoRA goes further: quantize W₀ to 4-bit (NF4 format), keep adapters in bf16, and use paged optimizers to handle memory spikes. This makes fine-tuning a 65B model possible on a single 48GB GPU.

LoRA adapters can be merged back into the base weights for zero-cost inference: W_merged = W₀ + (α/r)·B·A. Multiple LoRA adapters can be served simultaneously by switching the small adapter weights per request.

Interactive — low-rank decomposition

Rank r: Alpha α:

Python — LoRA with PEFT#

Pattern bridge: Low-rank updates that modify a frozen model with minimal parameters. The ML math behind LoRA is SVD-inspired rank reduction.

Worked example

Full fine-tuning of a 65B model with Adam needs about 16 bytes per parameter — fp16 weights and gradients, an fp32 master copy and two Adam moments — about 1,040 GB. QLoRA freezes the weights at 4 bits (32.5 GB) and trains rank-64 adapters on four projections in 80 layers: 336M parameters, about 5.4 GB with their optimizer state. That is how a 65B model fits on a single 48 GB GPU.

Where it misleads

Activations and the KV-cache also need memory; long sequences can still overflow a GPU that holds the weights.
Low-rank updates are a constraint; tasks far from the base model may need higher rank or full fine-tuning.
Quantization error in the frozen base adds noise the adapters must work around.

In code

params = 65e9
full_adam = params * (2 + 2 + 4 + 4 + 4) / 1e9   # fp16 weights + grads, fp32 master copy + Adam m and v
qlora_base = params * 0.5 / 1e9                   # 4-bit frozen weights
adapters = 2 * 8_192 * 64 * 4 * 80                 # rank-64 adapters on 4 projections in 80 layers
qlora_adapters = adapters * (2 + 2 + 4 + 4 + 4) / 1e9

Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                          # rank
    lora_alpha=32,                 # scaling factor
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable: 83M / 7B total = 1.2%

Sources

E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR, 2022
T. Dettmers, A. Pagnoni, A. Holtzman & L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs”, NeurIPS, 2023
S. Rajbhandari, J. Rasley, O. Ruwase & Y. He, “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”, SC20, 2020 — the 16-bytes-per-parameter accounting

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 2

← Previous

Fine-Tuning

RLHF

Open in the full reader, with the topic sidebar