Parameter-Efficient✓ Mathematical
◆ The PatternFine-tune a 70B model on a single GPU
LoRA (Low-Rank Adaptation) freezes all pre-trained weights and injects small trainable rank-decomposition matrices. Instead of updating W ∈ ℝ^(d×d), we learn W' = W + BA where B ∈ ℝ^(d×r), A ∈ ℝ^(r×d) with rank r ≪ d (typically 8–64).
W' = W₀ + (α/r) · B · A where B ∈ ℝ^(d×r), A ∈ ℝ^(r×d)
Only B and A are trained. W₀ stays frozen. α/r scales the adapter contribution.
Trainable params: 2 · d · r per adapter ≪ d²
For d=4096, r=16: 131K params per adapter vs 16.7M for the full matrix — 128× reduction.
QLoRA goes further: quantize W₀ to 4-bit (NF4 format), keep adapters in bf16, and use paged optimizers to handle memory spikes. This makes fine-tuning a 65B model possible on a single 48GB GPU.
LoRA adapters can be merged back into the base weights for zero-cost inference: W_merged = W₀ + (α/r)·B·A. Multiple LoRA adapters can be served simultaneously by switching the small adapter weights per request.
Interactive — low-rank decomposition
Python — LoRA with PEFT#
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16, # rank
lora_alpha=32, # scaling factor
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable: 83M / 7B total = 1.2%Pattern bridge: Low-rank updates that modify a frozen model with minimal parameters. The ML math behind LoRA is SVD-inspired rank reduction.