Fine-tuning✓ Mathematical
◆ The PatternTraining billion-parameter models by updating only tiny low-rank matrices
LoRA freezes the original weights and injects small trainable rank-r matrices alongside each layer. Instead of updating a d×d weight matrix (millions of params), you update d×r + r×d (thousands).
W' = W + ΔW = W + B·A B ∈ ℝ^(d×r), A ∈ ℝ^(r×d)
W = frozen original weights | B·A = low-rank update | r = rank (4–64 typical)
Params: d² → 2·d·r (e.g., 4096² = 16.7M → 2·4096·16 = 131K)
A 99.2% reduction in trainable parameters for rank r=16
// Parameter savings — full fine-tuning vs LoRA
Model dim d4096
Rank r16
Savings—
# Using PEFT library from peft import LoraConfig, get_peft_model config = LoraConfig( r=16, # rank lora_alpha=32, # scaling factor target_modules=["q_proj", "v_proj"], # which layers lora_dropout=0.05, ) model = get_peft_model(base_model, config) model.print_trainable_parameters() # "trainable: 0.1% of total"
QLoRA takes this further: quantise the frozen weights to 4-bit, then apply LoRA. This enables fine-tuning a 65B model on a single 48GB GPU.
Pattern bridge: Low-rank adaptation fine-tunes with tiny matrices — the same rank reduction as principal components. LoRA in LLM engineering is the applied version.