Fine-tuningMathematical
1 min read
◆ The PatternTraining billion-parameter models by updating only tiny low-rank matrices
LoRA freezes the original weights and injects small trainable rank-r matrices alongside each layer. Instead of updating a d×d weight matrix (millions of params), you update d×r + r×d (thousands).
W' = W + ΔW = W + B·A B ∈ ℝ^(d×r), A ∈ ℝ^(r×d)
W = frozen original weights | B·A = low-rank update | r = rank (4–64 typical)
Params: d² → 2·d·r (e.g., 4096² = 16.7M → 2·4096·16 = 131K)
A 99.2% reduction in trainable parameters for rank r=16
// Parameter savings — full fine-tuning vs LoRA
Model dim d4096
Rank r16
Savings—
QLoRA takes this further: quantise the frozen weights to 4-bit, then apply LoRA. This enables fine-tuning a 65B model on a single 48GB GPU.
Pattern bridge: Low-rank adaptation fine-tunes with tiny matrices — the same rank reduction as principal components. LoRA in LLM engineering is the applied version.
Worked example
A 4096×4096 weight matrix has 16,777,216 parameters. A rank-16 LoRA update trains two thin matrices with 131,072 — 0.78% as many. The update B·A can only have rank 16 (in the small check, rank 4 for r = 4), which is the bet: that fine-tuning changes lie in a low-dimensional subspace.
Where it misleads
- The low-rank bet can fail for tasks far from pre-training; full fine-tuning can still win there.
- The rank and the scaling factor α interact; changing one without the other changes the effective learning rate.
- LoRA saves training memory, not inference cost, unless the update is merged into the weights.
In code
d, r = 4096, 16 full = d * d lora = 2 * d * r # B (d x r) and A (r x d) rng = np.random.default_rng(10) B, A = rng.normal(size=(64, 4)), rng.normal(size=(4, 64)) rank = np.linalg.matrix_rank(B @ A) # an update of rank at most r
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
# Using PEFT library from peft import LoraConfig, get_peft_model config = LoraConfig( r=16, # rank lora_alpha=32, # scaling factor target_modules=["q_proj", "v_proj"], # which layers lora_dropout=0.05, ) model = get_peft_model(base_model, config) model.print_trainable_parameters() # "trainable: 0.1% of total"
Sources
- E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR, 2022
- A. Aghajanyan, S. Gupta & L. Zettlemoyer, “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning”, ACL, 2021
- T. Dettmers, A. Pagnoni, A. Holtzman & L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs”, NeurIPS, 2023
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1