Linear AlgebraMathematical
1 min read
◆ The PatternDecomposing any matrix into rotation, scaling, and rotation
SVD factors any matrix into three parts: A = UΣVᵀ. It's the Swiss Army knife of linear algebra — used in PCA, compression, recommenders, and the mathematical foundation of LoRA.
A = UΣVᵀ
U = left singular vectors (m×m) | Σ = diagonal singular values | Vᵀ = right singular vectors (n×n)
A ≈ U_r · Σ_r · V_r^T (rank-r approximation)
Keep only top-r singular values → best rank-r approximation (Eckart-Young theorem)
// Low-rank approximation — how many singular values do you need?
Rank r3
Energy kept—
SVD → LoRA: LoRA exploits the fact that weight updates during fine-tuning are often low-rank. Instead of updating a full (d×d) matrix, it learns two small matrices (d×r) and (r×d) where r ≪ d. This is fundamentally SVD thinking.
Pattern bridge: Decomposing a matrix into rank-1 layers is the math behind factor analysis in statistics and LoRA’s low-rank updates.
Worked example
A 50×50 matrix whose singular values fall by 30% each. Keeping the top five captures 97.2% of its energy with a relative error of 16.8%, using 505 numbers instead of 2,500. The Eckart–Young theorem says no rank-5 matrix does better — the idea behind LoRA and compressed recommenders.
Where it misleads
- The approximation is only good when singular values decay; for noise-like matrices every component matters.
- Energy captured is not task performance; check the error on what the matrix is used for.
- Full SVD is expensive for large matrices; randomised or truncated methods are used in practice.
In code
rng = np.random.default_rng(8) U, _ = np.linalg.qr(rng.normal(size=(50, 50))) V, _ = np.linalg.qr(rng.normal(size=(50, 50))) s = 0.7 ** np.arange(50) # singular values that decay A = U @ np.diag(s) @ V.T k = 5 A_k = U[:, :k] @ np.diag(s[:k]) @ V[:, :k].T # best rank-5 approximation energy = (s[:k] ** 2).sum() / (s ** 2).sum() rel_err = np.linalg.norm(A - A_k) / np.linalg.norm(A) numbers = (50 * 50, k * (50 + 50 + 1)) # entries stored: full vs rank-5
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
U, S, Vt = torch.linalg.svd(A) # full SVD A_approx = U[:, :r] @ torch.diag(S[:r]) @ Vt[:r, :] # rank-r # NumPy equivalent U, s, Vt = np.linalg.svd(A, full_matrices=False)
Sources
- C. Eckart & G. Young, “The Approximation of One Matrix by Another of Lower Rank”, Psychometrika 1(3), 1936
- Matrix Computations (4th ed.), G. H. Golub & C. F. Van Loan, Johns Hopkins University Press, 2013
- Introduction to Linear Algebra (5th ed.), G. Strang, Wellesley-Cambridge Press, 2016
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1