Linear Algebra✓ Mathematical
◆ The PatternDecomposing any matrix into rotation, scaling, and rotation
SVD factors any matrix into three parts: A = UΣVᵀ. It's the Swiss Army knife of linear algebra — used in PCA, compression, recommenders, and the mathematical foundation of LoRA.
A = UΣVᵀ
U = left singular vectors (m×m) | Σ = diagonal singular values | Vᵀ = right singular vectors (n×n)
A ≈ U_r · Σ_r · V_r^T (rank-r approximation)
Keep only top-r singular values → best rank-r approximation (Eckart-Young theorem)
// Low-rank approximation — how many singular values do you need?
Rank r3
Energy kept—
U, S, Vt = torch.linalg.svd(A) # full SVD A_approx = U[:, :r] @ torch.diag(S[:r]) @ Vt[:r, :] # rank-r # NumPy equivalent U, s, Vt = np.linalg.svd(A, full_matrices=False)
SVD → LoRA: LoRA exploits the fact that weight updates during fine-tuning are often low-rank. Instead of updating a full (d×d) matrix, it learns two small matrices (d×r) and (r×d) where r ≪ d. This is fundamentally SVD thinking.
Pattern bridge: Decomposing a matrix into rank-1 layers is the math behind factor analysis in statistics and LoRA’s low-rank updates.