Linear AlgebraMathematical
1 min read
◆ The PatternThe language of ML — every model is matrix multiplication under the hood
Machine learning is built on linear algebra. Inputs are vectors, weights are matrices, and the forward pass is matrix multiplication. Understanding dot products, shapes, and transposes is non-negotiable.
a · b = Σᵢ aᵢbᵢ = |a||b|cos(θ)
Dot product = sum of element-wise products = measures alignment between vectors
C = A × B Cᵢⱼ = Σₖ Aᵢₖ · Bₖⱼ
Matrix multiply: A is (m×k), B is (k×n) → C is (m×n). Inner dimensions must match.
(AB)ᵀ = BᵀAᵀ (A⁻¹)⁻¹ = A
Transpose reverses multiplication order. Only square, full-rank matrices are invertible.
// Interactive 2D vectors — drag to change, see dot product
Vector A angle30°
Vector B angle80°
Dot Product—
| Operation | Shape | Usage |
|---|---|---|
| Dot product a·b | (n,)·(n,)→scalar | Similarity, projections |
| Matrix-vector Ax | (m,n)·(n,)→(m,) | Linear layer (no batch) |
| Matrix multiply AB | (m,k)·(k,n)→(m,n) | Batched forward passes |
| Hadamard A⊙B | (m,n)⊙(m,n)→(m,n) | Gating (LSTM, attention masks) |
| Outer product abᵀ | (m,)·(n,)→(m,n) | Rank-1 updates, LoRA |
Shape debugging: Most PyTorch errors are shape mismatches. Use
tensor.shape liberally. The rule: (…, m, k) @ (…, k, n) → (…, m, n).Pattern bridge: Dot products measure alignment here and in cosine similarity for statistics. The same operation that scores attention weights in a transformer also measures how two price series co-move in market correlation.
Worked example
For a = (1, 2, 3) and b = (4, 5, 6) the dot product is 32; divided by their lengths it gives a cosine of 0.975, an angle of 12.9°. A 2×3 matrix times a 3×4 matrix gives a 2×4 result — the inner sizes must match and disappear. And (AB)ᵀ = BᵀAᵀ holds (True): transposing a product reverses its order.
Where it misleads
- Shape errors are the most common bug in ML code; broadcasting can hide them by silently stretching a vector instead of failing.
- The dot product mixes direction and length; for similarity you usually want the cosine.
- Intuitions from 2 or 3 dimensions mislead in high dimensions, where random vectors are almost always nearly perpendicular.
In code
a, b = np.array([1, 2, 3]), np.array([4, 5, 6]) dot = a @ b cos = dot / (np.linalg.norm(a) * np.linalg.norm(b)) angle = np.degrees(np.arccos(cos)) A, B = np.ones((2, 3)), np.ones((3, 4)) shape = (A @ B).shape # (2,3) x (3,4): inner sizes must match transpose_rule = np.allclose((A @ B).T, B.T @ A.T)
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
# PyTorch matrix operations a = torch.randn(3) b = torch.randn(3) dot = torch.dot(a, b) # scalar C = A @ B # matrix multiply C = torch.matmul(A, B) # same thing D = A * B # element-wise (Hadamard) E = torch.outer(a, b) # outer product
Sources
- Introduction to Linear Algebra (5th ed.), G. Strang, Wellesley-Cambridge Press, 2016
- Deep Learning, I. Goodfellow, Y. Bengio & A. Courville, MIT Press, 2016 — chapter 2
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1