Linear Algebra✓ Mathematical
◆ The PatternThe language of ML — every model is matrix multiplication under the hood
Machine learning is built on linear algebra. Inputs are vectors, weights are matrices, and the forward pass is matrix multiplication. Understanding dot products, shapes, and transposes is non-negotiable.
a · b = Σᵢ aᵢbᵢ = |a||b|cos(θ)
Dot product = sum of element-wise products = measures alignment between vectors
C = A × B Cᵢⱼ = Σₖ Aᵢₖ · Bₖⱼ
Matrix multiply: A is (m×k), B is (k×n) → C is (m×n). Inner dimensions must match.
(AB)ᵀ = BᵀAᵀ (A⁻¹)⁻¹ = A
Transpose reverses multiplication order. Only square, full-rank matrices are invertible.
// Interactive 2D vectors — drag to change, see dot product
Vector A angle30°
Vector B angle80°
Dot Product—
| Operation | Shape | Usage |
|---|---|---|
| Dot product a·b | (n,)·(n,)→scalar | Similarity, projections |
| Matrix-vector Ax | (m,n)·(n,)→(m,) | Linear layer (no batch) |
| Matrix multiply AB | (m,k)·(k,n)→(m,n) | Batched forward passes |
| Hadamard A⊙B | (m,n)⊙(m,n)→(m,n) | Gating (LSTM, attention masks) |
| Outer product abᵀ | (m,)·(n,)→(m,n) | Rank-1 updates, LoRA |
# PyTorch matrix operations a = torch.randn(3) b = torch.randn(3) dot = torch.dot(a, b) # scalar C = A @ B # matrix multiply C = torch.matmul(A, B) # same thing D = A * B # element-wise (Hadamard) E = torch.outer(a, b) # outer product
Shape debugging: Most PyTorch errors are shape mismatches. Use
tensor.shape liberally. The rule: (…, m, k) @ (…, k, n) → (…, m, n).Pattern bridge: Dot products measure alignment here and in cosine similarity for statistics. The same operation that scores attention weights in a transformer also measures how two price series co-move in market correlation.