02 — Foundations

Linear Regression#

SupervisedMathematical
◆ The PatternFitting a line through data — the simplest predictive model

Linear regression finds the best-fit line through data. Given input x, we predict ŷ by learning weight w (slope) and bias b (intercept) that minimise prediction error.

ŷ = w · x + b
ŷ = prediction  |  w = weight  |  x = input  |  b = bias
MSE = (1/n) · Σ (yᵢ − ŷᵢ)²
MSE = Mean Squared Error loss  |  minimised by gradient descent
// Interactive — drag sliders to fit the line
Weight w0.50
Bias b0.00
MSE—
OLS Solution: For simple linear regression, the optimal weights have a closed-form: w = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)². Gradient descent finds the same answer iteratively.
Pattern bridge: Fitting a line is the same act everywhere — linear regression in statistics, simple moving averages in markets. Wrap the same linear score in a sigmoid and it becomes a classifier: logistic regression.
Worked example

Six points that lie roughly on y = 2x. Least squares finds slope 2.02 and intercept −0.02, with a mean squared error of 0.021. With one input and a closed-form solution there is nothing to tune: the fit is the answer to a single matrix equation.

Where it misleads
  • A good fit does not mean a straight line is right; always plot the residuals.
  • Least squares is pulled hard by outliers, because errors are squared (see Loss Functions).
  • Coefficients are only interpretable as effects when inputs are not strongly correlated (see Regularization).
In code
x = np.array([1, 2, 3, 4, 5, 6], dtype=float)
y = np.array([2.1, 3.9, 6.2, 7.8, 10.1, 12.2])            # made up, roughly y = 2x
X = np.column_stack([x, np.ones_like(x)])
(w, b), *_ = np.linalg.lstsq(X, y, rcond=None)             # least squares, closed form
mse = np.mean((y - (w * x + b)) ** 2)
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources
  • The Elements of Statistical Learning (2nd ed.), T. Hastie, R. Tibshirani & J. Friedman, Springer, 2009 — chapter 3
  • Pattern Recognition and Machine Learning, C. M. Bishop, Springer, 2006 — chapter 3

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 4
← Previous
Vectors & Matrices
Open in the full reader, with the topic sidebar