07 — Training

Loss Functions#

OptimizationMathematical 1 min read
◆ The PatternMeasuring how wrong the model is — the objective being minimised

The loss function defines what the model is optimising. Different tasks need different loss functions. Getting this wrong is one of the most common ML mistakes.

MSE = (1/n)·Σ(y−ŷ)²
Regression — squared penalty, sensitive to outliers
MAE = (1/n)·Σ|y−ŷ|
Robust regression — linear penalty, outlier-resistant
CE = −Σ yᵢ·log(ŷᵢ)
Classification — penalises confident wrong predictions exponentially
Huber = { ½(y−ŷ)² if |y−ŷ|≤δ, δ|y−ŷ|−½δ² otherwise }
Huber — smooth MSE near zero, MAE for large errors. Best of both.
// MSE vs MAE vs Huber — drag to see how penalty scales with error
Error magnitude1.00
MSE1.000
MAE1.000
Pattern bridge: Loss functions measure distance from truth — like variance measures distance from the mean.
Worked example

Five targets, 1, 2, 2, 3 and an outlier at 30. The single constant that minimises squared error is 7.6 — the mean, dragged towards the outlier. The constant that minimises absolute error is 2 — the median, which ignores it. Choosing a loss is choosing which summary of the data the model is aiming at.

Where it misleads
  • MSE is right when errors are roughly Gaussian; with heavy tails it is dominated by a few points. Huber’s loss is the compromise.
  • The loss you train on and the metric you care about are often different; optimising one does not guarantee the other.
  • Cross-entropy on imbalanced classes is dominated by the majority class unless weighted.
In code
y = np.array([1, 2, 2, 3, 30.0])               # one outlier
c = np.linspace(0, 30, 30001)                      # candidate constant predictions
mse = ((y[:, None] - c) ** 2).mean(0)
mae = np.abs(y[:, None] - c).mean(0)
best_mse, best_mae = c[mse.argmin()], c[mae.argmin()]   # the mean and the median
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources
  • P. J. Huber, “Robust Estimation of a Location Parameter”, Annals of Mathematical Statistics 35(1), 1964
  • The Elements of Statistical Learning (2nd ed.), T. Hastie, R. Tibshirani & J. Friedman, Springer, 2009
  • Deep Learning, I. Goodfellow, Y. Bengio & A. Courville, MIT Press, 2016 — chapter 6

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 5
← Previous
Bias-Variance Tradeoff
Open in the full reader, with the topic sidebar