ProbabilityMathematical
1 min read
◆ The PatternMaximum Likelihood Estimation — why MSE and Cross-Entropy exist
MLE asks: what parameters make the observed data most probable? Most ML training objectives are secretly MLE under a particular assumed distribution.
θ* = argmax_θ Σ log p(xᵢ | θ)
Maximise log-likelihood (summing logs avoids numerical underflow with tiny probabilities)
Gaussian: p(x|μ,σ) = (1/√2πσ²) · exp(−(x−μ)²/2σ²)
MLE on Gaussian noise assumption → MSE loss. MLE on Bernoulli → Cross-Entropy loss.
// Gaussian distribution — adjust mean and variance
Mean μ0.0
Std σ1.0
Key insight: Training with MSE loss = assuming your errors are Gaussian distributed. Training with Cross-Entropy = assuming Bernoulli/Categorical outputs. The loss function encodes your distributional assumption.
Pattern bridge: Maximum likelihood estimation is the mathematical foundation of fitting a normal distribution to data. The same principle drives volatility estimation in markets — finding the parameters that best explain observed returns.
Worked example
Seven heads in ten tosses: the probability that makes this most likely is 0.7. For the values 4, 6, 5, 9 and 6, the maximum-likelihood mean is 6.0 and the maximum-likelihood variance 2.8 — it divides by n. The usual unbiased estimate, dividing by n − 1, is 3.5. MLE is a principle for estimates, not a guarantee they are unbiased.
Where it misleads
- With little data MLE overfits: after three heads in three tosses it says the coin never lands tails. Priors (MAP) or smoothing fix this.
- The likelihood assumes a model; if the model is wrong, the best fit to it can still be a poor description.
- Minimising MSE is MLE under Gaussian noise, and cross-entropy is MLE for categorical outcomes — useful to know which assumption a loss carries.
In code
heads, n = 7, 10 p = np.linspace(0.01, 0.99, 99) loglik = heads * np.log(p) + (n - heads) * np.log(1 - p) p_hat = p[loglik.argmax()] # the MLE: 7/10 x = np.array([4.0, 6.0, 5.0, 9.0, 6.0]) mu_hat = x.mean() var_mle = ((x - mu_hat) ** 2).mean() # MLE divides by n, not n - 1 var_unbiased = x.var(ddof=1)
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
Sources
- R. A. Fisher, “On the Mathematical Foundations of Theoretical Statistics”, Philosophical Transactions of the Royal Society A 222, 1922
- Pattern Recognition and Machine Learning, C. M. Bishop, Springer, 2006 — chapters 1–2
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1