06 — Foundations

Bias-Variance Tradeoff#

TheoryMathematical 1 min read
◆ The PatternThe fundamental tension between underfitting and overfitting

Every model makes two types of errors: Bias — systematic error from wrong assumptions (underfitting). Variance — sensitivity to noise in training data (overfitting). You can't minimize both simultaneously.

E[(y−ŷ)²] = Bias² + Variance + Irreducible Noise
Total expected error decomposes into these three independent terms
// Model complexity vs. error — the classic U-curve
Model Complexity3.0
Bias²—
Variance—
⬇

High Bias (Underfitting)

Model too simple — misses real patterns. Both train and test error are high.

⬆

High Variance (Overfitting)

Model too complex — memorises noise. Low train error, high test error.

✓

Sweet Spot

Regularization, dropout, cross-validation help find the optimal complexity.

How to diagnose this in practice
  1. Train a simple baseline first and record train/validation metrics.
  2. If both train and validation scores are weak, add features or use a more expressive model.
  3. If train score is strong but validation score is weak, add regularization, simplify the model, or collect more data.
  4. Use learning curves to test whether more data is likely to help before spending time collecting it.
  5. Confirm the diagnosis with cross-validation; a single lucky split can hide high variance.
Common pitfall — tuning to the test set: If you keep checking the final test set while reducing variance, you are training on it indirectly. Keep one final holdout untouched.
Pattern bridge: The U-curve of bias vs. variance is the same tradeoff between confidence interval width and precision in statistics. In markets, overconfidence is low bias, high variance — the model fits noise.
Performance in practice
  • Random Forests = low bias, moderate variance → bagging reduces variance. Rarely overfit severely, which is why they're the go-to baseline
  • Deep neural networks = very low bias, potentially high variance → need dropout, weight decay, early stopping, data augmentation
  • Linear models = high bias, low variance → add polynomial features or switch to a more expressive model if underfitting
  • The "double descent" phenomenon: very large neural nets can go past the interpolation threshold and generalize well again — the classic U-curve doesn't always hold
When to use this
✓ Use when: Diagnosing why your model performs poorly. Deciding between a simpler or more complex model. Choosing regularization strength. Understanding why ensemble methods work.
✗ Skip when: You already know the problem is data quality (garbage in, garbage out). Using pre-trained models where the bias-variance trade-off was already optimized by the pre-training team.
Worked example

Fit polynomials to 200 noisy samples of a sine wave, 30 points each, and compare the fits. A straight line (degree 1) has squared bias 0.155 and variance 0.020: consistently wrong. Degree 3 has 0.003 and 0.014: about right. Degree 9 has almost no bias, 0.007, but variance 0.962: it chases the noise in each sample.

Where it misleads
  • Very large neural networks can generalise better as they grow past the point of fitting the training data (“double descent”, Belkin et al. 2019); the classic U-curve is not the whole story.
  • Bias and variance are properties of a method across datasets; one fitted model has neither on its own.
  • More data reduces variance, not bias.
In code
rng = np.random.default_rng(1)
f = lambda x: np.sin(2 * np.pi * x)
x_test = np.linspace(0.05, 0.95, 50)

def bias_var(degree, sets=200, n=30, noise=0.3):
    preds = []
    for _ in range(sets):                    # many training sets from the same world
        x = rng.random(n); y = f(x) + rng.normal(0, noise, n)
        preds.append(np.polyval(np.polyfit(x, y, degree), x_test))
    preds = np.array(preds)
    return ((preds.mean(0) - f(x_test)) ** 2).mean(), preds.var(0).mean()

result = {d: np.round(bias_var(d), 3).tolist() for d in (1, 3, 9)}   # degree: [bias^2, variance]
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources
  • S. Geman, E. Bienenstock & R. Doursat, “Neural Networks and the Bias/Variance Dilemma”, Neural Computation 4(1), 1992
  • M. Belkin, D. Hsu, S. Ma & S. Mandal, “Reconciling Modern Machine-Learning Practice and the Classical Bias–Variance Trade-off”, PNAS 116(32), 2019
  • The Elements of Statistical Learning (2nd ed.), T. Hastie, R. Tibshirani & J. Friedman, Springer, 2009

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. Training error is 2% and validation error is 25%. What is the main problem?

Show answer

A. A large gap between training and validation error is the signature of variance. Regularisation, more data or a simpler model narrow it.

2. When does adding more training data help most?

Show answer

B. More data constrains a flexible model. A model that is too simple stays too simple however much data it sees.

3. Training error is 20% and validation error 21%, on a task where people manage 5%. What should you try?

Show answer

C. Both errors are high and close together: the model cannot represent the pattern, so more capacity, not more constraint, is needed.

Linked from 7
← Previous
Activation Functions
Open in the full reader, with the topic sidebar