Theory✓ Mathematical
1 min read
◆ The PatternThe fundamental tension between underfitting and overfitting
Every model makes two types of errors: Bias — systematic error from wrong assumptions (underfitting). Variance — sensitivity to noise in training data (overfitting). You can't minimize both simultaneously.
E[(y−ŷ)²] = Bias² + Variance + Irreducible Noise
Total expected error decomposes into these three independent terms
// Model complexity vs. error — the classic U-curve
Model Complexity3.0
Bias²—
Variance—
⬇
High Bias (Underfitting)
Model too simple — misses real patterns. Both train and test error are high.
⬆
High Variance (Overfitting)
Model too complex — memorises noise. Low train error, high test error.
✓
Sweet Spot
Regularization, dropout, cross-validation help find the optimal complexity.
How to diagnose this in practice
- Train a simple baseline first and record train/validation metrics.
- If both train and validation scores are weak, add features or use a more expressive model.
- If train score is strong but validation score is weak, add regularization, simplify the model, or collect more data.
- Use learning curves to test whether more data is likely to help before spending time collecting it.
- Confirm the diagnosis with cross-validation; a single lucky split can hide high variance.
Common pitfall — tuning to the test set: If you keep checking the final test set while reducing variance, you are training on it indirectly. Keep one final holdout untouched.
Pattern bridge: The U-curve of bias vs. variance is the same tradeoff between confidence interval width and precision in statistics. In markets, overconfidence is low bias, high variance — the model fits noise.
Performance in practice
- Random Forests = low bias, moderate variance → bagging reduces variance. Rarely overfit severely, which is why they're the go-to baseline
- Deep neural networks = very low bias, potentially high variance → need dropout, weight decay, early stopping, data augmentation
- Linear models = high bias, low variance → add polynomial features or switch to a more expressive model if underfitting
- The "double descent" phenomenon: very large neural nets can go past the interpolation threshold and generalize well again — the classic U-curve doesn't always hold
When to use this
✓ Use when: Diagnosing why your model performs poorly. Deciding between a simpler or more complex model. Choosing regularization strength. Understanding why ensemble methods work.
✗ Skip when: You already know the problem is data quality (garbage in, garbage out). Using pre-trained models where the bias-variance trade-off was already optimized by the pre-training team.