10 — Training

Regularization#

GeneralizationMathematical 1 min read
◆ The PatternPreventing overfitting by constraining model complexity

Regularization adds a penalty for complexity to the loss, discouraging the model from memorising noise and improving generalisation to unseen data.

L_total = L_data + λ·||w||²₂    (L2 / Ridge)
λ = regularization strength  |  drives weights toward zero but not exactly zero
L_total = L_data + λ·||w||₁    (L1 / Lasso)
L1 produces sparse weights — many exactly zero (implicit feature selection)
Dropout: h̃ = h ⊙ mask/p    mask ~ Bernoulli(p)
Each neuron zeroed with prob (1−p) during training; scaled by 1/p to preserve expected value
// L1 vs L2 penalty contours — see how they constrain weights differently
λ strength1.0
Pattern bridge: L1/L2 penalties constrain complexity. In markets, loss aversion acts as a natural regularizer, penalizing risky bets.
How to apply regularization safely
  1. Start with an unregularized baseline and save train/validation metrics.
  2. Add L2/weight decay first; tune it on validation data, not the test set.
  3. Use L1 only when sparsity or feature selection matters.
  4. Add dropout for neural nets only when the validation gap is real.
  5. Re-check calibration and feature importance after regularization; it can change model behavior.
Common pitfall — regularizing underfit models: If both train and validation loss are high, regularization will usually make the model worse. Increase capacity or improve features first.
Performance in practice
  • Dropout of 0.1 is common in transformers (BERT uses 0.1); many recent large language models use little or none
  • PyTorch’s AdamW defaults to a weight decay of 0.01; large pre-training runs often use 0.1 (GPT-3, LLaMA)
  • L1 gives sparse linear models, which helps feature selection and interpretation; in neural networks, unstructured sparsity rarely speeds up inference on standard hardware
  • Data augmentation is among the most effective regularisers for vision (flips, crops, colour jitter); Mixup and CutMix extend it (Zhang et al. 2018; Yun et al. 2019)
When to use this
✓ Use when: Train loss is much lower than val loss (classic overfitting). Small dataset relative to model capacity. Fine-tuning a large model on a small domain dataset.
✗ Skip when: Model is underfitting (both train and val loss high) — you need more capacity, not less. Very large datasets where overfitting is unlikely (e.g. training CLIP on 400M image-text pairs).
Experiment — regularization strength
Adjust sliders to see how regularization affects your model's behavior.
Worked example

Two inputs that are almost copies of each other, and a target built from 1 × each. Ordinary least squares returns weights of 0.33 and 1.78 — an arbitrary split that would change with the next sample. Ridge regression with λ = 1 returns 1.05 and 1.05. The penalty picks the stable answer among many that fit equally well.

Where it misleads
  • Regularization strength is a hyperparameter; too much and the model underfits. Choose it by cross-validation.
  • L2 and L1 penalties depend on the scale of the inputs; standardize first.
  • Shrunken coefficients are biased on purpose, so read them as predictions, not as causal effects.
In code
rng = np.random.default_rng(2)
x1 = rng.normal(size=50)
x2 = x1 + rng.normal(0, 0.01, 50)                  # almost a copy of x1
X = np.column_stack([x1, x2])
y = x1 + x2 + rng.normal(0, 0.5, 50)               # the truth: weights 1 and 1
ols = np.linalg.solve(X.T @ X, X.T @ y)
ridge = np.linalg.solve(X.T @ X + 1.0 * np.eye(2), X.T @ y)   # L2 penalty, lambda = 1
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

The same idea with a library
# L2 via weight_decay in optimizer
optim = torch.optim.AdamW(model.parameters(), weight_decay=1e-4)

# Dropout layer
self.dropout = nn.Dropout(p=0.3)   # drop 30% of neurons

model.eval()   # disables dropout at test time
model.train()  # re-enables dropout
Sources
  • A. E. Hoerl & R. W. Kennard, “Ridge Regression: Biased Estimation for Nonorthogonal Problems”, Technometrics 12(1), 1970
  • R. Tibshirani, “Regression Shrinkage and Selection via the Lasso”, Journal of the Royal Statistical Society B 58(1), 1996
  • N. Srivastava et al., “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, Journal of Machine Learning Research 15, 2014

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. What is the practical difference between L1 and L2 regularisation?

Show answer

C. The L1 penalty has a corner at zero, so the optimum often sits exactly there; the L2 penalty is smooth.

2. What happens as you increase the regularisation strength λ?

Show answer

C. A stronger penalty restricts the model: less sensitive to the training sample, but less able to fit the true pattern.

3. You fit ridge regression without standardising the features. What goes wrong?

Show answer

A. A feature in millimetres needs a larger coefficient than the same feature in metres, so the penalty treats them differently. Standardise first.

Linked from 4
← Previous
Optimizers
Open in the full reader, with the topic sidebar