Generalization✓ Mathematical
1 min read
◆ The PatternPreventing overfitting by constraining model complexity
Regularization adds a penalty for complexity to the loss, discouraging the model from memorising noise and improving generalisation to unseen data.
L_total = L_data + λ·||w||²₂ (L2 / Ridge)
λ = regularization strength | drives weights toward zero but not exactly zero
L_total = L_data + λ·||w||₁ (L1 / Lasso)
L1 produces sparse weights — many exactly zero (implicit feature selection)
Dropout: h̃ = h ⊙ mask/p mask ~ Bernoulli(p)
Each neuron zeroed with prob (1−p) during training; scaled by 1/p to preserve expected value
// L1 vs L2 penalty contours — see how they constrain weights differently
λ strength1.0
# L2 via weight_decay in optimizer optim = torch.optim.AdamW(model.parameters(), weight_decay=1e-4) # Dropout layer self.dropout = nn.Dropout(p=0.3) # drop 30% of neurons model.eval() # disables dropout at test time model.train() # re-enables dropout
Pattern bridge: L1/L2 penalties constrain complexity. In markets, loss aversion acts as a natural regularizer, penalizing risky bets.
How to apply regularization safely
- Start with an unregularized baseline and save train/validation metrics.
- Add L2/weight decay first; tune it on validation data, not the test set.
- Use L1 only when sparsity or feature selection matters.
- Add dropout for neural nets only when the validation gap is real.
- Re-check calibration and feature importance after regularization; it can change model behavior.
Common pitfall — regularizing underfit models: If both train and validation loss are high, regularization will usually make the model worse. Increase capacity or improve features first.
Performance in practice
- Dropout 0.1-0.3 is standard for transformers. BERT uses 0.1; higher values hurt for large pre-trained models that already have strong representations
- Weight decay 1e-2 (AdamW) is the default for most LLM fine-tuning. Higher values (0.1) for small datasets, lower (1e-4) for large
- L1 creates sparse models that are 2-10x faster at inference — great for mobile/edge deployment
- Data augmentation is the most powerful regularizer for vision (flips, crops, color jitter add 2-5% accuracy). Mixup and CutMix push it further
When to use this
✓ Use when: Train loss is much lower than val loss (classic overfitting). Small dataset relative to model capacity. Fine-tuning a large model on a small domain dataset.
✗ Skip when: Model is underfitting (both train and val loss high) — you need more capacity, not less. Very large datasets where overfitting is unlikely (e.g. training CLIP on 400M image-text pairs).
Experiment — regularization strength
Adjust sliders to see how regularization affects your model's behavior.