Diagnostics✓ Mathematical
◆ The PatternTraining vs validation curves — reading the gap to diagnose models
A learning curve plots training and validation scores as data increases. The gap between them tells you everything: large gap = overfitting, both low = underfitting, converging = sweet spot. It also tells you if more data would help.
Gap = Scoretrain − Scoreval
Large gap = overfitting (model memorises) | Both low = underfitting (model too simple)
// Interactive — adjust complexity, see the gap
Model complexity50
Diagnosis—
# Python — learning curves from sklearn.model_selection import learning_curve import matplotlib.pyplot as plt sizes, train_scores, val_scores = learning_curve( model, X, y, cv=5, n_jobs=-1, train_sizes=np.linspace(0.1, 1.0, 10), scoring='accuracy' ) plt.plot(sizes, train_scores.mean(axis=1), label='Train') plt.plot(sizes, val_scores.mean(axis=1), label='Val') plt.legend(); plt.show()
Pattern bridge: The learning curve gap is the visual form of the bias-variance tradeoff. The same tension appears in walk-forward backtesting — overfitting to past market conditions.
How to use this in practice
- Plot learning curves early in your project — before spending time on feature engineering
- Large gap (overfit) → try regularization, dropout, less complex model, or more data
- Both scores low (underfit) → try more features, more complex model, or feature engineering
- Curves converging but slowly → more data will help — go collect it
- Curves flat and apart → more data won't help — simplify the model instead
Common pitfall — early stopping too early: If your validation curve is still improving, you're stopping before convergence. If your training curve keeps rising while validation drops, you've gone past the sweet spot. Use a patience parameter (e.g. 10 epochs with no improvement) to find the right moment.
Experiment — diagnose the learning curve
Diagnosis: Adjust the sliders to see how model complexity, data size, and noise affect the learning curve gap.
→ Balanced setup — moderate gap between train and val scores
→ Balanced setup — moderate gap between train and val scores
When to use this
✓ Use when: You need to decide between getting more data vs. improving the model. Diagnosing overfitting vs underfitting. Justifying compute budget — will more training help? Before deploying to production as a sanity check.
✗ Skip when: You're doing a quick prototype where directional results are enough. Using pre-trained models where the learning dynamics are already well-studied. The dataset is fixed and you can't get more data anyway.