Why is my model not generalising?
A diagnosis in a few questions — from the gap between training and validation error to leakage and drift.
Both are high
The model cannot express the pattern. Try a more flexible model, better features or weaker regularisation — and check whether the labels are too noisy to learn from.
Training is low, validation much higher
Yes, the gap is still closing
Collect more data or augment it; until then, regularise.
No, it has flattened
Regularise, simplify, stop earlier, or prune features — more data alone will not close the gap.
Validation looks good, but production is worse
Preprocessing was fitted on all the data, or time-ordered data were split at random
The validation score saw information the live model never gets. Fit every step inside each fold and split time series by time.
The best of many settings was picked on the validation set
The winning score was partly luck. Use nested cross-validation or a test set touched once.
The live data look different from the training data
Inputs, base rates or the relationship itself have moved. Monitor them and retrain on recent data.
The validation score jumps between runs
A small validation set or random seeds move the score more than the change you are testing. Use repeated cross-validation and several seeds, and compare runs with a paired test.
Last reviewed: 2 October 2026 · How this site checks its content · Report a mistake