Cross-Validation Done Right

04 — Evaluate Your Model

Cross-Validation Done Right#

ValidationStatistical 2 min read

◆ The PatternSplitting your data honestly — k-fold, stratified, time-series split, and the leakage traps

Cross-validation rotates which data is used for training and testing. K-fold splits data into k parts, trains on k−1, tests on the held-out fold, and repeats. The mean score across folds is a robust estimate of generalisation.

CV Score = (1/k) · Σ Score(foldᵢ)

Average test score across k folds. Standard deviation shows stability.

// Interactive — k folds visualised with train/test splits

K folds5

Mode

Method	Use when	Watch out
K-Fold	General purpose, enough data	Shuffling breaks time order
Stratified K-Fold	Imbalanced classes	Preserves class ratios per fold
Time Series Split	Temporal data (stocks, logs)	Train always before test
Leave-One-Out	Tiny datasets	Expensive, high variance
Nested CV	Hyperparameter tuning + eval	Inner loop tunes, outer evaluates

Data leakage trap: If you scale or impute before splitting, information from the test fold leaks into training. Always put preprocessing inside a Pipeline.

Pattern bridge: Cross-validation in ML and walk-forward validation in backtesting are the same principle — testing on data the model has never seen. The time-series variant here connects directly to market backtesting.

How to use this in practice

Pick your CV strategy: StratifiedKFold for classification, TimeSeriesSplit for temporal data, standard KFold for regression
Wrap all preprocessing in a sklearn.pipeline.Pipeline — scaling, imputation, encoding must happen inside each fold
Use cross_val_score() for a quick check — report mean ± std across folds
For hyperparameter tuning, use nested CV: inner loop tunes (GridSearchCV), outer loop evaluates
If scores vary a lot across folds, your model or data is unstable — investigate data quality or try more folds

Common pitfall — overfitting to CV score: If you run CV many times with different hyperparameters and pick the best, you're overfitting to the CV folds. Use nested CV or hold out a final test set that you touch only once.

Performance in practice

CV fold count impacts both reliability and compute cost:

k=5 is the standard — good bias-variance trade-off, 5x training cost
k=10 reduces variance slightly but doubles compute vs k=5. Rarely worth it for large datasets
LOOCV (k=n) is nearly unbiased but has high variance and n× compute — avoid for n > 10K
On very large datasets even 5-fold CV is expensive. For quick iteration, use a single holdout, then CV for the final report

When to use this

✓ Use when: Small-to-medium datasets where every sample matters. Model comparison and selection. Hyperparameter tuning (inside nested CV). Reporting final model performance for publication.

✗ Skip when: Very large datasets (>500K samples) where a single 80/20 split gives stable estimates. Real-time/streaming data where temporal order matters — use walk-forward instead. Quick prototyping where a holdout split is sufficient.

Worked example

100 examples, 5 of them positive, split into 5 folds of 20 at random. In 96% of random splits at least one fold contains no positives at all, so recall cannot even be computed on it. Stratified folds put exactly one positive in each. With rare classes, how you split matters as much as how many folds you use.

Where it misleads

Stratification fixes the class balance, not group structure; patients, users or time periods still need grouped or time-ordered folds.
Fold scores share most of their training data, so their spread understates the true uncertainty.
Five folds of a tiny dataset give five very noisy estimates; repeated cross-validation helps.

In code

rng = np.random.default_rng(2)
y = np.r_[np.ones(5), np.zeros(95)]                 # 5 positives in 100
empty = 0
for _ in range(10_000):                              # plain 5-fold on a shuffled order
    folds = rng.permutation(y).reshape(5, 20)
    empty += (folds.sum(axis=1) == 0).any()
p_some_fold_without_positives = empty / 10_000

Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library

# Python — cross-validation
from sklearn.model_selection import (
    cross_val_score, StratifiedKFold, TimeSeriesSplit
)

# Standard k-fold
scores = cross_val_score(model, X, y, cv=5, scoring='f1')
print(f"F1: {scores.mean():.3f} ± {scores.std():.3f}")

# Time series — never leak future data
tscv = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(model, X, y, cv=tscv)

Sources

R. Kohavi, “A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection”, IJCAI, 1995
S. Arlot & A. Celisse, “A Survey of Cross-Validation Procedures for Model Selection”, Statistics Surveys 4, 2010
S. Varma & R. Simon, “Bias in Error Estimation When Using Cross-Validation for Model Selection”, BMC Bioinformatics 7, 2006

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. You standardize all features with the mean and standard deviation of the whole dataset, then run 5-fold cross-validation. What is wrong?

Show answer

B. Every preprocessing step that learns from data has to be fitted inside each training fold. In scikit-learn, put the scaler and the model in one Pipeline and cross-validate the pipeline.

2. Why is shuffled k-fold a poor choice for a daily price series?

Show answer

C. Neighbouring days are related, and shuffling puts tomorrow in the training set while testing on today. Use walk-forward validation or TimeSeriesSplit, ideally with a gap between training and test.

3. You try 200 hyperparameter settings and report the best cross-validated score. That score is…

Show answer

A. Selecting the maximum of many noisy scores flatters it. Nested cross-validation, or a test set touched only once at the end, gives an honest estimate.

Linked from 9

← Previous

Regression Metrics

Comparing Model Runs

Open in the full reader, with the topic sidebar