Validation✓ Mathematical
1 min read
◆ The PatternSplitting your data honestly — k-fold, stratified, time-series split, and the leakage traps
Cross-validation rotates which data is used for training and testing. K-fold splits data into k parts, trains on k−1, tests on the held-out fold, and repeats. The mean score across folds is a robust estimate of generalisation.
CV Score = (1/k) · Σ Score(foldᵢ)
Average test score across k folds. Standard deviation shows stability.
// Interactive — k folds visualised with train/test splits
K folds5
Mode
| Method | Use when | Watch out |
|---|---|---|
| K-Fold | General purpose, enough data | Shuffling breaks time order |
| Stratified K-Fold | Imbalanced classes | Preserves class ratios per fold |
| Time Series Split | Temporal data (stocks, logs) | Train always before test |
| Leave-One-Out | Tiny datasets | Expensive, high variance |
| Nested CV | Hyperparameter tuning + eval | Inner loop tunes, outer evaluates |
# Python — cross-validation from sklearn.model_selection import ( cross_val_score, StratifiedKFold, TimeSeriesSplit ) # Standard k-fold scores = cross_val_score(model, X, y, cv=5, scoring='f1') print(f"F1: {scores.mean():.3f} ± {scores.std():.3f}") # Time series — never leak future data tscv = TimeSeriesSplit(n_splits=5) scores = cross_val_score(model, X, y, cv=tscv)
Data leakage trap: If you scale or impute before splitting, information from the test fold leaks into training. Always put preprocessing inside a
Pipeline.Pattern bridge: Cross-validation in ML and walk-forward validation in backtesting are the same principle — testing on data the model has never seen. The time-series variant here connects directly to market backtesting.
How to use this in practice
- Pick your CV strategy: StratifiedKFold for classification, TimeSeriesSplit for temporal data, standard KFold for regression
- Wrap all preprocessing in a
sklearn.pipeline.Pipeline— scaling, imputation, encoding must happen inside each fold - Use
cross_val_score()for a quick check — report mean ± std across folds - For hyperparameter tuning, use nested CV: inner loop tunes (GridSearchCV), outer loop evaluates
- If std across folds is high (>10% of mean), your model is unstable — investigate data quality or try more folds
Common pitfall — overfitting to CV score: If you run CV many times with different hyperparameters and pick the best, you're overfitting to the CV folds. Use nested CV or hold out a final test set that you touch only once.
Performance in practice
CV fold count impacts both reliability and compute cost:
- k=5 is the standard — good bias-variance trade-off, 5x training cost
- k=10 reduces variance slightly but doubles compute vs k=5. Rarely worth it for large datasets
- LOOCV (k=n) is nearly unbiased but has high variance and n× compute — avoid for n > 10K
- At Google scale, even 5-fold CV on large datasets uses distributed computing. For quick iteration, use a single holdout, then CV for the final report
When to use this
✓ Use when: Small-to-medium datasets where every sample matters. Model comparison and selection. Hyperparameter tuning (inside nested CV). Reporting final model performance for publication.
✗ Skip when: Very large datasets (>500K samples) where a single 80/20 split gives stable estimates. Real-time/streaming data where temporal order matters — use walk-forward instead. Quick prototyping where a holdout split is sufficient.