PythonPractice
1 min read
◆ The PatternThe Swiss army knife — classification_report, cross_val_score, learning_curve, and the metrics module
scikit-learn includes everything covered in this toolkit under one roof. The metrics module has every scorer, model_selection has every CV strategy, and inspection has permutation importance and partial dependence.
// The scikit-learn evaluation workflow
| Module | Key functions | This toolkit topic |
|---|---|---|
| sklearn.metrics | classification_report, roc_auc_score, r2_score | Confusion Matrix, Regression |
| sklearn.model_selection | cross_val_score, TimeSeriesSplit, learning_curve | CV, Learning Curves |
| sklearn.inspection | permutation_importance, PartialDependenceDisplay | Permutation, PDP/ICE |
| sklearn.feature_selection | mutual_info_classif, SelectKBest | Information Gain |
Tip: Always use
Pipeline to chain preprocessing + model. This prevents data leakage in CV and makes deployment trivial.Pattern bridge: The same metrics follow a model into production, where model monitoring tracks them over time. For ordered data, time-series cross-validation replaces the shuffled folds.
Worked example
Three classes with 900, 80 and 20 examples and per-class F1 of 0.95, 0.60 and 0.20. The macro average, which counts each class once, is 0.583; the weighted average, which counts each example once, is 0.907. classification_report prints both. The weighted number mostly describes the big class.
Where it misleads
- Micro, macro and weighted averages answer different questions; report the one that matches what you care about.
- Default scorers in cross_val_score (accuracy, R²) may not be the metric you need; set scoring explicitly.
- Reports on the training data say nothing about generalisation; evaluate on held-out folds.
In code
support = np.array([900, 80, 20]) # three classes, very unequal f1 = np.array([0.95, 0.60, 0.20]) # per-class F1 from a classification report macro = f1.mean() # every class counts the same weighted = (f1 * support).sum() / support.sum() # every example counts the same
Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Complete sklearn evaluation workflow from sklearn.model_selection import ( cross_val_score, learning_curve, GridSearchCV ) from sklearn.metrics import ( classification_report, confusion_matrix, roc_auc_score, mean_squared_error ) from sklearn.inspection import permutation_importance # 1. Cross-validated scores scores = cross_val_score(model, X, y, cv=5, scoring='f1') # 2. Full classification report model.fit(X_train, y_train) print(classification_report(y_test, model.predict(X_test))) # 3. Learning curves sizes, train_s, val_s = learning_curve(model, X, y, cv=5) # 4. Feature importance pi = permutation_importance(model, X_test, y_test, n_repeats=30) # 5. Hyperparameter tuning grid = GridSearchCV(model, param_grid, cv=5, scoring='f1') grid.fit(X_train, y_train)
Sources
- F. Pedregosa et al., “Scikit-learn: Machine Learning in Python”, Journal of Machine Learning Research 12, 2011
- M. Sokolova & G. Lapalme, “A Systematic Analysis of Performance Measures for Classification Tasks”, Information Processing & Management 45(4), 2009
- J. Opitz & S. Burst, “Macro F1 and Macro F1”, arXiv:1911.03347, 2019
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1