Experiment tracking logs hyperparameters, metrics, artifacts, and code versions for every training run. Three months from now, when someone asks "which model was that?" you can answer. MLflow, Weights & Biases, and Neptune are the main tools.
Two configurations, five seeds each. The single best run is config B at 0.853 AUC, so B looks like the winner. Averaged over seeds, A is 0.8446 and B 0.8450, with seed-to-seed spreads of 0.005 to 0.007. The difference is a twentieth of the noise; tracking is what lets you see that.
- Reporting the best seed is selection on noise; report the mean and spread over seeds.
- Tracking every metric invites picking the one that looks good afterwards; decide the main metric first.
- A run log without the data version cannot be reproduced.
runs = pd.DataFrame({'config': ['A'] * 5 + ['B'] * 5,
'seed': list(range(5)) * 2,
'auc': [0.842, 0.851, 0.838, 0.847, 0.845, # five seeds of each config, made up
0.849, 0.836, 0.853, 0.841, 0.846]})
best = runs.loc[runs.auc.idxmax()]
by_config = runs.groupby('config').auc.agg(['mean', 'std'])Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
# Python — MLflow experiment tracking import mlflow mlflow.set_experiment("fraud-detection") with mlflow.start_run(): mlflow.log_param("n_estimators", 200) mlflow.log_param("max_depth", 8) mlflow.log_metric("auc", 0.942) mlflow.log_metric("f1", 0.873) mlflow.sklearn.log_model(model, "model")
- M. Zaharia et al., “Accelerating the Machine Learning Lifecycle with MLflow”, IEEE Data Engineering Bulletin 41(4), 2018
- X. Bouthillier et al., “Accounting for Variance in Machine Learning Benchmarks”, Proceedings of Machine Learning and Systems 3, 2021
- P. Henderson et al., “Deep Reinforcement Learning That Matters”, AAAI, 2018
Last reviewed October 2026 · How this site checks its content · Report a mistake