Optuna is a hyperparameter optimisation framework that uses TPE (Tree-structured Parzen Estimator) to intelligently search the parameter space. It supports pruning (killing bad trials early) and integrates with scikit-learn, XGBoost, PyTorch, and more.
| Feature | What it does |
|---|---|
| TPE sampler | Bayesian sampling — learns from previous trials |
| Pruning | Early stopping for bad trials (MedianPruner) |
| Study dashboard | optuna-dashboard for real-time monitoring |
| Multi-objective | Optimise accuracy AND speed simultaneously |
Pure random search lands at least one trial in the best 5% of a search space with probability 64% after 20 trials, 95% after 60 and near certainty after 200 — whatever the number of dimensions, if only a few really matter (Bergstra & Bengio 2012). Smarter samplers such as TPE beat this baseline; it is the bar they have to clear.
- The best of many trials is optimistically biased; evaluate it on data the search never saw.
- Pruning stops slow starters that might finish best; it is a heuristic, not a guarantee.
- A wide, poorly scaled space (learning rate on a linear rather than log scale) wastes most trials.
p_top5 = {n: round(1 - 0.95 ** n, 3) for n in (20, 60, 200)} # random search: at least one trial in the top 5%Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Optuna — hyperparameter optimisation import optuna from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score def objective(trial): params = { 'n_estimators': trial.suggest_int('n_estimators', 50, 500), 'max_depth': trial.suggest_int('max_depth', 3, 20), 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20), 'max_features': trial.suggest_categorical( 'max_features', ['sqrt', 'log2', None] ), } model = RandomForestClassifier(**params) return cross_val_score(model, X, y, cv=5, scoring='f1').mean() study = optuna.create_study(direction='maximize') study.optimize(objective, n_trials=100) print(study.best_params) optuna.visualization.plot_optimization_history(study)
- T. Akiba, S. Sano, T. Yanase, T. Ohta & M. Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework”, KDD, 2019
- J. Bergstra & Y. Bengio, “Random Search for Hyper-Parameter Optimization”, Journal of Machine Learning Research 13, 2012
- J. Bergstra, R. Bardenet, Y. Bengio & B. Kégl, “Algorithms for Hyper-Parameter Optimization”, NeurIPS, 2011
Last reviewed October 2026 · How this site checks its content · Report a mistake