Guides

Home
Guides · Decision guide · 3 min

Which metric should I evaluate my model with?

From what your model predicts and what its mistakes cost to the metric that measures it honestly.

Pick the metric before you run experiments, compare every model against a simple baseline, and measure on held-out data with a validation scheme that matches how the model will be used.
Background Cross-Validation Done Right · scikit-learn Evaluation Suite · The Garden of Forking Paths
What does the model predict?
A class (yes/no, or one of several)
How common is the class you care about?
Reasonably common (20–80%)
Will you use the predicted probabilities, or only the decisions?
The probabilities (risk scores, pricing, ranking with thresholds later)
Use
Log loss or Brier score, plus a calibration plot

Rewards probabilities that are both sharp and honest; accuracy ignores how confident the model was.

Read Entropy · Confusion Matrix & Classification Metrics
Only the decisions
Use
Accuracy with the confusion matrix (F1 if errors cost about the same)

With balanced classes accuracy is meaningful — but always read the matrix behind it.

Read Confusion Matrix & Classification Metrics · Eval Metrics
Rare (below 10%) — fraud, failures, disease
Which mistake costs more?
Missing a real case
Use
Recall at a fixed precision, and the area under the precision–recall curve

Accuracy is meaningless here (predicting “no” for everything scores 99%). Set the threshold from the cost of each error.

Read Sampling & Class Imbalance · Confusion Matrix & Classification Metrics · ROC & AUC Curves
A false alarm
Use
Precision at a fixed recall

Holds the alert volume to what people can handle, then asks how many alerts are real.

Read Confusion Matrix & Classification Metrics · Sampling & Class Imbalance
I need one number for comparing models
Use
PR-AUC (precision–recall area)

With rare positives ROC-AUC can look good while most alarms are false; PR-AUC focuses on the rare class.

Read ROC & AUC Curves · Sampling & Class Imbalance
A score used to rank or sort
Does the whole ranking matter, or only the top of the list?
The whole ranking
Use
ROC-AUC

The probability that a random positive ranks above a random negative — threshold-free.

Read ROC & AUC Curves
Only the top results
Use
Precision@k (or recall@k)

Measures what users actually see. Common for retrieval and recommendation.

Read ROC & AUC Curves · RAG
A number (price, demand, duration)
What do large errors cost?
In proportion to their size
Use
MAE (mean absolute error)

In the units of the target and robust to outliers; it rewards predicting the median.

Read Regression Metrics · Loss Functions
Big errors are much worse than small ones
Use
RMSE

Squares errors before averaging, so big misses dominate; it rewards predicting the mean.

Read Regression Metrics · Loss Functions
I compare across series of very different scale
Use
MASE (not MAPE)

Scales error by a naive forecast’s error; MAPE explodes near zero and treats over- and under-forecasts differently.

Read Backtesting Forecasts · N-BEATS & N-HiTS
I need to say how much variation is explained
Use
R² on held-out data

A comparison with predicting the mean; on new data it can be negative.

Read Regression Metrics
Future values of a time series
Use
MASE against seasonal naive, on a rolling-origin backtest — plus interval coverage

Forecasts must beat the obvious baseline across many forecast origins, and their intervals should contain the outcome as often as promised.

Read Backtesting Forecasts · Cross-Validation for TS · SARIMA

Last reviewed: 2 October 2026 · How this site checks its content · Report a mistake