03 — Evaluate Your Model

Regression Metrics#

RegressionStatistical 1 min read
◆ The PatternMAE, RMSE, R² — which metric for which problem and what the numbers actually mean

Regression metrics measure how far predictions are from truth. MAE treats every error equally, RMSE penalises big errors more, and R² tells you how much variance your model explains vs a baseline mean prediction.

MAE = (1/n) · Σ |yᵢ − ŷᵢ|
MAE = average absolute error. Robust to outliers, interpretable in target units.
RMSE = √((1/n) · Σ (yᵢ − ŷᵢ)²)
RMSE = penalises large errors disproportionately. Same units as target.
R² = 1 − SSres / SStot
R² = fraction of variance explained. 1 = perfect, 0 = no better than predicting the mean.
// Interactive — drag points, see metrics update
Noise0.30
MAE—
RMSE—
R²—
Adjusted R²: R²adj = 1 − (1−R²)(n−1)/(n−p−1). Penalises adding features. Always use adjusted R² when comparing models with different feature counts.
Pattern bridge: Choosing between MAE and RMSE is choosing how hard to punish the big misses — the same choice a loss function makes during training, and the one you face again when backtesting forecasts. Guide: Which metric should I evaluate my model with?
Worked example

A model that learned the wrong pattern on eight held-out points: MAE 2.12, RMSE 2.14 — and an R² of −2.04. On new data R² can go below zero: the model is worse than predicting the mean every time. R² is a comparison with that baseline, not a number guaranteed to sit between 0 and 1.

Where it misleads
  • R² rises with every added feature on training data; judge it on held-out data, or use adjusted R².
  • RMSE and MAE answer different questions (the mean versus the median of the error); pick the one that matches the cost of mistakes.
  • R² depends on the spread of the target; the same model scores higher on a more varied test set.
In code
y = np.array([10.0, 12, 11, 13, 12, 14, 11, 13])
model = np.array([12.5, 10, 13, 11, 14, 12, 13.5, 11])          # a model that learned the wrong pattern
r2 = 1 - ((y - model) ** 2).sum() / ((y - y.mean()) ** 2).sum()
mae, rmse = np.abs(y - model).mean(), np.sqrt(((y - model) ** 2).mean())
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library
# Python — regression metrics
from sklearn.metrics import (
    mean_absolute_error, mean_squared_error, r2_score
)

mae  = mean_absolute_error(y_true, y_pred)
rmse = mean_squared_error(y_true, y_pred, squared=False)
r2   = r2_score(y_true, y_pred)

print(f"MAE: {mae:.3f}  RMSE: {rmse:.3f}  R²: {r2:.3f}")
Sources
  • T. O. Kvålseth, “Cautionary Note about R²”, The American Statistician 39(4), 1985
  • C. J. Willmott & K. Matsuura, “Advantages of the Mean Absolute Error (MAE) over the Root Mean Square Error (RMSE) in Assessing Average Model Performance”, Climate Research 30, 2005
  • T. Chai & R. R. Draxler, “Root Mean Square Error (RMSE) or Mean Absolute Error (MAE)?”, Geoscientific Model Development 7, 2014

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 4
← Previous
ROC & AUC Curves
Open in the full reader, with the topic sidebar