23 — Practice & Tooling

Backtesting Forecasts#

PracticeStatistical 1 min read
◆ The PatternHow good are your predictions, really?

Rolling-origin backtesting simulates how your model would have performed if deployed at each past point. Evaluate with MAE (robust to outliers), RMSE (penalises large errors), MAPE (scale-free), and SMAPE (symmetric). Always evaluate across multiple horizons.

MAE = (1/n)∑|yt − 𝑓̂t|   |   MAPE = (100/n)∑|yt − 𝑓̂t| / |yt|
Choose your metric based on your decision context. MAE in original units, MAPE for stakeholder communication, RMSE when large errors are costly.
// Interactive — forecast accuracy metrics
Pattern bridge: Forecast backtesting is the time-series version of loss metrics from statistics. In markets, this directly maps to strategy backtesting — both test historical performance without look-ahead bias. Guide: Which forecasting model should I start with?
Worked example

Five forecasts. MAE is 8.6 and RMSE 8.82, both in the units of the data. MAPE is 86.6%, because one actual value is 2 and an error of 8 there counts as 400%. Leave that point out and MAPE is 8.2%. Percentage errors explode near zero; scaled errors such as MASE do not.

Where it misleads
  • MAPE punishes over-forecasts and under-forecasts differently and is undefined at zero.
  • One backtest origin is one draw; roll the origin and report the spread.
  • Choosing the metric after seeing the results is another forking path; decide it first.
In code
actual = np.array([120.0, 80.0, 2.0, 150.0, 95.0])
forecast = np.array([110.0, 90.0, 10.0, 140.0, 100.0])
err = actual - forecast
mae = np.abs(err).mean()
rmse = np.sqrt((err ** 2).mean())
mape = np.abs(err / actual).mean() * 100
mape_without_small = np.abs(err / actual)[actual > 10].mean() * 100
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd. Each snippet simulates or makes up its own series, as the comments say.

The same idea with a library
# Python — backtesting with Darts
from darts.metrics import mae, rmse, mape

backtest = model.historical_forecasts(
    series, start=0.7, forecast_horizon=12,
    stride=1, retrain=False
)
print(f"MAE: {mae(series, backtest):.3f}")
print(f"RMSE: {rmse(series, backtest):.3f}")
Sources
  • R. J. Hyndman & A. B. Koehler, “Another Look at Measures of Forecast Accuracy”, International Journal of Forecasting 22(4), 2006
  • L. J. Tashman, “Out-of-Sample Tests of Forecasting Accuracy: An Analysis and Review”, International Journal of Forecasting 16(4), 2000
  • Forecasting: Principles and Practice (3rd ed.), R. J. Hyndman & G. Athanasopoulos, OTexts, 2021

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. Model A beats a seasonal-naive forecast by 3% MAE on one test window. Is that enough?

Show answer

A. Rolling-origin evaluation gives many test windows, so you can see whether the gain is consistent or a fluke of one period.

2. Why compare a forecasting model with a naive forecast at all?

Show answer

C. Scaled errors such as MASE build this in: a value below 1 means better than the naive benchmark.

3. You will forecast 7 days ahead but evaluate only 1-step-ahead errors. What is the problem?

Show answer

A. Evaluate at the horizon you will use, with the information available at the forecast origin.

Linked from 2
← Previous
Cross-Validation for TS
Open in the full reader, with the topic sidebar