Time-series transformers adapt the attention mechanism from NLP for temporal data. Informer uses ProbSparse attention for long sequences. Autoformer integrates decomposition into the architecture. PatchTST treats time windows as patches, achieving state-of-the-art results with channel-independent processing.
| Model | Key Innovation | Best For |
|---|---|---|
| Informer | ProbSparse attention | Long sequence forecasting |
| Autoformer | Built-in decomposition | Seasonal data |
| PatchTST | Channel-independent patches | Multivariate benchmarks |
| TimesFM | Pre-trained foundation model | Zero-shot forecasting |
Hourly data with a daily cycle. A single linear layer mapping the last 48 hours to the next hour has a mean absolute error of 0.82 on held-out data, against 1.08 for the seasonal naive forecast. Zeng et al. (2023) showed that linear models like this match or beat several published transformer forecasters on standard benchmarks.
- Many transformer forecasting results were not compared against strong simple baselines; check that they are.
- Attention over individual time steps is a poor fit for noisy series; patching (PatchTST) helped by attending over segments.
- Pre-trained “foundation” forecasters are promising but vary by domain; evaluate them on your own data.
rng = np.random.default_rng(12) t = np.arange(600) y = 10 * np.sin(2 * np.pi * t / 24) + 0.01 * t + rng.normal(0, 1, 600) # hourly with a daily cycle L = 48 # look back two days X = np.array([y[i:i + L] for i in range(len(y) - L)]); target = y[L:] split = 400 w = np.linalg.lstsq(np.c_[X[:split], np.ones(split)], target[:split], rcond=None)[0] # one linear layer linear = np.abs(np.c_[X[split:], np.ones(len(X) - split)] @ w - target[split:]).mean() seasonal_naive = np.abs(X[split:, -24] - target[split:]).mean()
Assumes import numpy as np and import pandas as pd. Each snippet simulates or makes up its own series, as the comments say.
The same idea with a library
# Python — PatchTST with HuggingFace from transformers import PatchTSTForPrediction model = PatchTSTForPrediction.from_pretrained( "ibm/patchtst-etth1-forecasting" ) # Or with Darts library from darts.models import TFTModel model = TFTModel(input_chunk_length=96, output_chunk_length=24)
- H. Zhou et al., “Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting”, AAAI, 2021
- A. Zeng, M. Chen, L. Zhang & Q. Xu, “Are Transformers Effective for Time Series Forecasting?”, AAAI, 2023
- Y. Nie, N. H. Nguyen, P. Sinthong & J. Kalagnanam, “A Time Series Is Worth 64 Words: Long-term Forecasting with Transformers”, ICLR, 2023
Last reviewed October 2026 · How this site checks its content · Report a mistake