In 1885 Hermann Ebbinghaus learned lists of nonsense syllables and tested himself at intervals afterwards. What he retained fell quickly at first and then more slowly: a forgetting curve. Some sixty-five years later the hydrologist Harold Edwin Hurst, planning dams on the Nile, found the opposite surprise. Wet years clustered with wet years and dry with dry over spans far longer than anyone expected; the river remembered.
Between those two lies a question every model of a sequence has to answer: how far back does the past still matter, and how quickly should it fade? The answer is rarely argued for. It hides in a window length, a decay rate, a lag order or a context size, and it shapes everything the model can see.
Averages — memory as a window
A moving average is a memory with a fixed shape. The simple average remembers N days equally and then forgets at once; the weighted moving average lets the weights fall in a straight line; the exponential average lets them fade without ever quite reaching zero; the double exponential average leans even harder on the present to cut the lag. Choosing between them is choosing how fast to forget. People make the same choice without noticing — recency bias is a memory that is too short, and waiting for mean reversion assumes one long enough to know where the mean is.
Statistics — memory as a correlation
Autocorrelation measures how much a series remembers itself at each lag. The classical models are shapes of that memory: moving-average models remember each shock for exactly q steps and then drop it; autoregressive models remember forever but ever more faintly; SARIMA adds a second memory one season back; VAR models remember the pasts of other series too. Feature engineering for forecasting is largely the craft of choosing which lags to hand a model.
Networks — memory as a state, or as a search
A recurrent network carries a summary of the past forward step by step, and in its plain form the summary fades fast — gradients over long spans vanish. The LSTM added gates that learn what to keep and what to erase. A temporal CNN sees a fixed horizon set by how its dilations stack; N-BEATS simply takes a fixed window of history. Transformers changed the question: with attention, every position in the context can look at every other, so memory becomes a search rather than a fading trace. The KV-cache keeps that past so it need not be recomputed, KV-cache optimisation decides what to drop when it fills, and retrieval moves memory outside the model altogether.
People and products — memory as retention
A retention curve is Ebbinghaus’s curve for a product: how many of the people who joined in a given month are still around one, three, twelve months later. It drops steeply and then flattens, and where it flattens is how long the product is remembered. The Kalman filter takes yet another route, keeping no history at all — only a running estimate and its uncertainty, which together carry everything the past has to say.
Last reviewed October 2026 · How this site checks its content · Report a mistake