SequentialMathematical
1 min read
◆ The PatternGated memory cells that solve the vanishing gradient problem
LSTMs add a cell state c — a "memory highway" — alongside the hidden state. Three gates control information flow.
fₜ=σ(Wf·[hₜ₋₁,xₜ]) iₜ=σ(Wi·[hₜ₋₁,xₜ]) oₜ=σ(Wo·[hₜ₋₁,xₜ])
f = forget | i = input | o = output gates
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ tanh(Wc·[hₜ₋₁,xₜ])
Cell state update: forget old + write new candidate
hₜ = oₜ ⊙ tanh(cₜ)
Hidden state = filtered cell state
// LSTM gate diagram
Pattern bridge: The forget gate decides what to keep and what to discard — the same selective memory behind recency bias.
Worked example
The forget gate decides how much of the cell state to keep each step. At 0.9, after 100 steps 0.003% is left; at 0.99, 37%; at 0.999, 90%. The memory length is roughly 1/(1 − f): 10, 100 and 1,000 steps. Because the gate is learned, the network chooses its own memory span.
Where it misleads
- The original LSTM had no forget gate; it was added by Gers et al. (2000), and initialising its bias high helps.
- In large comparisons most LSTM variants perform about the same; the forget gate and output activation matter most (Greff et al. 2017).
- For long sequences transformers usually win, given enough data.
In code
keep = {f: f ** 100 for f in (0.9, 0.99, 0.999)} # share of the cell state left after 100 steps
span = {f: 1 / (1 - f) for f in (0.9, 0.99, 0.999)} # rough memory length in stepsAssumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
self.lstm = nn.LSTM(input_size=10, hidden_size=64, num_layers=2, batch_first=True, dropout=0.2) out, (hn, cn) = self.lstm(x)
Sources
- S. Hochreiter & J. Schmidhuber, “Long Short-Term Memory”, Neural Computation 9(8), 1997
- F. A. Gers, J. Schmidhuber & F. Cummins, “Learning to Forget: Continual Prediction with LSTM”, Neural Computation 12(10), 2000
- K. Greff et al., “LSTM: A Search Space Odyssey”, IEEE Transactions on Neural Networks and Learning Systems 28(10), 2017
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1