SequentialMathematical
◆ The PatternProcessing sequences by passing hidden state through time
RNNs process sequences step-by-step, maintaining a hidden state h that carries memory of previous inputs.
hₜ = tanh(Wₕ·hₜ₋₁ + Wₓ·xₜ + b)
hₜ = new hidden state | same weights reused at every timestep
// RNN unrolled through time
Seq length4
Click Animate
Vanishing gradient: Gradients shrink exponentially over many timesteps. LSTM and GRU solve this with gated memory.
Pattern bridge: Hidden state flowing through time steps is sequential memory — like exponential moving averages where each value inherits from the past.
Worked example
Backpropagating through 50 time steps multiplies 50 factors. If each shrinks the gradient to 0.9 of its size, 0.005 of it survives; if each grows it to 1.1, it becomes 117 times larger. Plain RNNs live between vanishing and exploding, which is why they forget long-range patterns.
Where it misleads
- Gradient clipping handles explosions but not vanishing; gates (LSTM, GRU) address vanishing.
- Truncated backpropagation through time limits how far back the model can learn, whatever the architecture.
- RNNs process steps one at a time, so they parallelise poorly compared with transformers.
In code
steps = 50 vanish, explode = 0.9 ** steps, 1.1 ** steps # gradient factor through 50 steps, per-step gain 0.9 or 1.1
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
Sources
- J. L. Elman, “Finding Structure in Time”, Cognitive Science 14(2), 1990
- Y. Bengio, P. Simard & P. Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult”, IEEE Transactions on Neural Networks 5(2), 1994
- R. Pascanu, T. Mikolov & Y. Bengio, “On the Difficulty of Training Recurrent Neural Networks”, ICML, 2013
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 2