Deep Learning✓ Mathematical
◆ The PatternSequence processing with recurrent memory
Recurrent Neural Networks process sequences step by step, carrying a hidden state that summarises the past. Each step updates the hidden state and optionally produces an output. The RNN learns temporal patterns through backpropagation through time (BPTT), though vanilla RNNs struggle with long-range dependencies.
ht = tanh(Whht−1 + Wxxt + b)
The hidden state ht is a compressed summary of all inputs seen so far. Vanishing gradients make plain RNNs forget early inputs.
// Interactive — RNN unrolled through time
# PyTorch — basic RNN for forecasting import torch.nn as nn class TSRNN(nn.Module): def __init__(self, input_dim, hidden_dim): super().__init__() self.rnn = nn.RNN(input_dim, hidden_dim, batch_first=True) self.fc = nn.Linear(hidden_dim, 1) def forward(self, x): out, _ = self.rnn(x) return self.fc(out[:, -1, :])
Pattern bridge: RNN hidden states are a temporal version of embeddings — compressed representations of input context. The vanishing gradient problem connects to gradient dynamics in deep networks and the residual connections that solve it in transformers.