Deep Learning✓ Mathematical
◆ The PatternGated architectures for long-range memory
LSTM adds a cell state and three gates (forget, input, output) to control information flow, solving the vanishing gradient problem. GRU simplifies this to two gates (reset, update) with comparable performance. Both are the default deep learning baseline for time series.
LSTM: ft = σ(Wf[ht−1,xt]) | it = σ(Wi[ht−1,xt]) | ot = σ(Wo[ht−1,xt])
The forget gate decides what to discard, the input gate what to write, and the output gate what to expose.
// Interactive — LSTM gate activations
# PyTorch — LSTM for multi-step forecasting class LSTMForecaster(nn.Module): def __init__(self, n_features, hidden, n_out): super().__init__() self.lstm = nn.LSTM(n_features, hidden, num_layers=2, batch_first=True, dropout=0.2) self.fc = nn.Linear(hidden, n_out) def forward(self, x): out, _ = self.lstm(x) return self.fc(out[:, -1, :])
Pattern bridge: LSTM gates are an attention mechanism before attention existed — they learn what to remember, connecting to the formal self-attention in transformers. The GRU/LSTM choice mirrors the GRU topic in ML Math.