Deep Learning✓ Mathematical
◆ The PatternConvolutions that see the past, never the future
Temporal Convolutional Networks apply 1D causal convolutions with increasing dilation rates to capture long-range dependencies. Unlike RNNs, they are fully parallelisable during training. The receptive field grows exponentially with depth, allowing efficient long-context processing.
Receptive field = 1 + 2 × (k−1) × ∑ di
k = kernel size, di = dilation at layer i. Doubling dilation each layer (1,2,4,8…) yields exponential growth.
// Interactive — dilated causal convolutions
Layers4
Kernel Size3
# PyTorch — basic TCN block class TCNBlock(nn.Module): def __init__(self, in_ch, out_ch, k, d): super().__init__() self.conv = nn.Conv1d(in_ch, out_ch, k, padding=(k-1)*d, dilation=d) self.relu = nn.ReLU() def forward(self, x): out = self.conv(x)[:, :, :x.size(2)] # causal trim return self.relu(out) + x
Pattern bridge: Dilated convolutions trade sequential processing for parallelism — the same tradeoff that led from LSTMs to Transformers. The receptive field concept maps to context windows in LLMs.