18 — Deep Learning

Temporal CNN (TCN)#

Deep LearningStatistical
◆ The PatternConvolutions that see the past, never the future

Temporal Convolutional Networks apply 1D causal convolutions with increasing dilation rates to capture long-range dependencies. Unlike RNNs, they are fully parallelisable during training. The receptive field grows exponentially with depth, allowing efficient long-context processing.

Receptive field = 1 + 2 × (k−1) × ∑ di
k = kernel size, di = dilation at layer i. Doubling dilation each layer (1,2,4,8…) yields exponential growth.
// Interactive — dilated causal convolutions
Layers4
Kernel Size3
Pattern bridge: Dilated convolutions trade sequential processing for parallelism — the same tradeoff that led from LSTMs to Transformers. The receptive field concept maps to context windows in LLMs.
Worked example

With kernel size 3 and two convolutions per block, dilations 1, 2, 4 and 8 give a receptive field of 61 steps. Doubling up to 128 over eight blocks gives 1,021. The history a TCN can use is fixed by its architecture — for hourly data with a weekly pattern (168 steps), the smaller network cannot even see one week back.

Where it misleads
  • A receptive field longer than the useful history adds parameters and noise.
  • Causal padding must be right; a single non-causal layer leaks the future and makes results look excellent.
  • TCNs and RNNs trade places across benchmarks (Bai, Kolter & Koltun 2018); neither dominates.
In code
def receptive_field(k, dilations, convs_per_block=2):
    return 1 + convs_per_block * (k - 1) * sum(dilations)

small = receptive_field(3, [1, 2, 4, 8])
large = receptive_field(3, [2 ** i for i in range(8)])     # dilations 1..128
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd. Each snippet simulates or makes up its own series, as the comments say.

The same idea with a library
# PyTorch — basic TCN block
class TCNBlock(nn.Module):
    def __init__(self, in_ch, out_ch, k, d):
        super().__init__()
        self.conv = nn.Conv1d(in_ch, out_ch, k, padding=(k-1)*d, dilation=d)
        self.relu = nn.ReLU()
    def forward(self, x):
        out = self.conv(x)[:, :, :x.size(2)]  # causal trim
        return self.relu(out) + x
Sources
  • S. Bai, J. Z. Kolter & V. Koltun, “An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling”, arXiv:1803.01271, 2018
  • A. van den Oord et al., “WaveNet: A Generative Model for Raw Audio”, arXiv:1609.03499, 2016

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 1
← Previous
LSTM & GRU
Open in the full reader, with the topic sidebar