05 — Foundations

Activation Functions#

Non-linearityMathematical 1 min read
◆ The PatternWhat makes deep networks more than stacked linear transforms

Without activations, any stack of linear layers collapses to a single linear transform. Activation functions introduce non-linearity, giving networks the power to approximate any function.

// Select function — solid = f(x), dashed = derivative f′(x)
FunctionFormulaRangeDerivativeUsed In
Sigmoid1/(1+e⁻ˣ)(0,1)σ(x)(1−σ(x))Binary output, old nets
ReLUmax(0,x)[0,∞)0 or 1ResNets, most CNNs
Tanh(eˣ−e⁻ˣ)/(eˣ+e⁻ˣ)(−1,1)1−tanh²(x)RNNs, LSTMs
GELUx·Φ(x)(−∞,∞)ComplexBERT, GPT
SiLUx·σ(x)(−∞,∞)σ(x)(1+x(1−σ(x)))EfficientNet, LLaMA
Pattern bridge: ReLU clips everything below zero — a threshold, like support and resistance levels that prices bounce off.
Worked example

Two linear layers, 4 → 8 → 3, applied to 100 inputs give the same output as the single matrix W2·W1 — the largest difference is 4×10⁻¹⁵, rounding error. Put a ReLU between them and outputs differ by up to 15.7. Without a non-linearity, depth adds nothing.

Where it misleads
  • ReLU units can “die”: once their input is always negative they output zero and stop learning.
  • Sigmoid and tanh saturate, giving tiny gradients at large inputs; they were a main cause of vanishing gradients.
  • The universal approximation theorem says a wide enough network can represent a function, not that training will find it.
In code
rng = np.random.default_rng(0)
W1, W2 = rng.normal(size=(8, 4)), rng.normal(size=(3, 8))
x = rng.normal(size=(4, 100))
two_linear = W2 @ (W1 @ x)
one_linear = (W2 @ W1) @ x                  # the same map, one matrix
collapse_gap = np.abs(two_linear - one_linear).max()
with_relu = W2 @ np.maximum(W1 @ x, 0)
relu_gap = np.abs(with_relu - one_linear).max()
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources
  • G. Cybenko, “Approximation by Superpositions of a Sigmoidal Function”, Mathematics of Control, Signals and Systems 2(4), 1989
  • X. Glorot, A. Bordes & Y. Bengio, “Deep Sparse Rectifier Neural Networks”, AISTATS, 2011
  • D. Hendrycks & K. Gimpel, “Gaussian Error Linear Units (GELUs)”, arXiv:1606.08415, 2016

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 7
← Previous
Gradient Descent
Open in the full reader, with the topic sidebar