Non-linearity✓ Mathematical
◆ The PatternWhat makes deep networks more than stacked linear transforms
Without activations, any stack of linear layers collapses to a single linear transform. Activation functions introduce non-linearity, giving networks the power to approximate any function.
// Select function — solid = f(x), dashed = derivative f′(x)
| Function | Formula | Range | Derivative | Used In |
|---|---|---|---|---|
| Sigmoid | 1/(1+e⁻ˣ) | (0,1) | σ(x)(1−σ(x)) | Binary output, old nets |
| ReLU | max(0,x) | [0,∞) | 0 or 1 | ResNets, most CNNs |
| Tanh | (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ) | (−1,1) | 1−tanh²(x) | RNNs, LSTMs |
| GELU | x·Φ(x) | (−∞,∞) | Complex | BERT, GPT |
| SiLU | x·σ(x) | (−∞,∞) | σ(x)(1+x(1−σ(x))) | EfficientNet, LLaMA |
Pattern bridge: ReLU clips everything below zero — a threshold, like support and resistance levels that prices bounce off.