SwiGLU · GELU · Expansion✓ Mathematical
◆ The PatternThe per-token MLP after every attention layer
After attention mixes information across tokens, the feed-forward network (FFN) processes each token independently. It expands to a higher dimension, applies a nonlinearity, and projects back down. This is where most of the model's "knowledge" is stored.
FFN(x) = W₂ · σ(W₁x + b₁) + b₂ where W₁ ∈ ℝ^(d × 4d)
Classic design: expand 4×, activate, contract. About 2/3 of transformer parameters live here.
SwiGLU(x) = (W₁x ⊙ Swish(W_gate·x)) · W₂ where W₁,W_gate ∈ ℝ^(d × ⅔·4d)
Gated variant used in LLaMA, Mistral, Gemma. ⅔ factor keeps parameter count equal to standard 4d expansion.
SwiGLU consistently outperforms ReLU and GELU. The gating mechanism lets the network learn to suppress/amplify features multiplicatively — more expressive than additive bias alone.
In a 70B-parameter model, the FFN layers contain roughly 47B parameters. They act as massive key-value memories: keys are W₁ rows, values are W₂ columns.
Interactive — activation functions comparison
Python — SwiGLU FFN#
import torch, torch.nn as nn, torch.nn.functional as F
class SwiGLU_FFN(nn.Module):
def __init__(self, d_model, expansion=8/3):
super().__init__()
hidden = int(d_model * expansion)
self.w1 = nn.Linear(d_model, hidden, bias=False)
self.w_gate = nn.Linear(d_model, hidden, bias=False)
self.w2 = nn.Linear(hidden, d_model, bias=False)
def forward(self, x):
return self.w2(self.w1(x) * F.silu(self.w_gate(x)))Pattern bridge: The MLP after attention stores factual knowledge — the model’s memory bank. Like activation functions that add non-linearity after linear attention.