Information Theory✓ Mathematical
◆ The PatternMeasuring how different two probability distributions are
KL Divergence measures how much information is lost when using distribution Q to approximate P. It is not symmetric — KL(P||Q) ≠ KL(Q||P).
KL(P||Q) = Σ P(x) · log(P(x)/Q(x))
Always ≥ 0 | = 0 only when P = Q exactly
KL(P||Q) = ∫ p(x) · log(p(x)/q(x)) dx
Continuous case | used in VAE loss, RLHF, variational inference
// Visualise KL divergence between two Gaussians
Q mean offset1.0
KL(P||Q)—
Pattern bridge: KL divergence measures how one distribution diverges from another — the same logic as comparing observed vs. expected in hypothesis testing. In markets, the gap between smart money and dumb money positioning is a kind of divergence signal.