Information TheoryMathematical
1 min read
◆ The PatternMeasuring how different two probability distributions are
KL Divergence measures how much information is lost when using distribution Q to approximate P. It is not symmetric — KL(P||Q) ≠ KL(Q||P).
KL(P||Q) = Σ P(x) · log(P(x)/Q(x))
Always ≥ 0 | = 0 only when P = Q exactly
KL(P||Q) = ∫ p(x) · log(p(x)/q(x)) dx
Continuous case | used in VAE loss, RLHF, variational inference
// Visualise KL divergence between two Gaussians
Q mean offset1.0
KL(P||Q)—
Pattern bridge: KL divergence measures how one distribution diverges from another — the same logic as comparing observed vs. expected in hypothesis testing. In markets, the gap between smart money and dumb money positioning is a kind of divergence signal. The KL term that keeps a VAE’s latent space smooth is this divergence to a standard normal.
Worked example
P is a fair coin and Q a 90/10 coin. KL(P‖Q) is 0.511 nats, KL(Q‖P) is 0.368. The order matters: the first asks how badly Q explains data from P, the second the reverse. When Q gives some outcome almost no probability that P considers likely, KL(P‖Q) blows up.
Where it misleads
- KL is not a distance: it is not symmetric and does not obey the triangle inequality.
- If Q assigns zero probability to anything P can produce, KL(P‖Q) is infinite; smoothing is needed in practice.
- Minimising KL(P‖Q) or KL(Q‖P) gives different fits — one covers all modes, the other locks onto one (a key choice in variational inference).
In code
kl = lambda p, q: np.sum(p * np.log(p / q)) P_, Q_ = np.array([0.5, 0.5]), np.array([0.9, 0.1]) kl_pq, kl_qp = kl(P_, Q_), kl(Q_, P_) # not symmetric
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
Sources
- S. Kullback & R. A. Leibler, “On Information and Sufficiency”, Annals of Mathematical Statistics 22(1), 1951
- Elements of Information Theory (2nd ed.), T. M. Cover & J. A. Thomas, Wiley, 2006
- Pattern Recognition and Machine Learning, C. M. Bishop, Springer, 2006 — chapter 10
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1