17 — Core Math

Entropy#

Information TheoryMathematical
◆ The PatternMeasuring uncertainty — how much "surprise" is in a distribution

Entropy measures the average amount of information (surprise) in a distribution. A fair coin has maximum entropy (1 bit). A loaded coin has lower entropy. Cross-entropy extends this to compare two distributions.

H(P) = −Σ P(x) · log₂P(x)
Shannon Entropy — measured in bits (log₂) or nats (ln). Maximum when uniform, zero when deterministic.
H(P,Q) = −Σ P(x) · log Q(x)
Cross-Entropy — expected surprise when using Q to encode events from P. Always ≥ H(P).
H(P,Q) = H(P) + KL(P||Q)
Cross-entropy = entropy of P + extra bits from approximation error. Minimising CE ≡ minimising KL.
// Binary entropy — adjust probability of heads
P(heads)0.50
Entropy1.000 bits
Why CE loss works: When labels are one-hot, cross-entropy reduces to −log(ŷ_correct). The model only needs to maximise the probability of the correct class.
Pattern bridge: Shannon entropy measures uncertainty — identical to the spread of a distribution. High entropy in LLM sampling means many plausible next tokens.
Worked example

A fair coin carries 1 bit of surprise per toss; a 90/10 coin only 0.47 bits; a fair eight-sided die 3 bits. Encoding the 90/10 coin with a code built for a fair coin costs 1 bit per toss — the cross-entropy — instead of 0.47. The difference is the waste from using the wrong distribution, which is the KL divergence.

Where it misleads
  • Entropy measures unpredictability, not meaning; a random string has maximum entropy and no information you care about.
  • Estimating entropy from samples is biased low when the data are few relative to the number of outcomes.
  • Bits (log base 2) and nats (natural log) differ by a factor of ln 2; check which a library uses.
In code
H = lambda p: -np.sum(np.asarray(p) * np.log2(p))
fair, loaded, eight = H([0.5, 0.5]), H([0.9, 0.1]), H(np.full(8, 1 / 8))
p, q = np.array([0.9, 0.1]), np.array([0.5, 0.5])
cross = -np.sum(p * np.log2(q))                      # coding p's outcomes with q's code
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources
  • C. E. Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal 27(3), 1948
  • Elements of Information Theory (2nd ed.), T. M. Cover & J. A. Thomas, Wiley, 2006

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. What is the entropy of a fair coin flip, in bits?

Show answer

A. H = −2 × ½ log2½ = 1. A biased coin has less: its outcome is easier to guess.

2. Cross-entropy H(p, q) is never smaller than the entropy H(p). When are they equal?

Show answer

A. The gap H(p, q) − H(p) is the KL divergence, which is zero only when q = p. Minimising cross-entropy pushes q towards p.

3. Which distribution over four outcomes has the highest entropy?

Show answer

B. Entropy is highest when you know least: every outcome equally likely.

Linked from 1
← Previous
MLE & Gaussian
Open in the full reader, with the topic sidebar