Information Theory✓ Mathematical
◆ The PatternMeasuring uncertainty — how much "surprise" is in a distribution
Entropy measures the average amount of information (surprise) in a distribution. A fair coin has maximum entropy (1 bit). A loaded coin has lower entropy. Cross-entropy extends this to compare two distributions.
H(P) = −Σ P(x) · log₂P(x)
Shannon Entropy — measured in bits (log₂) or nats (ln). Maximum when uniform, zero when deterministic.
H(P,Q) = −Σ P(x) · log Q(x)
Cross-Entropy — expected surprise when using Q to encode events from P. Always ≥ H(P).
H(P,Q) = H(P) + KL(P||Q)
Cross-entropy = entropy of P + extra bits from approximation error. Minimising CE ≡ minimising KL.
// Binary entropy — adjust probability of heads
P(heads)0.50
Entropy1.000 bits
Why CE loss works: When labels are one-hot, cross-entropy reduces to −log(ŷ_correct). The model only needs to maximise the probability of the correct class.
Pattern bridge: Shannon entropy measures uncertainty — identical to the spread of a distribution. High entropy in LLM sampling means many plausible next tokens.