Information TheoryStatistical
1 min read
◆ The PatternBeyond linear correlation — information-theoretic measures that capture any kind of dependency
Mutual information measures how much knowing one variable reduces uncertainty about another — capturing any dependency (linear, nonlinear, categorical). Unlike correlation, it detects complex relationships that Pearson's r misses entirely.
I(X;Y) = Σ p(x,y) · log(p(x,y) / (p(x)·p(y)))
MI = 0 means independent | Higher = more dependency. Always ≥ 0, unbounded above.
IG(Y|X) = H(Y) − H(Y|X)
Information Gain = entropy before − entropy after splitting. Used in decision trees.
// Interactive — see MI vs correlation for different relationships
Relationship
MI—
Correlation—
Pattern bridge: Information gain is how decision trees choose splits. Entropy from the ML Math collection is the foundation. In markets, high mutual information between an indicator and future returns would mean that indicator has real predictive value.
Worked example
y = x² plus a little noise: y is almost entirely determined by x. The Pearson correlation is 0.008. Mutual information, estimated on a 10×10 grid, is 1.56 bits. Correlation measures straight-line dependence; mutual information measures any dependence.
Where it misleads
- Histogram estimates of mutual information depend on the bin count and are biased with little data; nearest-neighbour estimators help (Kraskov et al. 2004).
- Mutual information has no sign and no units you can act on directly; it says that there is a relationship, not which.
- Ranking many features by information on the same data overfits; validate the selection.
In code
rng = np.random.default_rng(7) x = rng.uniform(-1, 1, 20_000) y = x ** 2 + rng.normal(0, 0.05, 20_000) # fully dependent, not linear r = np.corrcoef(x, y)[0, 1] joint, _, _ = np.histogram2d(x, y, bins=10) pxy = joint / joint.sum(); px = pxy.sum(1, keepdims=True); py = pxy.sum(0, keepdims=True) nz = pxy > 0 mi_bits = (pxy[nz] * np.log2(pxy[nz] / (px @ py)[nz])).sum()
Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Python — mutual information for feature selection from sklearn.feature_selection import mutual_info_classif mi = mutual_info_classif(X, y, random_state=42) mi_series = pd.Series(mi, index=X.columns).sort_values(ascending=False) print(mi_series)
Sources
- Elements of Information Theory (2nd ed.), T. M. Cover & J. A. Thomas, Wiley, 2006
- A. Kraskov, H. Stögbauer & P. Grassberger, “Estimating Mutual Information”, Physical Review E 69, 2004
- D. N. Reshef et al., “Detecting Novel Associations in Large Data Sets”, Science 334(6062), 2011
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 2