Information Theory✓ Mathematical
◆ The PatternBeyond linear correlation — information-theoretic measures that capture any kind of dependency
Mutual information measures how much knowing one variable reduces uncertainty about another — capturing any dependency (linear, nonlinear, categorical). Unlike correlation, it detects complex relationships that Pearson's r misses entirely.
I(X;Y) = Σ p(x,y) · log(p(x,y) / (p(x)·p(y)))
MI = 0 means independent | Higher = more dependency. Always ≥ 0, unbounded above.
IG(Y|X) = H(Y) − H(Y|X)
Information Gain = entropy before − entropy after splitting. Used in decision trees.
// Interactive — see MI vs correlation for different relationships
Relationship
MI—
Correlation—
# Python — mutual information for feature selection from sklearn.feature_selection import mutual_info_classif mi = mutual_info_classif(X, y, random_state=42) mi_series = pd.Series(mi, index=X.columns).sort_values(ascending=False) print(mi_series)
Pattern bridge: Information gain is how decision trees choose splits. Entropy from the ML Math collection is the foundation. In markets, high mutual information between an indicator and future returns would mean that indicator has real predictive value.