ExplainabilityStatistical
1 min read
◆ The PatternGame-theoretic feature attribution — understand exactly why your model made each prediction
SHAP (SHapley Additive exPlanations) assigns each feature a contribution to each prediction. Based on Shapley values from cooperative game theory, SHAP is the only method with mathematical guarantees: consistency, local accuracy, and missingness.
φᵢ = ΣS |S|!(M−|S|−1)!/M! · [f(S∪{i}) − f(S)]
φᵢ = SHAP value for feature i | averaged marginal contribution across all coalitions
f(x) = E[f(X)] + Σ φᵢ
Prediction = base value + sum of all SHAP values. Additive: contributions always sum to prediction.
// Interactive — feature contributions to a prediction
Feature count6
Explainer choice: Use
TreeExplainer for tree models (fast, exact). KernelExplainer for any model (slow, approximate). DeepExplainer for deep learning.Pattern bridge: SHAP reveals which features drive a prediction. In markets, volume analysis asks the same question — which factors are driving price? Decomposing into contributions is a universal pattern. The tooling — explainers and plots — is in the SHAP library topic.
How to use this in practice
- Train your model first — SHAP explains a trained model, it doesn't improve it
- Use
shap.plots.waterfall()to explain individual predictions (e.g. "why was this loan denied?") - Use
shap.plots.beeswarm()for global feature importance — which features matter most overall - Check for feature interactions with
shap.plots.scatter()— colored by another feature - Compare SHAP with permutation importance — if they disagree, you likely have correlated features
When SHAP misleads: SHAP assumes feature independence when computing marginal contributions. With highly correlated features (e.g. height and weight), SHAP may distribute credit unevenly. Always check the correlation matrix first. For critical decisions, combine SHAP with domain knowledge.
Performance in practice
SHAP compute cost varies dramatically by explainer type:
- TreeExplainer: O(TLD²) per prediction — fast, handles 100K samples in seconds for XGBoost/LightGBM
- KernelExplainer: O(2^M) where M=features — exponential. With 50 features, use background subsampling (100-200 samples) or wait hours
- Production tip: Pre-compute SHAP for common feature ranges and cache them. Real-time SHAP on every API call is expensive — batch process nightly
- EU AI Act and US lending regulations increasingly require explainability — SHAP is widely used for this
When to use this
✓ Use when: Regulatory compliance requires explanations (finance, healthcare). Debugging model behavior on specific predictions. Stakeholder communication about model decisions. Feature selection guided by contribution analysis.
✗ Skip when: Prototyping where speed matters more than explanation. Linear models where coefficients already tell the story. Very high-dimensional data (1000+ features) — use permutation importance first to narrow down, then SHAP on the top features.
Try it on real data
Kaggle: Credit Card Default (30K clients, explain why predictions differ)
Kaggle: Home Credit Default Risk (300K loans, real feature interactions)
Train any tree model, then run SHAP — you'll immediately see which features the model relies on most. Compare waterfall plots for approved vs denied loans.
Quick start — copy to notebook
pip install shap xgboost pandas matplotlib
# ────────────────────────────────────────
import shap, xgboost, pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv('your_data.csv')
X_train, X_test, y_train, y_test = train_test_split(df.drop('target',1), df['target'])
model = xgboost.XGBClassifier().fit(X_train, y_train)
explainer = shap.TreeExplainer(model)
sv = explainer(X_test)
shap.plots.beeswarm(sv) # global importance
shap.plots.waterfall(sv[0]) # explain one predictionWorked example
For a linear model the SHAP value of each feature is exactly its weight times the distance from the average input. Here the contributions are 2, 0 and −1, the base value is 5, and base plus contributions gives 6 — the prediction, every time. Duplicate the first feature and the model can split its weight; SHAP then gives each copy 1, half the credit.
Where it misleads
- Credit is shared among correlated features in ways that depend on the model, not on the world.
- SHAP explains the model, not the outcome; a feature with large SHAP values is not shown to cause anything.
- Different background datasets give different SHAP values for the same prediction.
In code
w, b = np.array([2.0, -1.0, 0.5]), 1.0 # a linear model X = np.array([[1.0, 2.0, 3.0], [3.0, 0.0, 1.0], [2.0, 1.0, 2.0]]) # background data x = np.array([3.0, 1.0, 0.0]) phi = w * (x - X.mean(axis=0)) # exact SHAP values for a linear model base = w @ X.mean(axis=0) + b check = (base + phi.sum(), w @ x + b) # additivity dup = np.array([1.0, 1.0]) * w[0] / 2 * (x[0] - X[:, 0].mean()) # feature 1 copied twice, weight split
Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Python — SHAP waterfall for a single prediction import shap explainer = shap.TreeExplainer(model) shap_values = explainer(X_test) # Waterfall for one prediction shap.plots.waterfall(shap_values[0]) # Global summary shap.plots.beeswarm(shap_values)
Sources
- S. M. Lundberg & S.-I. Lee, “A Unified Approach to Interpreting Model Predictions”, NeurIPS, 2017
- E. Štrumbelj & I. Kononenko, “Explaining Prediction Models and Individual Predictions with Feature Contributions”, Knowledge and Information Systems 41(3), 2014
- K. Aas, M. Jullum & A. Løland, “Explaining Individual Predictions When Features Are Dependent”, Artificial Intelligence 298, 2021
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 2