Explainability✓ Mathematical
1 min read
◆ The PatternGame-theoretic feature attribution — understand exactly why your model made each prediction
SHAP (SHapley Additive exPlanations) assigns each feature a contribution to each prediction. Based on Shapley values from cooperative game theory, SHAP is the only method with mathematical guarantees: consistency, local accuracy, and missingness.
φᵢ = ΣS |S|!(M−|S|−1)!/M! · [f(S∪{i}) − f(S)]
φᵢ = SHAP value for feature i | averaged marginal contribution across all coalitions
f(x) = E[f(X)] + Σ φᵢ
Prediction = base value + sum of all SHAP values. Additive: contributions always sum to prediction.
// Interactive — feature contributions to a prediction
Feature count6
# Python — SHAP waterfall for a single prediction import shap explainer = shap.TreeExplainer(model) shap_values = explainer(X_test) # Waterfall for one prediction shap.plots.waterfall(shap_values[0]) # Global summary shap.plots.beeswarm(shap_values)
Explainer choice: Use
TreeExplainer for tree models (fast, exact). KernelExplainer for any model (slow, approximate). DeepExplainer for deep learning.Pattern bridge: SHAP reveals which features drive a prediction. In markets, volume analysis asks the same question — which factors are driving price? Decomposing into contributions is a universal pattern.
How to use this in practice
- Train your model first — SHAP explains a trained model, it doesn't improve it
- Use
shap.plots.waterfall()to explain individual predictions (e.g. "why was this loan denied?") - Use
shap.plots.beeswarm()for global feature importance — which features matter most overall - Check for feature interactions with
shap.plots.scatter()— colored by another feature - Compare SHAP with permutation importance — if they disagree, you likely have correlated features
When SHAP misleads: SHAP assumes feature independence when computing marginal contributions. With highly correlated features (e.g. height and weight), SHAP may distribute credit unevenly. Always check the correlation matrix first. For critical decisions, combine SHAP with domain knowledge.
Performance in practice
SHAP compute cost varies dramatically by explainer type:
- TreeExplainer: O(TLD²) per prediction — fast, handles 100K samples in seconds for XGBoost/LightGBM
- KernelExplainer: O(2^M) where M=features — exponential. With 50 features, use background subsampling (100-200 samples) or wait hours
- Production tip: Pre-compute SHAP for common feature ranges and cache them. Real-time SHAP on every API call is expensive — batch process nightly
- EU AI Act and US lending regulations increasingly require explainability — SHAP is the de facto standard for regulatory compliance
When to use this
✓ Use when: Regulatory compliance requires explanations (finance, healthcare). Debugging model behavior on specific predictions. Stakeholder communication about model decisions. Feature selection guided by contribution analysis.
✗ Skip when: Prototyping where speed matters more than explanation. Linear models where coefficients already tell the story. Very high-dimensional data (1000+ features) — use permutation importance first to narrow down, then SHAP on the top features.
Try it on real data
Kaggle: Credit Card Default (30K clients, explain why predictions differ)
Kaggle: Home Credit Default Risk (300K loans, real feature interactions)
Train any tree model, then run SHAP — you'll immediately see which features the model relies on most. Compare waterfall plots for approved vs denied loans.
Quick start — copy to notebook
pip install shap xgboost pandas matplotlib
# ────────────────────────────────────────
import shap, xgboost, pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv('your_data.csv')
X_train, X_test, y_train, y_test = train_test_split(df.drop('target',1), df['target'])
model = xgboost.XGBClassifier().fit(X_train, y_train)
explainer = shap.TreeExplainer(model)
sv = explainer(X_test)
shap.plots.beeswarm(sv) # global importance
shap.plots.waterfall(sv[0]) # explain one prediction