The Toolkit

TreeExplainer on a classifier reports in log-odds. With a 20% base rate the base value is −1.386; two features adding 1.2 and 0.9 give a predicted probability of 0.671. Read the same contributions as probability points and you would get 2.3 — an impossible “230%”. Check the output space before reading a waterfall plot.

• TreeExplainer’s default interventional or path-dependent settings give different values for correlated features.
• Global importance from mean |SHAP| is not the same as the importance of a feature for the outcome (Kumar et al. 2020).
• Explanations of a model trained on biased data explain the bias faithfully.

sigmoid = lambda z: 1 / (1 + np.exp(-z))
base_logodds = np.log(0.2 / 0.8)                      # TreeExplainer's base value for a 20% base rate
contrib = np.array([1.2, 0.9])                        # two features' SHAP values, in log-odds
p = sigmoid(base_logodds + contrib.sum())
naive = 0.2 + contrib.sum()                           # treating them as probability points

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

# SHAP — complete reference
import shap

# 1. Choose the right explainer
explainer = shap.TreeExplainer(model)     # XGBoost, LightGBM, RF
# explainer = shap.KernelExplainer(model.predict, X_bg)
# explainer = shap.DeepExplainer(model, X_bg)

# 2. Compute SHAP values
shap_values = explainer(X_test)

# 3. Visualisations
shap.plots.waterfall(shap_values[0])       # single prediction
shap.plots.beeswarm(shap_values)            # global summary
shap.plots.bar(shap_values)                 # mean |SHAP|
shap.plots.scatter(shap_values[:,"age"])   # dependence

# 4. Interaction values (slow but powerful)
inter = explainer.shap_interaction_values(X_test)