Investigated
What is SHAP and how it works?
Short answer
Image: christophm.github.io
SHAP, or SHapley Additive exPlanations, explains an individual machine-learning prediction by assigning each feature a contribution relative to a baseline prediction. It adapts Shapley values from cooperative game theory, averaging a feature’s marginal contribution across feature coalitions. In practice, SHAP uses model-specific or model-agnostic algorithms and masking schemes to estimate these contributions. The resulting values can clarify a model’s behavior, but they are not automatically causal explanations and can mislead about relative importance under some conditions.
The views on this question
Two lenses on the same evidence. Source weight and the primary source ratio show what each rests on.
Alternative
Source weight 55%Critical research argues that SHAP’s mathematical allocation rules do not always track the features that are genuinely relevant to a prediction. In particular, constructed classifiers can receive misleading relative rankings, and correlated features or unrealistic masked inputs can distort results. These critiques do not reject SHAP as a useful diagnostic, but challenge treating its values as definitive feature importance or human-centered explanations.
- Some classifiers admit cases where SHAP ranks an irrelevant feature above a feature that is provably relevant to the prediction.
- Critics argue that resolving problems in Shapley-based feature importance can require additional assumptions, including causal reasoning.
- When predictors are correlated, marginalizing or masking features may create unrealistic combinations of feature values and produce misleading explanations.
- The computational difficulty of obtaining rigorous attribution values can make exact, reliable explanations impractical for broad classes of models.
- These critiques question whether Shapley values naturally satisfy human goals of explainability, even when their mathematical properties are attractive.
Follow the white rabbit
Deeper threads worth pulling on next.
- What are Shapley values exactly?Different angle, same storyUnderstanding the game-theory math behind SHAP shows why contributions are averaged across coalitions.
- How do SHAP and LIME differ?Different angle, same storyComparing SHAP to LIME highlights differences in consistency, locality, and approximation.
- Can SHAP imply causation?