Investigated
How do feature attributions work?
Short answer
Image: christophm.github.io
Feature attributions assign scores to input features—such as pixels, words, or tabular variables—to estimate how they influenced a particular prediction, often relative to a baseline or through perturbations, gradients, or cooperative-game calculations. They are useful diagnostic summaries, not automatically faithful explanations: different methods can produce ambiguous results, and studies identify important failure cases and evaluation traps.
The views on this question
Two lenses on the same evidence. Source weight and the primary source ratio show what each rests on.
Skeptics
Source weight 42%Lens adapted to this topic: Why attribution may mislead
Methodological critics argue that an attribution map or ranked list can look persuasive without being a reliable account of the model’s reasoning. There is no universally accepted definition of attribution or consistent ground truth, and different techniques may produce divergent explanations. The PNAS work identifies provable limitations for complete and linear methods, while other research reports deficiencies in saliency maps, rationales, attention, and attribution evaluation. On this view, explanations require method-specific validation and should not be treated as causal evidence by default.
- There is no consensus definition of “attribution,” making comparisons and correctness claims difficult.
- Different attribution methods can yield ambiguous or conflicting explanations because they have different conceptual assumptions.
- Researchers report deficiencies in saliency maps, rationales, and attention when evaluated against induced ground-truth feature relevance.
- Impossibility results show that popular complete and linear methods can be less reliable than simpler methods for some basic attribution questions.
- Evaluation procedures themselves can contain logical traps, producing inaccurate or unfair comparisons of attribution methods.
Follow the white rabbit
Deeper threads worth pulling on next.
- What is SHAP and how it works?Different angle, same storySHAP is a cooperative-game-based attribution method with clear assumptions and trade-offs to unpack.
- How do saliency maps fail?Different angle, same storyImage and text saliency maps have concrete failure modes that make them easy to misinterpret.
- Can attributions be gamed?