Investigated
What is interpretability research?
Short answer
Image: hai.stanford.edu
Interpretability research studies how to make machine-learning systems understandable to people: which inputs matter, how internal components represent concepts, and why a model produces a prediction. It supports debugging, evaluation, accountability, and safety, but the field remains divided over definitions and whether post-hoc explanations reliably reflect a model’s actual reasoning.
The views on this question
Two lenses on the same evidence. Source weight and the primary source ratio show what each rests on.
Alternative
Source weight 36%Critical researchers argue that interpretability is not one clearly defined property and that calls to open the black box can conceal unresolved questions about what counts as understanding. They distinguish transparency from post-hoc explanation, warn that explanations may be unfaithful or misleading, and question whether simple models are always interpretable or complex models always unintelligible.
- Critics argue that “interpretability” has multiple, sometimes conflicting meanings, making claims about it difficult to assess consistently.
- Post-hoc explanations may not faithfully describe what a black-box model actually computes and can therefore mislead users.
- Some critics challenge the assumption that linear models are automatically interpretable and deep neural networks automatically uninterpretable.
- Understanding individual neural-network components may not compose into understanding the behavior of the whole system.
- A critical view questions whether the black-box problem is always well-defined or whether interpretability is necessarily the right solution to every practical concern.
Follow the white rabbit
Deeper threads worth pulling on next.
- How do feature attributions work?Different angle, same storyExplains methods like saliency maps and SHAP and whether they identify inputs that drive a model's decision.
- What is mechanistic interpretability?A different rabbit holeDives into reverse-engineering model internals into human-understandable circuits and debates over generality.
- Can explanations be audited legally?Different angle, same storyConsiders regulatory and evidentiary standards for algorithmic explanations in consumer protection and liability.