How are attributions evaluated?

Short answer

Attributions are mainly evaluated by testing faithfulness: whether features given high importance actually influence the model’s prediction when perturbed. Researchers also use constructed ground truth, multiple metrics, robustness checks, and benchmark datasets, but no single test is decisive because each can introduce biases or measure a different notion of explanation.

The deeper question

Are attribution methods just gaming fragile perturbation tests rather than revealing how models really decide?

Sources show researchers mainly judge attributions by faithfulness: perturb or remove high-importance features and see if outputs change, augmented by synthetic ground truth, multiple metrics, and robustness checks. Critics counter that these evaluations can reward procedural artifacts, create unrealistic inputs, conflate prompt versus stored knowledge in language models, and reflect different, incompatible definitions of “explanation.” The evidence supports both: perturbation-based faithfulness tests catch some failure modes but are brittle and can be manipulated, while provenance-aware benchmarks and distinct soundness/completeness checks are suggested but not yet standard.

Tech

The views on this question

Two lenses on the same evidence. Source weight and the primary source ratio show what each rests on.

Critical

Source weight 50%

Lens adapted to this topic: Challenges to standard attribution tests

A critical perspective argues that common evaluation scores may reward procedural artifacts rather than genuine explanations. Attribution is not uniformly defined, perturbation can create unrealistic inputs, and context attribution in language models may confuse information supplied in the prompt with knowledge already encoded in model weights. This view favors explicit sanity checks, provenance-aware benchmarks, and separate tests for soundness and completeness.

There is no consensus on what attribution means, so a method can perform well under one definition while failing another.
Standard ablation-based tests may suffer from missingness bias, costly retraining, or out-of-domain inputs created by feature removal.
Attribution quality should be examined through distinct properties such as soundness—whether selected features are predictive—and completeness.
For language-model context attribution, overlap between prompt information and knowledge stored in model weights can produce unreliable contribution scores.
Sanity checks and provenance-aware ground-truth benchmarks are needed to distinguish faithful explanations from methods exploiting evaluation artifacts.

Short answer

The deeper question

The views on this question

Follow the white rabbit

How are attributions evaluated?

Short answer

The views on this question

Critical

Follow the white rabbit