Human evaluation limits

Lens adapted to this topic: How better study design can address known limits

The mainstream methodological view treats human evaluation as necessary but fallible measurement. Its evidence points to weaknesses in sampling, evaluator selection, instructions, metrics, reporting, reproducibility, and statistical analysis. It therefore favors explicit protocols, calibration, adjudication, uncertainty estimates, and carefully designed combinations of human and automated judgments rather than abandoning human evaluation.

• A review of 142 healthcare-LLM studies found gaps in reliability, generalizability, and applicability, and proposed the QUEST framework for planning, implementation, adjudication, scoring, and review.
• A review of NLP human evaluations estimated that only about 5% were repeatable from publicly available information, rising to about 20% when author help was sought.
• Human evaluators may fail to distinguish fluent machine-generated text from human text: one study found non-experts identified GPT-3 text at chance level without training.
• The methodological response is to improve task specification, evaluator recruitment and training, reporting, statistical analysis, and the use of complementary automated or LLM-based judges.

Lens adapted to this topic: What methodological fixes may leave unresolved

The critical or outsider view argues that human-evaluation problems are not only technical defects. Evaluation choices can encode assumptions about which outcomes matter, whether context can be ignored, whether impacts can be quantified, and whether failures are comparable. On this account, better protocols are useful but cannot by themselves settle value conflicts, distributional effects, or broader governance questions.

• Evaluation practices can embed normative commitments, including consequentialism, acontextual assessment, limited attention to inputs, quantification of impacts, and commensurability of failure modes.
• Some problems presented as evaluation challenges may be better understood as governance issues concerning norms, power, and societal impacts.
• Large-scale and survey research highlights that human evaluation can lack a dependable gold standard and can be undermined by evaluator fatigue, shared biases, and omitted uncertainty estimates.
• Replacing human judges with LLM judges raises validity and reproducibility questions, so automation may reproduce or compound weaknesses rather than remove them.
• Poor documentation and experimental flaws can make previous human evaluations impossible to assess for reproducibility, limiting confidence in published conclusions.