Investigated
How do replication crisis audits work?
Short answer
Image: frontiersin.org
Audits can inspect research practices or evaluate a claim’s evidential strength. One proposal randomly samples papers or labs for errors and false positives; another specifies a claim, sets priors, updates belief with the evidence, tests sensitivity, and reports a calibrated conclusion. Audits are safeguards, not automatic verdicts: replication, reproducibility, robustness, and contextual validity measure different things, and an audit can itself be limited by its assumptions or by examining too little evidence.
The views on this question
Two lenses on the same evidence. Source weight and the primary source ratio show what each rests on.
Alternative
Source weight 50%Lens adapted to this topic: Critiques of audit design and interpretation
Dissenting methodological critiques argue that audits can appear rigorous while relying on selective evidence, questionable operationalizations, or unequal standards for original and replication studies. On this view, the audit itself must be stress-tested: its data coverage, methods, assumptions, contextual differences, and treatment of competing explanations all matter.
- A critique of one Bayesian audit argues that it misrepresented aspects of the original study, considered only part of the available evidence, and reduced a broader robustness question to a narrower comparison.
- Critics argue that replication failures may be attributed to contextual or procedural differences, while comparable scrutiny is not always applied to original studies.
- A claim audit should not be treated as a debunking checklist; its purpose is to calibrate confidence and specify what evidence could change the conclusion.
- The existence of replication failures does not by itself settle how widespread the underlying problem is or what any particular failed replication means.
Follow the white rabbit
Deeper threads worth pulling on next.
- How do statistical power audits work?Different angle, same storyAudits that assess power show how small samples inflate false positives and quantify that risk.
- What is p-hacking and how is it detected?A different rabbit holeIdentifying selective reporting and data-peeking reveals specific behaviors audits target.
- How do registered reports prevent bias?Different angle, same storyPre-registration separates hypothesis generation from testing, a procedural fix audits favor.