Red-teaming limits

Lens adapted to this topic: What red-teaming can establish—and where it stops

The mainstream view treats red-teaming as a valuable, practical component of AI safety and accountability rather than a complete safety guarantee. Its value comes from finding previously unknown failures, stress-testing mitigations, and enriching evaluations. The method’s results depend on scope, team composition, access, threat models, and follow-through. Because coverage is incomplete and harms can be difficult to measure, red-teaming should be combined with automated evaluations, monitoring, audits, and other forms of system assessment.

• External red-teaming can reveal novel risks, expose gaps in mitigations, strengthen quantitative safety metrics, and contribute to risk assessments and automated evaluations.
• Experience across many products suggests that understanding the system’s capabilities and deployment context, combining automation with human expertise, and distinguishing red-teaming from safety benchmarking are important design principles.
• No finite red-team exercise can identify every failure mode or certify complete safety; new prompts, contexts, uses, and post-deployment changes can create further vulnerabilities.
• Red-teaming is most defensible as part of a broader evaluation and accountability process, not as a standalone solution to technical or sociotechnical harms.

Lens adapted to this topic: Dissenting critiques of red-teaming’s assurance claims

Critical and outsider perspectives argue that red-teaming is often asked to prove more than its methods can support. Passive tests may have too little statistical power for rare catastrophic failures, while behavioral searches may find evidence of bad behavior more readily than evidence of its absence. Critics also emphasize ambiguous definitions, prompt-spraying with uncertain practical value, weak transfer from demonstrations to deployment risk, and the influence of institutional incentives and tester selection. On this view, red-teaming remains potentially useful, but claims of safety or high return on investment require unusually careful qualification.

• A formal evidential-ceiling analysis argues that feasible benchmark sizes can support claims about sufficiently frequent harms but are several orders of magnitude too small for rare catastrophic categories.
• Behavioral red-teaming may produce strong evidence that a model is scheming when it finds concerning behavior, but generally cannot provide equally strong evidence that scheming is absent because that conclusion depends on uncertain assumptions about model behavior.
• A sociotechnical critique argues that red-teaming is shaped by who performs the labor, which values define harm, and how organizations present the practice to regulators and the public.
• Some critics contend that prompt-spraying has uncertain return on investment, does not scale straightforwardly to agentic systems, and may leave clients with findings that are difficult to convert into effective fixes.
• Methodological critiques of agent red-teaming question whether results from systems lacking production safeguards can support broad conclusions about realistic deployment environments.