Adaptive attacks

Lens adapted to this topic: What the established evaluation approach shows

The established view treats adaptive attacks as a necessary part of serious robustness evaluation. An evaluator should account for the defense’s mechanisms and threat model, test with appropriately tuned attacks, and avoid inferring robustness from a narrow or fixed attack set. Evidence from adversarial-example research shows that incomplete adaptive evaluations can leave multiple published defenses vulnerable.

• Adaptive attacks have become a standard approach for evaluating defenses against adversarial examples because they test whether a defense remains effective when the attacker accounts for it.
• Evaluations can be adaptive yet still incomplete: a study of 13 defenses found that all could be circumvented using carefully selected, defense-specific strategies.
• Adaptive attacks cannot simply be fully automated in every case; effective testing may require careful tuning appropriate to the particular defense.
• For LLM agents, defenses against indirect prompt injection should also be tested against attackers that adapt to the defense and interaction history.

Lens adapted to this topic: Broader challenges and competing interpretations

A broader view argues that conventional adaptive evaluation may itself be too narrow if it assumes a fixed attacker, a single turn, or complete knowledge of the defense. It emphasizes realistic black-box interaction, multi-turn adaptation, multiple attacker models, and defenses that adapt as well. This perspective treats robustness as a moving target and questions whether one benchmark or ranking can represent security across scenarios.

• Adaptive robustness should include realistic decision-based or black-box settings, where attacks and defenses learn through interaction rather than relying only on complete knowledge of the defense.
• Adaptive defenses may be necessary alongside model hardening because attackers can adjust to how a system responds; even active defenses can then become targets of further adaptation.
• In LLM-agent security, allowing attackers to observe prior responses and adapt across multiple rounds can reveal more successful attacks than evaluating only the first turn.
• Different attackers and scenarios can expose different weaknesses, so aggregate rankings of defenders may disagree across test scenarios.
• The arms-race framing suggests that definitive proofs of operational robustness remain difficult for real-world AI systems, making threat-model and evaluation-design choices central to conclusions.