Do robustness methods and evaluations for adversarial examples actually generalize beyond the specific threat models, attacks, and data used to design them?

Adversarial training

Adversarial training incorporates deliberately perturbed inputs into model training, typically by solving a min-max problem: generate perturbations that increase loss, then update the model to reduce that loss. The mainstream research account holds that adversarial training can substantially improve robustness to specified attack classes, with PGD-based training among the most effective established defenses. Important qualifications include dependence on attack quality and threat model, possible computational and clean-accuracy costs, and evidence that adversarial training can hurt robust generalization when data are scarce. The central disagreement is not whether adversarial training can help, but how reliably it generalizes beyond the attacks and data conditions used during training, and when its costs outweigh its benefits.

10 sources

PGD attacks

Projected Gradient Descent (PGD) iteratively changes an input in a loss-increasing direction, projecting each step back into a permitted perturbation region such as an ℓp ball and the valid input domain. PGD is widely used to generate adversarial examples and to evaluate or improve neural-network robustness; randomized initialization corresponds to the Madry et al. method in the cited implementation. Its results depend strongly on the loss, norm, projection, initialization, iteration budget, and constraints. Research reports stronger results from alternating objectives, adaptive methods, and more exact constraint handling. The main disagreement is whether standard PGD is a sufficiently strong and representative robustness test: some work treats it as a reliable first-order baseline, while other work finds important cases where it overestimates robustness or can be improved substantially.

9 sources

Certified defenses

Certified defenses aim to replace best-effort robustness with guarantees: for a specified model, input, and perturbation set, a certificate can establish that the prediction cannot change. Exact methods seek definitive answers, while relaxed methods may return “maybe.” Research demonstrates certificates for several threat models, including norm-bounded perturbations, differential-privacy-based robustness, and localized patch attacks, with reported results on datasets ranging from MNIST to ImageNet. The central disagreement is whether these guarantees are sufficiently accurate, broad, scalable, and operationally useful for real deployments: proponents emphasize rigorous, attack-independent guarantees, while critics report generalization costs, loose or incomplete certification, and attacks that exploit system behavior around certification.

10 sources

Adversarial transferability

Adversarial transferability is the ability of an input crafted to fool one model to fool another, potentially unknown model, enabling black-box attacks without access to the target’s parameters. Research links transferability to shared adversarial subspaces, similarities in decision boundaries, non-robust features, and the substitute model’s generalization behavior. Transfer success is not uniform: it depends on the source and target models, attack method, evaluation design, and target error being measured. Benchmarking studies report that some prior comparisons were unfair or incomplete. The main disagreement is whether a broadly unified mechanism explains transferability, or whether observed transfer reflects several interacting factors whose importance varies across settings.

9 sources

Adaptive attacks

Adaptive attacks tailor an attack to the target defense, often using knowledge of its behavior or design. They are widely treated as an important standard for evaluating robustness. Research on adversarial-example defenses found that 13 published defenses could be circumvented because evaluations were incomplete; no single attack strategy worked universally, so careful, defense-specific tuning was needed. More recent work extends the idea to LLM agents and prompt-injection defenses, reporting successful bypasses when attacks adapt to system responses or defense mechanisms. The main disagreement is over how broadly adaptivity should be defined and how robustness should be measured: fixed knowledge-based attacks, realistic black-box interaction, multi-turn attacks, adaptive defenses, or combinations of these.

7 sources

Robustness vs accuracy

Several theoretical and empirical studies find that adversarial training often improves robustness to worst-case perturbations while reducing accuracy on unperturbed inputs, especially with conventional methods. Other research argues that the trade-off is not inherent: suitable data geometry, locally adaptive robustness definitions, local Lipschitz methods, or alternative training procedures may preserve both objectives. The main disagreement is whether observed losses in clean accuracy reflect a fundamental limitation of robustness or limitations in the robustness definition, model class, data assumptions, and training algorithms.

9 sources