Are robustness certificates too conservative?

Lens adapted to this topic: Why certification remains useful despite conservatism

The mainstream case is that certificates answer a distinct question: whether a model is provably stable within a specified perturbation set. This assurance can matter in safety-critical or security-sensitive settings even when certificates are incomplete or conservative. The position does not require claiming that current methods are optimal; it treats tighter relaxations, better architectures, and better threat models as research problems rather than reasons to abandon certification.

• Certification supplies guarantees that empirical attack testing cannot establish for every perturbation in the specified set.
• The practical value of certification depends on whether the chosen norm-bounded threat model corresponds to realistic manipulation risks.
• Certification has progressed from toy settings to large-scale problems such as ImageNet classification, although its practical impact remains under examination.
• Loose or poorly chosen certificates indicate limitations of current methods, not necessarily that formal guarantees are inherently too conservative.

Lens adapted to this topic: Evidence that present certificates reject useful robustness

The dissenting position emphasizes a gap between formal certification and useful model behavior. Convex relaxations can be loose, certified training can reduce both standard and robust accuracy, and runtime systems may reject inputs that are not actually adversarial. On this view, certificates can impose substantial costs while covering only narrow perturbation models, so their guarantees may not justify deployment claims without better calibration and evaluation.

• Relaxation-based defenses can flag inputs whose classifications remain consistent throughout the allowed neighborhood, enabling attacks that degrade utility by inducing rejection.
• Across reported computer-vision tasks, certified training with convex relaxations generally had worse standard and robust error than adversarial training under tested ℓ∞ and ℓ2 threat models.
• The reported generalization gap is associated with relaxed non-convex constraints and alignment between adversarial perturbations and signal directions.
• Certification tightness can depend heavily on network configuration and approximation parameters, making certificate quality method-dependent.
• Piecewise-linear architectures impose limitations on efficient certification tightness and may require increased capacity, with possible robust-generalization difficulties.