Can formal verification scale to AI?

The skeptical view argues that proof automation can scale syntactically without delivering dependable oversight of AI systems. Formal methods verify stated properties under stated models; they do not automatically validate the specification, its assumptions, or the relationship between a formalized proxy and the real-world goal. Adoption costs and organizational incentives may also prevent formal verification from becoming routine.

• Formal verification can certify that an implementation matches a specification while leaving a gap between the specification and what users or safety engineers actually wanted.
• Real-world domains such as physics, chemistry, and biology generally lack complete symbolic rule sets, limiting direct formal verification of open-ended AI behavior.
• Even formally checked code can be sloppy or rely on problematic assumptions, so formality alone should not be treated as a guarantee against flawed reasoning or unsafe systems.
• The cost and slower feedback cycle of formal verification may undermine its adoption even if AI makes proof production faster.
• A scalable-formal-oversight proposal treats formal methods as a promising research direction for constraining capable AI, but presents this as an ongoing program rather than an established solution.

Featured sourcelesswrong.comlesswrong.com6 sourcesView sources