Investigated
Measuring corrigibility
Short answer
Image: lesswrong.com
Measuring corrigibility is not yet a settled engineering problem. One approach tests observable behavior—shutdown cooperation, non-interference, transparency, and preservation of human control—while another warns that such metrics may reward surface compliance or depend on hidden assumptions about the system’s goals.
The views on this question
Two lenses on the same evidence. Source weight and the primary source ratio show what each rests on.
Alternative
Source weight 50%Lens adapted to this topic: Why corrigibility metrics may mislead
The skeptical view argues that a numerical or behavioral corrigibility score may measure compliance in narrow test environments rather than robust openness to correction. A system can pass tests under assumptions that do not hold for another architecture or objective, and strong corrigibility may require resolving human values first. This view favors searching for a coherent underlying concept and stress-testing metrics against distribution shift, deception, and goal-dependent failure modes.
- Explicit metrics can blind researchers to nearby concepts that better capture what corrigibility is intended to mean, especially when the target is fuzzy or context-dependent.
- A corrigibility method may work for one AI design but fail for another if the designs imply different outcomes or assumptions about the agent’s goals.
- Strong corrigibility may require solving the problem of identifying human values; once values are fixed, corrigibility may add less than expected, while moderate forms may remain useful.
- Lists of desiderata may not reveal a unified concept, so measuring isolated behaviors risks mistaking tool-like surface behavior for robust corrigibility.
- The core challenge includes having the system treat its own design as potentially mistaken and regard external correction as beneficial, rather than merely obeying a test prompt.