Are current generalization theories sufficient?

Lens adapted to this topic: Arguments that current theories miss the target

The dissenting position argues that current theories often explain narrow statistical behavior while failing to explain the capabilities researchers care about. It stresses that no-free-lunch results make inductive bias unavoidable, that singular-learning accounts may not explain neural-network generalization, and that LLM abilities such as extrapolation and fine-tunability can differ despite similar loss. On this view, a satisfactory theory must address structure, information, and behavior—not merely average test error.

• A critical assessment argues that Singular Learning Theory does not by itself explain generalization in neural networks and may not resolve the problems it targets.
• A universal-induction perspective argues that statistical theory relies on inductive biases it does not justify, while no-free-lunch results limit data-only explanations.
• One analysis claims that LLM capabilities require more than statistical generalization because nearly equivalent test loss can correspond to substantially different behaviors.
• A critique of prevailing deep-learning theory describes memorization of random labels as a challenge to existing explanations and calls for rethinking the framework.
• Computational-complexity arguments suggest that some forms of structural generalization may exceed what pure Transformers can learn, though the cited review notes that the impossibility claim depends on an unproved learnability ceiling.

Featured sourceTowards a Simple Explanation of the Generalization Mystery ...blif.org6 sourcesView sources