Investigated
Do larger models need less data for robustness?
Short answer
Image: pixabay.com
Larger models can be more robust in some settings, including resistance to deceptive prompts and efficiency during adversarial training, but scale alone is not a dependable substitute for data or defenses. Whether they need less data depends on what “robustness” means: studies find inconsistent out-of-distribution benefits, while data scarcity can substantially reduce large-model performance.
The views on this question
Two lenses on the same evidence. Source weight and the primary source ratio show what each rests on.
Alternative
Source weight 56%Lens adapted to this topic: Why scale may not reduce data needs
A more skeptical reading treats “larger models need less data” as an overgeneralization. Sample efficiency and robustness do not reliably move together across interventions or datasets, and large transformers can be especially data-inefficient when training data is scarce. Improvements may therefore depend more on data quality, task fit, adaptation, or explicit robustness training than on parameter count alone.
- Across three tasks and 14 datasets, higher sample efficiency correlated with better average out-of-distribution robustness only for some interventions and tasks.
- On individual datasets, models with lower sample efficiency could nevertheless be more robust, weakening any universal link between needing less data and robustness.
- Large transformers can suffer significant performance drops when trained with disproportionately limited data; existing scaling laws may underestimate this data inefficiency.
- Data pruning and curation can improve scaling behavior, suggesting that the usefulness and selection of data may matter more than raw data volume alone in some regimes.