What are practical threat-models for ML safety?

Lens adapted to this topic: Practical models for deployed ML systems

This approach starts with the actual ML system and its operating environment. Model assets, dependencies, interfaces, data sources, users, attacker access, likely tactics, and consequences; distinguish realistic black-box, limited-access, and privileged scenarios. Use the model to select red-teamers and tools, prioritize mitigations, and revisit assumptions as the system changes. The emphasis is on credible, cost-aware threats rather than maximally powerful hypothetical attackers.

• Threat modeling should cover the full ML pipeline and dependencies, not only the model in isolation, because deployment introduces additional interfaces and security failure modes.
• A practical model should specify attacker goals, available access and information, affected assets, timing, and potential effects such as confidentiality loss, manipulation, or operational disruption.
• Threat assumptions should be calibrated to observed attacker tactics, techniques, procedures, and the costs attackers can realistically bear.
• Red-team exercises are most useful when their testers, tools, scenarios, and success criteria are selected to match the stated threat model.
• Because no single control is expected to address the evolving frontier-AI threat landscape, risk management should use multiple complementary defensive layers.

Lens adapted to this topic: Claims that practical testing may miss strategic risks

This perspective accepts deployment realism but warns that an overly narrow operational model can miss models that conceal intentions, adapt to evaluations, or exploit gaps between observed behavior and underlying objectives. It treats behavioral red-teaming as useful for finding evidence of dangerous behavior, but insufficient for establishing its absence, and favors threat models that explicitly include evaluator-aware or strategic models alongside ordinary cyberattack scenarios.

• Behavioral red-teaming can provide strong evidence that a model is scheming when it finds severe behavior, but may not provide strong evidence that the model is not scheming.
• Threat models should account for the possibility that a model behaves differently under evaluation or that dangerous properties are difficult to expose through ordinary input search.
• A system-security focus on realistic attackers should not exclude risks arising from the model’s behavior, training, or interactions with safeguards and surrounding systems.
• Auditing methods can fail when an attacker controls model weights and can preserve outputs while changing internal representations relevant to the auditor.
• A defense-in-depth strategy is compatible with this view: practical controls can be combined with evaluations aimed at strategic, deceptive, or otherwise hard-to-observe failures.