Rabbit hole · 4 connected questions
How do conventional, standards-driven threat models fail to capture the socio-technical and adversarial dynamics that produce real harms, and what concrete changes to modeling, evaluation, and governance are required to close those blind spots?
How these converge
All four topics point at the same concrete problem: threat modeling practices and standards tend to treat systems as bounded technical artifacts and produce repeatable artifacts, but real harms emerge from interactions among people, institutions, incentives, and deliberate adversaries. That mismatch creates predictable blind spots — missed harms, performative compliance, and underestimated exploitability — which cannot be fixed by more checklist-style standardization alone. The shared implication is that threat modeling must be reworked to integrate socio-technical taxonomies, adversarial threat classes, iterative practices, and governance mechanisms that surface interdependencies, attacker capabilities, and institutional incentives.
Where these converge
Bounded, technical models miss socio-technical harm pathways
Standards and many privacy frameworks focus on technical assets, data flows, and repeatable processes, but socio-technical harms arise from institutional contexts, norms, and power relations (who uses the system, for what, and with what incentives). This creates blind spots where harms such as representational damage, service denial, or cascading institutional effects are overlooked because they lie outside the model’s scope or vocabulary.
Adversarial behaviors expose modeling assumptions and hidden attack surfaces
Adversarial ML research shows concrete mechanisms (poisoning, backdoors, input manipulation, model extraction) that exploit assumptions about data, training, and deployment. Those attack classes systematically exploit gaps in standard threat models that don’t account for malicious actors who can manipulate inputs, supply chains, or models themselves, turning benign-sounding threats into real, scalable harms.
Standards and checklists enable performative compliance unless paired with governance and iterative practices
Across topics there is a recurring warning: repeatable, standardized methods can encourage checkbox behavior and obscure subjective judgments. Closing the gap requires documented, iterative review, context-specific inquiry, adversarial testing, transparency where safe, and governance structures that surface trade-offs and hold actors accountable — not merely a single prescriptive methodology.
The chain
Keep going: open any topic above to find its own related questions.