A dangerous capability evaluation measures whether an AI model has abilities that could enable severe harm if misused or if the model acted against its developers' intent: for example, finding and exploiting software vulnerabilities, giving meaningful uplift toward biological or chemical weapons, persuading or deceiving people, or copying and sustaining itself outside its environment. It asks what a model can do, not what it will do when asked politely.
Capability evaluations versus alignment evaluations
A capability evaluation estimates the worst a model could do with determined elicitation. An alignment or propensity evaluation asks whether the model tends to do harmful things by default. Both are needed: a model with no dangerous capability poses little risk even if poorly aligned, while a highly capable model needs strong evidence about propensity and safeguards.
How they are designed
- Task suites with graded difficulty, such as capture-the-flag challenges for cyber offence, or multi-step agentic tasks in a sandbox.
- Expert uplift studies comparing what people can achieve with and without model assistance.
- Thresholds set in advance, tied to required safeguards if crossed.
- Strong elicitation: scaffolding, tools, many attempts and fine-tuning, so that a low score reflects inability rather than a weak prompt or sandbagging.
Worked example
Illustrative numbers: a model solves 6 of 40 cyber tasks on a single attempt, a 15% success rate with a 95% Wilson interval of roughly 7% to 29%. With 10 independent attempts per task and a perfect check of success, a task with a 15% per-attempt rate would be solved with probability . A threshold defined on single attempts can therefore badly understate what an attacker who retries can get, which is why elicitation budget must be stated alongside the result. See pass@k.
Limits
A negative result is only as good as the elicitation behind it, and small task suites give wide intervals. Results go stale as scaffolds and tools improve, so evaluations need rerunning on deployed systems, not just at training time.
Why it matters
Dangerous capability evaluations are the evidence base for decisions about whether to train further, deploy, open-weight release, or add safeguards. They are a central input to AI safety frameworks and to regulation that asks providers to assess systemic risks, and they complement red teaming, which probes whether safeguards hold.