Dangerous capability evaluations: what they measure, elicitation, thresholds and uncertainty

Dangerous capability evaluations are tests that estimate whether an AI model could meaningfully help cause severe harm, for example by giving uplift towards weapons, carrying out cyberattacks, or acting autonomously in ways that would be hard to stop. They measure what a model can do when pushed, not what it usually does, and their results are compared against capability thresholds that trigger additional safeguards or restrict deployment.

This article covers what these evaluations measure, at the level of categories rather than operational content; why elicitation (getting the model to show its real ability) is the hardest part; how thresholds are set; and how to reason about the uncertainty in a threshold decision. A small statistical model, which you can adjust below, shows why "the model scored below the threshold" is weaker evidence than it sounds, and why more test items do not fix the most important source of error.

What do dangerous capability evaluations measure?#

Shevlane and colleagues set out the logic in 2023: to assess extreme risks, evaluate a model for dangerous capabilities (could it cause harm if used or directed to?) and for alignment (would it try to?), and tie both to decisions about training, deployment and security. Capability evaluations are the more developed of the two because capability is easier to measure than intent. The categories that recur in published evaluation programmes and developers' safety policies are:

Two design principles cut across categories. First, the evaluation must measure uplift rather than raw knowledge. Much hazardous information is already in textbooks and on the internet; the question is whether the model adds something, such as troubleshooting, integration of scattered steps or execution, that a person could not easily get otherwise. Knowledge benchmarks such as WMDP are useful proxies precisely because they are public and cheap, and their authors filtered them to remove sensitive content; the expensive gold standard is a controlled study in which people with and without model access attempt a task. Second, the evaluation must be agentic where the risk is agentic. A model that can answer questions about network exploitation is different from one that can plan and carry out an intrusion with tools, and Phuong and colleagues' evaluation programme, among others, tests the latter directly in sandboxed environments.

Results are reported in deliberately coarse terms. Phuong and colleagues, piloting evaluations across persuasion, cyber-security, self-proliferation and self-reasoning, reported no evidence of strong dangerous capabilities in the models they tested but flagged early warning signs. That phrasing, "no strong evidence, with warning signs", is typical, and the rest of this article is about what it can and cannot mean.

Elicitation: measuring what a model can do, not what it did#

A capability evaluation is an attempt to find a lower bound on the model's best performance. A positive result is strong evidence: if the model did it, it can do it. A negative result is weak evidence: the model may have failed because of a bad prompt, a missing tool, a refusal or an unhelpful scaffold. The gap between what an evaluation observes and what a determined user could get from the model is the elicitation gap, and closing it is most of the work.

Elicitation techniques, roughly in increasing strength:

• Prompting: task-specific instructions, examples and chain-of-thought.
• Scaffolding and tools: giving the model a shell, a browser, code execution and the ability to plan and retry over many steps. For agentic tasks this often matters more than the model itself.
• Sampling budget: many attempts with selection, which is what an attacker would do (the pass@k vs pass^k note covers how to report it).
• Removing safeguards: testing a variant trained without refusal behaviour, because refusals can be bypassed (see jailbreaks) and because weights may leak or be released.
• Fine-tuning: training on a small number of task demonstrations. This is the strongest general method and the one that best simulates a motivated actor with weight access.

How much each technique matters is itself an empirical question. Greenblatt and colleagues studied it with "password-locked" models, deliberately trained to hide a capability unless a password appears, as stand-ins for hidden capability; fine-tuning on a few good demonstrations was often enough to recover the hidden performance. Fine-tuning was less reliable when good demonstrations were unavailable, which is exactly the case once a model's hidden ability exceeds its demonstrators'. Hofstätter and colleagues compared techniques on model organisms built to resist elicitation: prompting recovered hidden capability on multiple-choice questions, activation steering did not, and on a code-generation task only fine-tuning did, so they recommend fine-tuning wherever it is possible. The general lesson is that an evaluation should report which techniques it used, how much effort went into each, and whether stronger ones were tried.

Sandbagging#

The hardest elicitation problem is a model that underperforms on purpose. Van der Weij and colleagues showed that models can be prompted or fine-tuned to selectively underperform on dangerous capability evaluations while keeping performance on general benchmarks, and to target a specific lower score. Tice and colleagues found that adding noise to a sandbagging model's weights can paradoxically improve its performance, because the noise disrupts the underperformance mechanism before it disrupts the capability, which gives a detection signal. Sandbagging can also be done by developers, by choosing weak prompts, scaffolds or budgets; it need not involve the model at all. The archive's note on sandbagging detection covers the statistical tests; for this article, the point is that under-elicitation, deliberate or not, is a bias, and biases behave differently from noise.

Capability thresholds#

Many developers now publish safety policies that define capability thresholds: levels of capability at which a model would significantly increase someone's ability to cause severe harm relative to existing tools, or would introduce a qualitatively new kind of risk. Crossing one commits the developer to specific safeguards, such as stronger security for the weights, deployment restrictions or additional evaluation, before proceeding. The thresholds are usually stated qualitatively (for example, a model that could significantly help someone with only basic technical training produce a known biological threat) and then operationalised through evaluations with numeric criteria.

Because evaluations are imperfect and capability grows between evaluations, policies typically include a buffer: an early warning or alert threshold, set below the level of real concern, which triggers more intensive evaluation or preparation of safeguards. The size of that buffer should depend on two things: how noisy the evaluation is, and how large the elicitation gap might be. The next section makes that dependence concrete.

The statistics of a threshold decision#

Consider a suite of $n$ tasks representative of a dangerous capability, and a threshold $\tau$: if the model's true success rate at full elicitation, $p$, reaches $\tau$, safeguards must apply. The evaluation does not see $p$. It sees $k$ successes out of $n$ under its own elicitation, which may fall short of full elicitation by a gap $g$ measured in logits:

where $\sigma$ is the logistic function and $\mathrm{logit}(p)=\log \frac{p}{1-p}$. A gap of one logit turns a true 50% into an observed 27% and a true 70% into 46%. Two decision rules are common:

• Point rule: flag if $k/n\ge \tau$.
• Upper-bound rule: flag unless the evaluation can rule out reaching $\tau$, that is, flag if the one-sided 95% upper confidence bound on the success rate is at least $\tau$. We use the Wilson bound,

The probability of flagging, as a function of the true capability, is the rule's operating characteristic: $P(\text{flag}\mid p)={\sum }_k\mathrm{Binom}(k;n,p_{\text{obs}})1[\text{rule flags at }k]$. We compute it exactly.

Worked numbers#

Threshold $\tau =50\%$. The table gives the probability of flagging a model exactly at the threshold, the probability of flagging a model ten points below it, and the true capability at which each rule flags with 95% probability.

Read the rows in pairs. With full elicitation, the point rule is a coin flip at the threshold: a model exactly at 50% is flagged about half the time, regardless of the number of tasks. The upper-bound rule fixes that, flagging 94% of models at the threshold, at the cost of more false alarms on models below it; with 200 tasks it keeps the protection and cuts false alarms at 40% from 55% to 11%. That is what more tasks buy: sharper discrimination around the threshold.

Now add an elicitation gap of one logit. The upper-bound rule, which was conservative a moment ago, flags a model at the threshold 3% of the time, and only reliably flags models whose true capability is above 73%. More tasks make this slightly worse (the confidence bound tightens around the biased estimate), not better. Sample size cures noise, not bias. The only remedies for an elicitation gap are better elicitation or a threshold buffer at least as large as the plausible gap.

Computing a rule's operating characteristic#

from math import lgamma, log, exp, sqrt

def wilson_upper(k, n, z=1.645):                  # one-sided 95% upper bound
    ph = k / n
    return (ph + z*z/(2*n) + z*sqrt(ph*(1-ph)/n + z*z/(4*n*n))) / (1 + z*z/n)

def p_flag(p_true, n, tau, gap=0.0, rule="upper"):
    """Exact probability that an n-task evaluation flags threshold tau."""
    lo = log(p_true / (1 - p_true)) - gap             # under-elicitation shifts the logit down
    po = 1 / (1 + exp(-lo))
    total = 0.0
    for k in range(n + 1):
        hit = (k / n >= tau) if rule == "point" else (wilson_upper(k, n) >= tau)
        if hit:
            total += exp(lgamma(n+1) - lgamma(k+1) - lgamma(n-k+1) + k*log(po) + (n-k)*log(1-po))
    return total

for rule in ("point", "upper"):
    print(rule, [round(p_flag(p, 50, 0.5, rule=rule), 3) for p in (0.4, 0.45, 0.5)],
          [round(p_flag(p, 50, 0.5, gap=1.0, rule=rule), 3) for p in (0.5, 0.6)])
# point [0.098, 0.284, 0.556] [0.0, 0.025]
# upper [0.554, 0.803, 0.941] [0.03, 0.302]

Uncertainty beyond sampling#

The binomial model captures only two of the uncertainties in a capability judgement. Others are harder to quantify and often larger.

Capability grows between evaluations. Evaluations happen at checkpoints, and capabilities improve with further training, better scaffolds and new elicitation tricks discovered after release. Kwa and colleagues found that the length of software tasks frontier models complete with 50% reliability has doubled roughly every seven months since 2019. If a threshold is checked once per release, the buffer has to cover growth until the next check, including post-deployment improvements by users. Forecasting methods such as observational scaling laws (Ruan, Maddison and Hashimoto) and the archive's note on eval scaling prediction help here.

Proxy validity. A knowledge benchmark is a proxy for uplift, and a capture-the-flag challenge is a proxy for real intrusions. Saturating a proxy does not show the real-world capability, and failing it does not rule the capability out. Item response theory, covered in IRT for LLM evals, helps make the link between task difficulty and capability explicit.

Human baselines. "Uplift" is relative, so it needs a baseline of what people can do without the model. RE-Bench, for example, compared agents with human experts given the same time budget on the same tasks, and GPQA's design used experts and skilled non-experts with web access to calibrate how resistant to web search its questions were. Without a baseline, a score has no meaning for risk.

Stated assumptions. Barnett and Thiergart argue that evaluation-based regulation only works if developers declare and justify the assumptions their safety claims rest on, such as that all relevant threat models were covered and that elicitation was close to what an adversary could achieve. Writing these down is the qualitative counterpart of the buffer in the table.

Designing a dangerous capability evaluation: a checklist#

The same structure applies, at much lower stakes, to any evaluation that gates a decision: decide the rule before seeing the result, make the rule err in the direction whose mistakes are cheaper, and remember that a confidence interval only covers the uncertainty you modelled. The LLM eval plan template has fields for each, and statistical significance for evals covers the interval mechanics.

What dangerous capability evaluations cannot tell you#

They do not measure intent: a model with a dangerous capability may never use it, and a model without it may still cause harm through others' misuse of modest capabilities at scale. They do not measure the effect of safeguards on misuse in the wild, which depends on jailbreak robustness and monitoring. And they cannot prove absence: the best they can do is make a credible argument that capability is below a threshold with a stated confidence, under stated elicitation, at a stated time. Their value lies in making that argument explicit and checkable, which is also why they sit at the centre of the evaluation side of AI safety and alongside safety benchmarks and red teaming in any serious assessment.

Frequently asked questions#

Sources#

1. Shevlane, Farquhar, Garfinkel, Phuong et al.. Model evaluation for extreme risks, 2023. arxiv.org
2. Rein, Hou, Stickland, Petty et al.. GPQA: A Graduate-Level Google-Proof Q&A Benchmark, 2023. arxiv.org
3. Kinniment, Sato, Du, Goodrich et al.. Evaluating Language-Model Agents on Realistic Autonomous Tasks, 2023. arxiv.org
4. Li, Pan, Gopal, Yue et al.. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning, 2024. arxiv.org
5. Phuong, Aitchison, Catt, Cogan et al.. Evaluating Frontier Models for Dangerous Capabilities, 2024. arxiv.org
6. Ruan, Maddison, Hashimoto. Observational Scaling Laws and the Predictability of Language Model Performance, 2024. arxiv.org
7. Greenblatt, Roger, Krasheninnikov, Krueger. Stress-Testing Capability Elicitation With Password-Locked Models, 2024. arxiv.org
8. van der Weij, Hofstätter, Jaffe, Brown, Ward. AI Sandbagging: Language Models can Strategically Underperform on Evaluations, 2024. arxiv.org
9. Laurent, Janizek, Ruzo, Hinks et al.. LAB-Bench: Measuring Capabilities of Language Models for Biology Research, 2024. arxiv.org
10. Zhang, Perry, Dulepet, Ji et al.. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models, 2024. arxiv.org
11. Chan, Chowdhury, Jaffe, Aung et al.. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, 2024. arxiv.org
12. Barnett, Thiergart. Declare and Justify: Explicit assumptions in AI evaluations are necessary for effective regulation, 2024. arxiv.org
13. Wijk, Lin, Becker, Jawhar et al.. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts, 2024. arxiv.org
14. Tice, Kreer, Helm-Burger, Shahani et al.. Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models, 2024. arxiv.org
15. Hofstätter, van der Weij, Teoh, Djoneva et al.. The Elicitation Game: Evaluating Capability Elicitation Techniques, 2025. arxiv.org
16. Kwa, West, Becker, Deng et al.. Measuring AI Ability to Complete Long Software Tasks, 2025. arxiv.org