AI safety benchmarks: what they measure, over-refusal, validity and how to choose

AI safety benchmarks are fixed, public test sets that score language models on safety-relevant behaviour: refusing harmful requests, resisting jailbreaks, not refusing harmless ones, avoiding hazardous knowledge, telling the truth and treating groups fairly. They make models comparable, but each measures a narrow slice under fixed conditions, many are saturated or contaminated, and a surprising number mostly track general capability rather than safety.

This article goes through what the main public safety benchmarks measure, how refusal and over-refusal pull against each other, how to check whether a safety benchmark is really measuring safety, and how to choose and combine benchmarks for a given decision. Two small simulations, one of the refusal trade-off and one of capability leaking into a safety score, make the validity problems concrete.

What do the main AI safety benchmarks measure?#

Röttger and colleagues reviewed 144 open datasets for LLM safety evaluation and found practice highly idiosyncratic: model releases and popular leaderboards used only a small fraction of them, with gaps in non-English and naturalistic data. The table groups widely cited benchmarks by what they measure. Sizes and design details are as described by their authors.

Two further tools are adjacent rather than benchmarks of models: safety classifiers such as WildGuard, which are themselves evaluated on moderation test sets and are often used as the judge inside other benchmarks; and attack generators, whose strength determines what a robustness benchmark means. For jailbreak benchmarks in particular, see LLM jailbreaks explained on why a success rate needs an attack budget. For capability benchmarks in general, LLM benchmarks covers saturation and contamination.

Refusal and over-refusal: two numbers, not one#

The most common safety number is a refusal rate on harmful prompts. On its own it is gameable by the simplest possible policy: refuse everything. Röttger and colleagues built XSTest because models were refusing clearly safe prompts that used language similar to unsafe ones, for example a question about how to kill a Python process. Any refusal benchmark should be paired with an over-refusal benchmark, and the pair should be read as a trade-off curve.

A simple model shows why. Suppose a model assigns each prompt a refusal propensity score and refuses when it exceeds a threshold $t$. On harmful prompts the score is normally distributed with mean 2; on borderline prompts that are actually benign (the XSTest kind) it has mean 0.5; on clearly benign prompts, mean $-2$; all with standard deviation 1. Then

where $\Phi$ is the standard normal distribution function. Moving $t$ slides the model along one curve:

Three things follow. A benchmark of harmful prompts alone ranks the $t=0.5$ setting best, even though it refuses half of all borderline benign requests. A benchmark of clearly benign prompts alone barely registers any over-refusal at any setting, which is why over-refusal sets must be built from borderline prompts. And the only way to improve both numbers at once is to separate the two distributions, which moves the whole curve towards the origin; Model B in Figure 1, whose harmful and borderline scores are 2.5 standard deviations apart instead of 1.5, does better at every threshold. Benchmark results should be reported as a point on such a curve, ideally with the curve. The archive's note on over-refusal calibration fits this kind of model to real refusal data, and guardrail thresholds covers choosing $t$.

Does a safety benchmark measure safety? Safetywashing#

Ren and colleagues asked a sharper question. For each of many safety benchmarks, they computed a "capabilities score" for a set of models from their results on standard capability benchmarks (the first principal component, which explained around 70% of performance variation), then measured the Spearman correlation between that score and the safety benchmark across models. A high correlation means the benchmark largely tracks general capability, so a bigger or better-trained model will improve on it without any safety-specific work. Presenting such gains as safety progress is what they called safetywashing.

The findings varied by area. TruthfulQA's multiple-choice score had a capabilities correlation of 81% across chat models. Benchmarks of recognising ethical considerations correlated highly, while MACHIAVELLI's behavioural measures did not. Static adversarial robustness benchmarks correlated with capabilities more than dynamic attacks did. Sycophancy got worse as capabilities rose, and the hazardous-knowledge score from WMDP correlated strongly negatively, as expected, since more capable models know more. The same pattern appears in honesty: the MASK authors found that larger models were more accurate but not more honest, and that most frontier models scored well on truthfulness benchmarks while lying under pressure.

A simulation of capability leaking into a safety score#

Suppose each of 40 models has a general capability $c$ and a safety property $s$ that is genuinely independent of capability. A benchmark's score mixes the two with a capability loading $\lambda$:

where $c$, $s$ and the noise $\epsilon$ are independent standard normal draws and $\lambda$ between 0 and 1 is how much of the benchmark is really capability. With one seeded draw of 40 models:

At $\lambda =0.7$ the benchmark already correlates more with capability than with the property it names, and its ranking of models agrees with the true safety ranking at only 0.64. Two practical points follow. First, the capability correlation is easy to compute and should be reported for any safety benchmark used to claim progress. Second, regressing out capability recovers much of the signal (a residual correlation of 0.91 at $\lambda =0.7$), so a capability-adjusted score, "safer than expected for its capability", is often the more honest headline. The sample capability-safety correlation in this draw is 0.10, not zero, which is why the $\lambda =0$ row shows a small correlation with capability: with 40 models, sampling noise alone produces correlations of that size.

Other limits of safety benchmarks#

Static and single-turn. Most benchmarks are fixed lists of single prompts. Real misuse is adaptive and multi-turn, and defences that look strong on static sets often fail against adaptive attackers; see adaptive attack evaluations. AILuminate's authors list single-turn interaction as a limitation of their own benchmark.

Saturation and training contamination. Once a benchmark is public, its prompts and close paraphrases end up in training data, and the most common patterns get trained against directly. Scores rise without the underlying behaviour generalising. Held-out paraphrases and private test sets are the standard checks; the archive's note on contamination detection covers how.

Judges. Almost every safety benchmark uses an automated judge to label responses as harmful, refusing or compliant. SimpleSafetyTests compared automated safety filters against its 3,000 human-labelled responses and found performance varied considerably: a widely used toxicity API reached 72% accuracy and the best option, a zero-shot prompt to a frontier model, 89%; StrongREJECT found that non-refusal judges overstate jailbreak success. A benchmark score inherits its judge's errors, so report the judge and its measured agreement with humans (see LLM-as-a-judge and LLM judge error).

Coverage. SORRY-Bench's authors found that across ten earlier datasets, self-harm instructions were over three times less represented than fraud. Aggregate refusal rates weight categories by how many prompts happen to be in each, which is rarely the weighting a deployment cares about.

Small samples and clustered items. Many safety sets have a few hundred items, often in clusters of paraphrases. A difference of two or three points between models is frequently inside the noise; use intervals computed over clusters (clustered standard errors) and see statistical significance for evals.

Context. A benchmark measures a model under the benchmark's system prompt, sampling settings and scaffold. Your deployment has its own. Treat public results as a prior and re-run the relevant benchmarks in your own configuration.

How to choose AI safety benchmarks#

Start from the decision, not the leaderboard. The questions below map a decision to the benchmarks and companion checks that bear on it.

Across all of them, five habits make benchmark results trustworthy:

• Report refusal and over-refusal together, as a trade-off.
• Report the capability correlation of any safety benchmark used to claim progress, and consider a capability-adjusted score.
• Name the judge, its agreement with human labels, and the attack budget for robustness results.
• Give intervals that respect clustering, and do not rank models whose intervals overlap.
• Re-run in your own deployment configuration and supplement with private, domain-specific prompts that cannot have leaked.

Checking a benchmark's capability correlation#

import numpy as np

def capability_correlation(safety_scores, capability_scores):
    """Spearman correlation across models, as in the safetywashing analysis."""
    rank = lambda x: np.argsort(np.argsort(x))
    return float(np.corrcoef(rank(safety_scores), rank(capability_scores))[0, 1])

def residual_safety(safety_scores, capability_scores):
    """Safety score with the linear capability trend removed."""
    beta = np.polyfit(capability_scores, safety_scores, 1)
    return safety_scores - np.polyval(beta, capability_scores)

rng = np.random.default_rng(1)
cap = rng.standard_normal(40)                      # capability score per model
true_safety = rng.standard_normal(40)              # the property we care about
bench = 0.8 * cap + 0.6 * true_safety + 0.3 * rng.standard_normal(40)
print(round(capability_correlation(bench, cap), 2),
      round(float(np.corrcoef(residual_safety(bench, cap), true_safety)[0, 1]), 2))
# 0.76 0.85

With real data, the capability scores come from the same models' results on standard capability benchmarks (or the first principal component of several). If a safety benchmark's capability correlation is above about 0.7, treat any improvement on it as unproven until the capability-adjusted score also improves.

What to take away#

Safety benchmarks are indispensable for comparison and nearly useless as standalone assurance. Each measures one behaviour under one set of conditions, through one judge. Read refusal next to over-refusal, robustness next to attack budget, and every safety score next to its correlation with capability. For how benchmarks fit with the rest of safety evaluation, see what is AI safety and what are AI evals; for building your own evaluation set, the LLM evaluation guide and safety training data guide.

Frequently asked questions#

Sources#

1. Gehman, Gururangan, Sap, Choi, Smith. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models, 2020. arxiv.org
2. Lin, Hilton, Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods, 2021. arxiv.org
3. Parrish, Chen, Nangia, Padmakumar et al.. BBQ: A Hand-Built Bias Benchmark for Question Answering, 2021. arxiv.org
4. Liang, Bommasani, Lee, Tsipras et al.. Holistic Evaluation of Language Models, 2022. arxiv.org
5. Pan, Chan, Zou, Li et al.. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark, 2023. arxiv.org
6. Wang, Chen, Pei, Xie et al.. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models, 2023. arxiv.org
7. Röttger, Kirk, Vidgen, Attanasio et al.. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023. arxiv.org
8. Wang, Li, Han, Nakov, Baldwin. Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs, 2023. arxiv.org
9. Vidgen, Scherrer, Kirk, Qian et al.. SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models, 2023. arxiv.org
10. Huang, Sun, Wang, Wu et al.. TrustLLM: Trustworthiness in Large Language Models, 2024. arxiv.org
11. Mazeika, Phan, Yin, Zou et al.. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org
12. Li, Dong, Wang, Hu et al.. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models, 2024. arxiv.org
13. Souly, Lu, Bowen, Trinh et al.. A StrongREJECT for Empty Jailbreaks, 2024. arxiv.org
14. Li, Pan, Gopal, Yue et al.. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning, 2024. arxiv.org
15. Chao, Debenedetti, Robey, Andriushchenko et al.. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models, 2024. arxiv.org
16. Röttger, Pernisi, Vidgen, Hovy. SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety, 2024. arxiv.org
17. Cui, Chiang, Stoica, Hsieh. OR-Bench: An Over-Refusal Benchmark for Large Language Models, 2024. arxiv.org
18. Xie, Qi, Zeng, Huang et al.. SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal, 2024. arxiv.org
19. Han, Rao, Ettinger, Jiang et al.. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs, 2024. arxiv.org
20. Ren, Basart, Khoja, Gatti et al.. Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?, 2024. arxiv.org
21. Andriushchenko, Souly, Dziemian, Duenas et al.. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents, 2024. arxiv.org
22. Ghosh, Frase, Williams, Luger et al.. AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons, 2025. arxiv.org
23. Ren, Agarwal, Mazeika, Menghini et al.. The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems, 2025. arxiv.org