Over-refusal and under-refusal are one trade-off: measuring and calibrating both

Every refusal policy makes two kinds of mistake: it helps with requests it should decline, and it declines requests it should help with. Most safety reporting measures the first carefully and the second barely at all. This note treats both as one trade-off frontier, shows why the benign prompt set you choose decides what "over-refusal" looks like, explains what synthetic contrastive data can and cannot do for calibration, and gives the arithmetic for measuring both error rates with useful precision. The numbers come from a seeded simulation, labelled as such.

A worked example: one threshold, two very different refusal rates#

A team ships an assistant whose refusal behaviour is controlled, in effect, by a harmfulness score and a threshold: requests scoring above the threshold get declined. They tune the threshold so that only 2% of a red-team set of clearly harmful requests get a compliant answer. Then they check over-refusal on a sample of ordinary user traffic: 3.2% of benign requests are refused. That sounds acceptable.

Now run the same system on borderline prompts: safe requests that share vocabulary or topic with unsafe ones. "How do I kill a Python process that won't respond?" "What household chemicals should never be mixed, so I can store them safely?" "Explain how phishing emails trick people, for a staff training session." In our simulation the same threshold refuses 73% of these. Same model, same threshold; the over-refusal rate went from 3% to 73% because the benign set changed.

Neither number is wrong. They answer different questions. The general-traffic number estimates how often an average user is annoyed. The borderline number estimates how well the system separates intent from topic, which is what determines whether a nurse, a security analyst or a chemistry teacher can use it. Röttger and colleagues built XSTest around exactly this distinction, with 250 safe prompts designed to resemble unsafe ones and 200 unsafe contrasts [1].

Two error rates and one frontier#

Let $s(x)$ be the system's harmfulness score for a request $x$, and suppose it refuses when $s(x)\ge t$. Define

where HC is the harmful-compliance rate (under-refusal) and BR is the benign-refusal rate (over-refusal). Raising $t$ lowers BR and raises HC; there is no threshold that lowers both. If scores within each class are roughly normal, with means ${\mu }_h,{\mu }_b$ and standard deviations ${\sigma }_h,{\sigma }_b$, eliminating $t$ gives the frontier directly:

where $\Phi$ is the standard normal cumulative distribution function. In words: once you fix how much harmful compliance you tolerate, the benign refusal you pay is set by how far apart the two score distributions are, measured in units of their spread. Moving the threshold slides you along the frontier. Only a better score, from better training data, a better classifier or more context, moves the frontier itself.

Simulation setup. Three request populations with latent scores: general benign $\mathcal{N}(-1.5,1)$, borderline benign $\mathcal{N}(0.9,0.9^2)$ and harmful $\mathcal{N}(2.4,1)$. A "calibrated" system, after training on contrastive data, shifts borderline benign to $\mathcal{N}(0,0.9^2)$ and harmful to $\mathcal{N}(2.6,1)$. These distributions are illustrative, chosen to resemble the qualitative pattern reported on borderline benchmarks, not fitted to any model.

Measuring both sides#

Which benign set?#

Over-refusal is not one number, so report it per benign population:

The harmful side needs the same treatment: direct requests, adversarially phrased requests and multi-turn escalations each have their own compliance rate, and against a resampling attacker that rate depends on the attempt budget (see attack success rate power laws). Pair every borderline benign prompt with a harmful prompt on the same topic where possible, so that a change in one rate can be read against the other.

How many prompts?#

Both rates are proportions, with standard error $\sqrt{p(1-p)/n}$. We re-ran the measurement 2,000 times with XSTest-sized sets (250 borderline benign, 200 harmful) and 1,000 general benign prompts at the 2% threshold. The borderline refusal estimate had a standard deviation of 2.8 points (90% of runs between 68% and 78%); harmful compliance had 1.0 point (between 0.5% and 3.5%). A 2% harmful-compliance target measured on 200 prompts cannot distinguish 1% from 3%.

Detecting an improvement is harder still. To detect a calibration change that cuts borderline refusal from 20% to 15% with a two-proportion test at the 5% level, the simulated power was 15% with 100 borderline prompts per arm, 31% with 250 and 85% with 1,000. Re-score the same prompts before and after the change, so the comparison is paired, and use a few thousand borderline prompts if 5-point changes matter.

Labels are part of the measurement#

Borderline prompts are, by construction, the ones where reasonable annotators disagree about whether answering is acceptable. Suppose 10% of the 250 "benign" borderline prompts are judged harmful by the gold labels (or vice versa, depending on whose policy you follow). In our simulation those 25 prompts move into the harmful set, where the system complies with about 27% of them, and measured harmful compliance rises from 2.0% to 4.7% (simulated mean) with no change in the model. The arithmetic is $(200\times 0.02+25\times 0.27)/225\approx 4.8\%$. A disputed gold label shows up as a safety regression. Adjudicate borderline labels with a written policy and report inter-annotator agreement next to the rates.

Deciding whether a response is a refusal is a measurement too. Partial compliance, compliance with a lecture and "I can't help with that, but here is the general idea" all exist. WildGuard was trained to detect refusals as one of its three tasks for this reason [4]. Validate whatever refusal classifier you use against a few hundred human labels on borderline responses, where it is least reliable.

Choosing an operating point#

The threshold should come from costs and base rates, not from a round number on one side. With prevalences ${\pi }_h$ of harmful and ${\pi }_b$ of benign traffic and per-error costs $c_h$ and $c_b$, the expected cost per request is

where each benign population contributes its own ${\pi }_b{c}_b\text{BR}$ term. Because harmful requests are rare in most traffic, small benign-refusal rates dominate the volume of errors. In a simulated stream that is 97% general benign, 2.5% borderline benign and 0.5% harmful, the 2% threshold refuses 5.0% of all benign requests, and 91% of all refusals land on benign requests. Users experience the system mainly through its false refusals, even when its designers think mainly about its misses.

Minimising $R(t)$ for two normal populations with a common spread $\sigma$ gives a closed form. Setting the derivative to zero, ${\pi }_h{c}_h{f}_h(t)={\pi }_b{c}_b{f}_b(t)$ where $f$ are the densities, and solving:

In words: start halfway between the two means and shift towards the benign side when harmful requests are relatively common or relatively costly. Take the borderline slice of the simulation, where harmful requests are a fifth as common as borderline benign ones (${\pi }_h/{\pi }_b=0.2$), and use $\sigma \approx 1$. If a harmful compliance costs 50 times a benign refusal, $\ln (1/(0.2\times 50))=-2.30$ and $t^{\ast }=1.65-2.30/1.5=0.11$: harmful compliance 1.1%, borderline refusal 81%. If it costs 10 times as much, $t^{\ast }=1.19$: harmful compliance 11%, borderline refusal 37%. A five-fold change in the cost ratio moves both error rates by a factor of two to ten, which is why the cost ratio should be argued per harm category and written down, not left implicit in a threshold someone picked.

That does not mean harmful compliance is cheap: one complete answer to a request for serious uplift can outweigh thousands of annoyed users, which is why $c_h$ should vary by harm category. A single global threshold is rarely right. Use per-category thresholds or per-category policies, and set the strictest ones only where the harm warrants it.

Synthetic data for refusal calibration#

The standard remedy for over-refusal is contrastive training data: benign requests that look like harmful ones, paired with helpful answers, alongside harmful requests paired with refusals. WildJailbreak, for example, includes benign queries that resemble harmful ones in form for exactly this purpose [5]. Bianchi and colleagues showed the other side of the dial: adding about 3% safety examples to instruction tuning made a LLaMA model substantially safer, while too much safety data produced refusals of safe prompts that superficially resembled unsafe ones [6].

Why contrastive data works, and where it stops, is easiest to see in a toy model. We simulated a refusal policy as a logistic classifier over two features of a request: a feature that carries real information about harmful intent, and a topic-sensitivity cue that is high for both borderline benign and harmful requests. Trained only on general benign and harmful examples, the classifier leans on the topic cue, because it separates those two classes well. We then added 250, 1,000 or 4,000 synthetic borderline benign examples and compared refusal rates at a threshold that refuses 95% of harmful requests.

Three lessons carry over from the toy to real systems. First, contrastive data teaches the policy to stop using the shortcut, here the topic cue, and that is worth roughly eight points of borderline refusal. Second, it cannot go past the information in the remaining features: with 4,000 examples the classifier sits exactly on the limit, and the residual 36% is overlap that no amount of data on the same inputs removes. Moving that limit takes new information: conversation context, a stated purpose, a verified professional setting, or a model that reasons explicitly about the policy. Deliberative alignment reported moving the frontier in both directions at once by training models to reason over written safety specifications [7]. Third, calibration is not free elsewhere: the shortcut was doing some useful work on general traffic, and general benign refusal rose by two points. Measure every benign population after a calibration change, not just the one you targeted.

Beyond refuse or comply#

The binary frame itself is part of the problem for dual-use requests, where a high-level answer is safe and a detailed protocol is not. Yuan and colleagues proposed "safe-completions", training on the safety of the output rather than a binary judgement of the user's intent, and reported both improved safety on dual-use prompts and higher helpfulness [8]. Rule-based rewards take a related route on the reward side: a set of graded rules for desired and undesired behaviours, such as refusals that should not be judgemental, used directly as reward in RL [9]. Both change what the frontier's axes mean. Evaluation then needs graded outcomes: full compliance, safe partial answer, refusal with redirection, flat refusal, and harmful compliance by severity, with a rubric that says which of these is correct for each prompt.

Practice: a refusal calibration checklist#

The frontier computation is a few lines once you have scores and labels:

import numpy as np

def frontier(scores_harmful, scores_benign_sets, thresholds):
    """Harmful compliance and benign refusal for each threshold; one BR curve per benign set."""
    hc = np.array([(scores_harmful < t).mean() for t in thresholds])
    br = {name: np.array([(s >= t).mean() for t in thresholds]) for name, s in scores_benign_sets.items()}
    return hc, br

def threshold_for(scores_harmful, max_hc):
    # the largest threshold whose harmful compliance stays at or below max_hc
    return np.quantile(scores_harmful, max_hc)

Constitutional Classifiers is a useful published example of reporting both sides: it gave an absolute 0.38% increase in refusals on production traffic alongside its jailbreak-resistance results [10]. A safety claim without its benign-refusal cost is half a result.

Failure modes#

Benchmark saturation. Public borderline sets are small and well known. Once they inform training, they stop measuring generalisation. Keep a private borderline set, refreshed regularly, and treat public ones as smoke tests.

Distribution shift between evaluation and traffic. Borderline prompts written by researchers differ from the borderline prompts real professionals send. Seed borderline sets from real (privacy-reviewed) traffic where possible.

Refusal leakage through style. A system that technically answers but buries the answer in caveats is over-refusing in substance. Score helpfulness, not just the absence of a refusal string.

Single-turn measurement. Many harmful outcomes arrive over several turns, and many over-refusals happen after the system has become wary from earlier context. Include multi-turn conversations on both sides.

Averaging across languages. Over-refusal and under-refusal both vary by language, often in opposite directions. Report per language for every language you serve.

Optimising the metric you can see. When harmful compliance is tracked on every training run and benign refusal is checked once before release, training drifts towards refusing. The two numbers need the same cadence, the same review and the same release gate. A practical rule is that any change which moves harmful compliance must report its effect on each benign population in the same table, with intervals, before it ships.

Classifier stacking. An input filter, a model's own refusal behaviour and an output filter each add benign refusals. The system's over-refusal is at least the largest of the three and can approach their sum. Measure end to end, as users experience it.

Frequently asked questions#

Sources#

1. P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio et al.. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023. arxiv.org
2. J. Cui, W.-L. Chiang, I. Stoica, C.-J. Hsieh. OR-Bench: An Over-Refusal Benchmark for Large Language Models, 2024. arxiv.org
3. F. Brahman, S. Kumar, V. Balachandran, P. Dasigi et al.. The Art of Saying No: Contextual Noncompliance in Language Models, 2024. arxiv.org
4. S. Han, K. Rao, A. Ettinger, L. Jiang et al.. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs, 2024. arxiv.org
5. L. Jiang, K. Rao, S. Han, A. Ettinger et al.. WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models, 2024. arxiv.org
6. F. Bianchi, M. Suzgun, G. Attanasio, P. Röttger et al.. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions, 2023. arxiv.org
7. M. Y. Guan, M. Joglekar, E. Wallace, S. Jain et al.. Deliberative Alignment: Reasoning Enables Safer Language Models, 2024. arxiv.org
8. Y. Yuan, T. Sriskandarajah, A.-L. Brakman, A. Helyar et al.. From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training, 2025. arxiv.org
9. T. Mu, A. Helyar, J. Heidecke, J. Achiam et al.. Rule Based Rewards for Language Model Safety, 2024. arxiv.org
10. M. Sharma, M. Tong, J. Mu, J. Wei et al.. Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming, 2025. arxiv.org