Harmful compliance is a model carrying out a request that its policy says it should decline or handle differently, such as giving operational detail for a weapon, writing targeted harassment, or executing a dangerous tool call. It is the opposite error to over-refusal, and the two together describe how well a model draws the line.
How it is measured
Run a set of prompts that the policy says must not be fulfilled, grade each response, and report
Grading needs care. A response that starts with a disclaimer and then gives the content is compliance. A response that is technically compliant but contains nothing useful may not be. Good rubrics score whether the output provides meaningful uplift toward the harmful goal, and they are applied by calibrated graders or experts per harm category.
Direct versus adversarial
Compliance on plainly worded harmful requests is the baseline. Under attack, the same quantity is the attack success rate: the share of jailbreak attempts that produce harmful compliance. A model can have near-zero direct compliance and a high attack success rate, so both belong in the eval suite.
Worked example
Illustrative numbers: a model complies with 12 of 400 direct harmful requests: 3.0%, with a 95% Wilson interval of roughly 1.7% to 5.2%. Zero failures does not mean zero risk: if a model complies with none of 300 prompts, the "rule of three" gives an approximate 95% upper bound of . Rare, severe categories need many targeted prompts before a low rate is convincing.
Pitfalls
- Averaging across categories hides concentrated failures; report per category and weight by severity.
- Single-turn tests miss multi-turn escalation such as a crescendo attack.
- In agents, compliance includes actions, not just text, so tool calls must be graded too.
Public suites such as HarmBench (Mazeika et al., 2024) and StrongREJECT (Souly et al., 2024) provide categorised harmful behaviours and graders that score whether a response actually helps, not merely whether it fails to refuse.
Why it matters
Harmful compliance is the most direct measure of whether safety training and guardrails do their job. Reported alone it rewards refusing everything; reported with over-refusal it shows the real trade-off a team is choosing.
Related terms
Sources
- Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org
- Alexandra Souly et al.. A StrongREJECT for Empty Jailbreaks, 2024. arxiv.org