Evaluating LLMs for clinical use: severity-weighted harm, escalation and small expert sets

Clinical evaluation of language models goes wrong in a predictable way: teams report accuracy, and accuracy treats a missed drug interaction the same as a clumsy sentence. This note sets out an evaluation design built around harm instead. Physicians write the rubrics, errors are weighted by severity, abstaining and escalating are scored as real outcomes, and uncertainty is reported honestly for the small expert-labelled sets that clinical work allows. A seeded simulation shows how often accuracy and harm disagree about which model is better, and how many labelled cases it takes to tell them apart.

Two models, one decision#

Imagine two assistants answering clinicians' questions about medication and triage. Physicians have sorted the 1,000 test cases into three tiers by what a wrong answer could cause: 60% low severity (a wrong answer is an inconvenience), 30% moderate (a wrong answer could delay care), and 10% high (a wrong answer could cause serious harm). The outcome rates are:

Model A is more accurate: 89.6% against 82.2%. It never declines to answer. Model B is less accurate overall, but on the high-severity cases it escalates to a human a quarter of the time and makes a harmful error only 5% of the time, against A's 20%. Which is better depends entirely on how much a high-severity error costs relative to everything else. Accuracy cannot answer that question, because it has no place to put the answer.

These rates are illustrative, chosen for the simulation in this note. They are not measurements of any real model.

Severity-weighted harm#

Let ${\pi }_s$ be the share of cases in severity tier $s$, $P_m(\text{err}\mid s)$ and $P_m(\text{abs}\mid s)$ the rates at which model $m$ makes a harmful error or abstains in that tier, $h_s$ the harm weight of an error in tier $s$, and $c_a$ the cost of an abstention. The expected harm per case is

Accuracy is the special case where every error costs the same and abstaining counts as an error. Weighting by severity says, explicitly, that some mistakes matter more than others. Pricing abstention says that escalating to a clinician is not free: it costs clinician time and delays an answer, but it is usually far cheaper than a serious error.

Worked example#

With weights $h=(1,5,50)$ for low, moderate and high severity, and $c_a=1$:

• Model A: $0.6\times 0.08\times 1+0.3\times 0.12\times 5+0.1\times 0.20\times 50=0.048+0.180+1.000=1.228$.
• Model B: $0.6\times (0.13+0.03)+0.3\times (0.12\times 5+0.06)+0.1\times (0.05\times 50+0.23)=0.096+0.198+0.273=0.567$.

B carries less than half of A's expected harm, although it is 7.4 points less accurate. Because $H$ is linear in each weight, the crossover is easy to find. Holding the other weights fixed, the two models are equally harmful when the high-severity weight is 5.93. Above that, B is safer; below it, A is. A clinical team that believes a serious error is worth more than about six minor ones should prefer B.

Two practical points follow. First, the case mix of the test set has to match the deployment, or at least be reweighted to it: an evaluation set enriched for dramatic cases will favour cautious models, and one built from routine questions will favour confident ones. Second, the weights are a clinical and governance decision. They should be written down by the clinicians accountable for the deployment, before results are seen, and published with the results.

Physician-written rubrics and severity labels#

Harm weighting needs two kinds of expert input per case: what a good answer contains, and how bad a wrong one would be. The first is a rubric (see our note on rubric design and grader noise). HealthBench is the largest public example: 5,000 multi-turn health conversations scored against 48,562 conversation-specific criteria written by 262 physicians, with criteria that carry positive or negative points [1]. Earlier clinical evaluations of Med-PaLM asked clinicians to rate answers along several axes, including the likelihood and extent of possible harm [2], and the follow-up work extended this with physician comparisons on long-form answers [3]. Benchmarks built from exam questions, such as MedQA [4], measure knowledge well but say little about harm, because a wrong option on an exam has no severity.

In practice we ask physicians for three labels per case:

1. Criteria. Atomic, checkable statements of what the answer must and must not contain, each with a weight. "Recommends urgent assessment for chest pain with shortness of breath" rather than "triages appropriately".
2. Severity tier. The worst plausible consequence of acting on a wrong answer to this case, on a short anchored scale. Severity is a property of the case, labelled before any model output is seen.
3. Escalation expectation. Whether a well-designed assistant should answer, answer with a caveat, or hand the case to a clinician. This turns abstention from a nuisance into something that can be right or wrong.

Two physicians label each case independently and a third adjudicates disagreements. Agreement on severity tiers is usually lower than on factual criteria, so it is worth measuring separately and reporting.

Scaling physician judgement with model graders#

Physicians cannot grade every output of every model version. The workable pattern is to keep physicians on the parts that need clinical judgement (writing criteria, assigning severity, adjudicating) and let a model grader apply the criteria at scale, after it has been checked against physicians criterion by criterion. HealthBench takes this approach, using a model-based grader and reporting a meta-evaluation of it against physician judgement [1].

The check has to be done per criterion and per severity tier, not in aggregate. A grader that agrees with physicians 95% of the time overall can still miss most harmful errors on high-severity cases, because those are a small share of the items it sees. What matters for the harm metric is the grader's sensitivity to harmful errors in the high tier, measured on a set where such errors are common enough to count. If that sensitivity is 80%, the grader under-reports the dominant term of $H$ by a fifth, and it does so for every model it grades. That bias shrinks real differences between models in the same way that rubric grader noise does, and it can be corrected only if it has been measured.

Two further rules keep model grading honest in clinical settings. First, the grader should never see which model produced an output, and should not be from the same model family as a model under test. Second, a fixed share of outputs goes back to physicians on every run (10% per run, stratified by tier, is a reasonable default), so drift in the grader shows up as a change in agreement before it shows up as a change in a model's score.

Abstention and escalation#

An assistant that declines everything has zero harmful errors and is useless. An assistant that never declines is maximally helpful and maximally exposed. The harm formula handles this trade-off through $c_a$, but only if abstentions are scored against the escalation expectation:

Report the appropriate escalation rate on cases that should be escalated, and the unnecessary escalation rate on cases that should be answered, as two separate numbers. A survey of abstention in language models gives a useful taxonomy of when and why models decline [9]. For selective prediction more generally (covered in our note on calibration and selective prediction), the right summary is a risk and coverage curve: harm per answered case against the share of cases answered, as the abstention threshold moves.

Uncertainty with small expert-labelled sets#

Physician time is the binding constraint, so clinical test sets are small, and harm is a heavy-tailed metric: most cases contribute nothing, and a few high-severity errors contribute almost everything. The variance of the estimated harm per case over $N$ cases is

The $h_s^2$ term means the rare high-weight tier dominates the variance. For model A at the default weights, the high tier contributes 50 of the 51 units inside the bracket. Accuracy has no such tail, so an accuracy comparison stabilises with far fewer cases than a harm comparison.

Setup (simulation). We drew $N$ cases from the tier mix above, sampled each model's outcome from its rates, and computed a paired bootstrap 95% interval (1,000 resamples, pairing on cases) for the difference in harm and in accuracy. Each $N$ was repeated 1,000 times.

The uncomfortable implication: with a typical 200 to 400 case expert set, you will usually be able to say confidently that A is more accurate, and usually not be able to say confidently that B is safer, even though B's harm advantage is large. Teams that act only on statistically significant differences will drift toward the accurate model by default. The remedy is to enrich the test set with high-severity cases (and reweight by ${\pi }_s$ when reporting), so the tier that drives the decision is not the tier with the least data.

When you observe zero severe errors#

A common report reads "no severe errors in 50 high-severity cases". The exact one-sided 95% upper bound on the true rate after zero events in $n$ trials is $1-0.05^{1/n}$, which is close to the "rule of three", $3/n$:

Zero errors in 50 cases is consistent with a true rate near 6%. Report the bound, not the zero.

Subgroup performance#

Clinical models can fail unevenly across patient groups, languages and care settings; one study found that several widely used chatbots repeated race-based medical claims that have been discredited [5]. Subgroup analysis is required, but the sample sizes are smaller still.

Setup (simulation). A subgroup makes up 15% of cases. In it, model B's harmful-error rate on moderate and high-severity cases is three times its rate elsewhere (30.8% against 10.3%, pooled over those tiers). We tested for the gap with a two-proportion $z$-test and computed Wilson intervals [10] for the subgroup rate, 1,000 simulated studies per size.

A threefold difference is large, and it is still missed almost a third of the time at 400 cases. Smaller differences, which are the realistic ones, need either much larger sets or deliberate oversampling of the subgroups you are accountable for. Report subgroup intervals even when no test is significant; an interval 34 points wide is information.

Where this sits in regulation and reporting#

This note describes evaluation design, not regulatory classification, but the evaluation has to produce evidence that fits the frameworks in force at the time of writing:

• United States. Software that supports clinical decisions may or may not be a medical device. FDA's guidance on Clinical Decision Support Software, revised in January 2026, interprets the statutory criteria for non-device CDS, including whether a clinician can independently review the basis for a recommendation [6]. For AI-enabled devices, FDA's January 2025 draft guidance on lifecycle management and marketing submissions sets out expectations for performance testing, including subgroup analyses [7].
• European Union. Under the AI Act, AI systems that are medical devices, or safety components of them, and that require third-party conformity assessment are classified as high-risk, with obligations covering data governance, accuracy and human oversight [8].
• Guidance and reporting. WHO's 2024 guidance on large multi-modal models in health sets out governance expectations for developers and deployers [11]. For reporting evaluation results, TRIPOD+AI gives a checklist for prediction-model studies that covers subgroup performance and handling of uncertainty [12].

None of these frameworks prescribes harm weights. They do expect that the evidence is relevant to the intended use, that limitations are stated, and that performance is examined across relevant populations, which is what severity tiers, escalation scoring and subgroup intervals provide.

Failure modes#

Evaluation design checklist#

1. Define the intended use and the population of questions; sample the test set from it.
2. Have physicians label severity tier and escalation expectation per case, blind to outputs.
3. Have physicians write atomic criteria with positive and negative weights; measure agreement per criterion.
4. Fix harm weights $h_s$ and abstention cost $c_a$ in writing before any results are seen.
5. Enrich the high-severity tier so it holds at least 100 to 150 cases; reweight to deployment shares when reporting.
6. Report accuracy, harm, appropriate and unnecessary escalation rates, each with paired bootstrap intervals.
7. Report subgroup rates with Wilson intervals, and upper bounds where zero errors were observed.
8. Rerun the full evaluation on every model or prompt change, with severity labels unchanged.

Code: harm with a paired bootstrap#

import numpy as np

def harm_compare(sev, out_a, out_b, h=(1, 5, 50), c_abs=1.0, n_boot=2000, seed=0):
    """sev: tier per case (0, 1, 2). out_*: 0 correct, 1 harmful error, 2 abstain."""
    rng = np.random.default_rng(seed)
    h = np.asarray(h, float)
    def case_harm(o):
        return np.where(o == 1, h[sev], 0.0) + np.where(o == 2, c_abs, 0.0)
    d = case_harm(out_a) - case_harm(out_b)          # positive: B safer
    acc = (out_a == 0).astype(float) - (out_b == 0)  # positive: A more accurate
    idx = rng.integers(0, len(d), (n_boot, len(d)))  # resample cases, paired
    ci = lambda x: np.quantile(x[idx].mean(1), [0.025, 0.975])
    return dict(harm_diff=d.mean(), harm_ci=ci(d), acc_diff=acc.mean(), acc_ci=ci(acc))

Pairing on cases matters: both models answer the same questions, so the case-to-case variation in difficulty cancels in the difference.

Frequently asked questions#

Sources#

1. R. K. Arora, J. Wei, R. S. Hicks et al.. HealthBench: Evaluating Large Language Models Towards Improved Human Health, 2025. arxiv.org
2. K. Singhal, S. Azizi, T. Tu et al. Nature 620:172-180. Large language models encode clinical knowledge, 2023. doi.org
3. K. Singhal, T. Tu, J. Gottweis et al. Nature Medicine. Toward expert-level medical question answering with large language models, 2025. doi.org
4. D. Jin, E. Pan, N. Oufattole et al.. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams, 2020. arxiv.org
5. J. A. Omiye, J. C. Lester, S. Spichak et al. npj Digital Medicine. Large language models propagate race-based medicine, 2023. doi.org
6. U.S. Food and Drug Administration. Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff (revised January 2026), 2026. fda.gov
7. U.S. Food and Drug Administration. Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations (draft guidance), 2025. fda.gov
8. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act), 2024. eur-lex.europa.eu
9. B. Wen, J. Yao, S. Feng et al.. Know Your Limits: A Survey of Abstention in Large Language Models, 2024. arxiv.org
10. E. B. Wilson. Journal of the American Statistical Association. Probable Inference, the Law of Succession, and Statistical Inference, 1927. doi.org
11. World Health Organization. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models, 2024. who.int
12. G. S. Collins, K. G. M. Moons, P. Dhiman et al. BMJ. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods, 2024. doi.org