Why exam benchmarks aren't enough#
Singhal et al. [1] showed in 2023 that instruction-tuned LLMs could reach state-of-the-art accuracy on medical multiple-choice benchmarks, and in the same paper that human evaluation of long-form answers revealed important gaps: answers that clinicians rated less aligned with scientific consensus, or more likely to cause harm, than clinician-written ones. That pattern recurs. Multiple-choice accuracy measures recall and reasoning on a closed set; deployment risk lives in open-ended generation, retrieval over patient records and actions in clinical workflows.
A clinical harm taxonomy for test design#
| Harm type | Example failure | Grader |
|---|---|---|
| Commission: incorrect action | Wrong dose, contraindicated drug, wrong patient | Clinician rubric + deterministic checks against formulary |
| Omission: missing critical information | Fails to flag red-flag symptoms needing urgent care | Clinician rubric with required-element checklist |
| Unsupported claim | Fabricated guideline or citation | Reference-based groundedness grader |
| Inappropriate certainty | Diagnostic statement without hedging or escalation | Rubric judge, validated against clinicians |
| Over-refusal | Refuses a pharmacist's legitimate interaction question | Refusal-calibration set |
| Privacy | PHI from another patient surfaces through retrieval | Cross-patient retrieval probes; PII/PHI egress grader |
| Equity | Quality differs across demographic or language groups | Stratified metrics with intervals per group |
Clinicians as graders, and when a judge can help#
Clinical correctness and harm need clinician ground truth. The practical design is a calibration set labelled by at least two clinicians per item, with disagreement adjudicated by a senior reviewer, and a model judge validated against that set per category. In our experience, judges are reliable for structural checks (required elements present, citation exists and supports the claim) and unreliable for subtle harm judgements, which stay with clinicians. The arithmetic of judge error matters even more here: at a low true harm rate, an uncorrected judge mostly reports its own false positives (worked example).
Over-refusal is a patient-safety issue#
Safety tuning applied without calibration makes clinical assistants refuse medication, dosing and symptom questions that clinicians and patients legitimately ask. A refusal sends a nurse at 3 a.m. to a worse source. We evaluate harmful compliance and over-refusal on the same matched prompt sets, so improvements on one can't hide regressions on the other.
Documentation for regulated deployments#
Depending on intended use and jurisdiction, clinical AI may fall under medical-device regulation, and in the EU, AI that is a safety component of, or itself, a regulated medical device is high-risk under the AI Act via Annex I (with obligations now applying from August 2028; see timeline). Whatever the regime, keep: intended-use statement; harm taxonomy; dataset composition and representativeness; grader validation; stratified results with intervals; adversarial and privacy testing; and post-deployment monitoring.
Frequently asked questions#
How do you evaluate an LLM for clinical use?
Test the deployed system on open-ended clinical tasks with a harm taxonomy (commission, omission, unsupported claims, inappropriate certainty, over-refusal, privacy, equity), graded by clinicians or by model judges validated against clinician labels, with stratified results and confidence intervals.
Are medical exam benchmarks enough to evaluate healthcare LLMs?
No. Multiple-choice accuracy doesn't capture open-ended answer safety, retrieval over patient data or workflow actions. Studies such as Singhal et al. (Nature, 2023) found gaps in long-form answers even where benchmark accuracy was high.
Sources#
- K. Singhal et al. Large language models encode clinical knowledge. Nature 620:172–180, 2023. doi.org
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Annex III. eur-lex.europa.eu