Calibration and abstention for LLMs: ECE pitfalls, Brier scores and risk-coverage curves

A model that reports "90% confident" should be right about nine times in ten when it says so. If it is, you can let it answer on its own above some threshold and route everything else to a person. This note covers how to measure whether LLM confidence means what it says (expected calibration error, its biases, and the Brier score), how token-probability and verbalised confidence differ, and how to turn a confidence score into an abstention policy with a risk you can defend. All numbers come from a seeded simulation we ran for this note, and the interactive figure regenerates the same dataset in your browser.

A model that says 76% and delivers 67%#

We simulated 3,000 short-answer questions. Each has a true probability that the model answers it correctly, and the model answers 67.0% of the 2,000 evaluation questions correctly. For every answer we generated three confidence signals of the kind teams actually have:

• Token probability (raw). The probability the model assigns to its own answer. We made it sharper than the truth (the log-odds are stretched by a factor of 1.8 and shifted up), which mimics what preference tuning tends to do to a well-calibrated pretrained model [9].
• Temperature-scaled. The same log-odds divided by a single temperature $T$ fitted on 1,000 held-out questions [1]. The fit gave $T=2.23$.
• Verbalised. A number the model writes when asked "how confident are you?", noisier and rounded to multiples of 5%, as such numbers usually are [6].

The raw token probability averages 76.1% on questions the model gets right 67.0% of the time. The verbalised score averages 85.5%, and 58.8% of all answers come with a stated confidence of 90% or more, while the model is right on only 67% of them overall. Neither is unusual. The question is how to measure the problem, and what to do about it.

Expected calibration error and the reliability diagram#

A confidence score $c$ is calibrated if, among all answers given confidence $c$, a fraction $c$ are correct: $\mathrm{Pr}(Y=1\mid C=c)=c$. The standard summary bins the answers by confidence and averages the gaps:

$B$ is the number of bins, $n_b$ the number of answers in bin $b$, $n$ the total, ${\bar{c}}_b$ the mean confidence in the bin and ${\bar{y}}_b$ the fraction correct. It is the average distance, weighted by how many answers fall where, between what the model claimed and what it delivered. A reliability diagram plots ${\bar{y}}_b$ against ${\bar{c}}_b$; a calibrated model sits on the diagonal [1].

On the evaluation set the raw token probability has an ECE of 10.7 points with 10 equal-width bins, the temperature-scaled version 4.0 points, and the verbalised score 18.6 points. So far this matches intuition. The trouble starts when you lean on ECE harder than it can bear.

Four ways ECE misleads#

1. A perfect model does not score zero#

ECE is computed from finite bins with finite samples, and $|{\bar{c}}_b-{\bar{y}}_b|$ is positive even when the true gap is zero, because ${\bar{y}}_b$ is noisy. The estimate is biased upwards, and the bias grows with the number of bins and shrinks with the sample size [3][10]. We simulated a perfectly calibrated model (its confidence equals its true probability of being right) 1,000 times at each sample size.

At 100 questions a perfect model averages 9.5 points with 10 bins, 12.1 with 15 equal-mass bins and 15.8 with 30 bins. At 1,000 questions the 10-bin figure is 3.0 points, and one run in twenty exceeds 4.3. That means the temperature-scaled model's 4.0 points on 2,000 questions is not far from the floor: the true probabilities themselves score 1.7 points on the same 2,000 questions. Any ECE you report should come with the value a perfectly calibrated model would get at your sample size and binning, or with a debiased estimator [3][10].

2. The number depends on the bins#

Equal-width bins put most answers into the top two or three bins when confidence is concentrated, so the summary is driven by a handful of bins. Equal-mass (adaptive) bins spread the answers evenly and are usually preferred [2], but they do not remove the bias, as Figure 2 shows. Pick one scheme before you look at results and report it.

3. A useless model can win#

Give every answer the same confidence, equal to the accuracy measured on the calibration split (63.9%). This predictor carries no information about which answers are right. Its ECE on the evaluation set is 3.0 points, lower than the temperature-scaled model's 4.0. ECE measures one property, agreement between stated and realised frequencies, and is blind to whether confidence separates right from wrong answers.

4. Rounded scores make bins meaningless#

Verbalised confidence takes about ten distinct values in practice. With equal-width bins of width 0.05 or less, each bin is a single stated value; with wider bins, distinct values are merged. In our data the verbalised ECE is 18.6 points under every binning we tried, which is a hint that the binning is doing nothing at all. For discrete scores, compute calibration per stated value directly.

Use a proper scoring rule alongside ECE#

The Brier score [4] is the mean squared difference between confidence and outcome. Unlike ECE it is a proper scoring rule: it is minimised in expectation by reporting the true probability, so it cannot be gamed by a constant. Murphy's decomposition [5] splits it into the parts you care about:

$c_i$ is the confidence for answer $i$ and $y_i$ is 1 if it was correct, $\bar{y}$ is overall accuracy, and the bin quantities are as in ECE. Reliability is a squared calibration gap (smaller is better). Resolution measures how much accuracy varies across confidence levels, which is the discrimination ECE ignores (larger is better). Uncertainty depends only on the task. The decomposition is exact when confidence is constant within bins and approximate otherwise.

Read the table by rows. The constant predictor's ECE looks good and its Brier score is the worst of the five apart from the verbalised score, because its resolution is zero. Temperature scaling cuts the reliability term by a factor of five and leaves AUROC unchanged at 0.819, because dividing log-odds by a positive constant cannot reorder answers. In this simulation the verbalised score is worse on both axes: worse calibrated and less discriminative. Published results on this are mixed. One broad study found verbalised confidence from preference-tuned models better calibrated than their token probabilities, often by a relative 50% in ECE [7]; another found verbalised scores overconfident and weaker than white-box signals at predicting failures [6]. The honest summary is that you have to measure both on your own task. Our simulation shows what the numbers look like when the verbalised score is the weaker one.

Where LLM confidence comes from#

Token probabilities. For multiple-choice or short answers, the probability of the chosen option is available from the logits. Large pretrained models are well calibrated on multiple-choice questions in the right format [8], and post-training tends to degrade that: the GPT-4 technical report shows a well-calibrated pretrained model becoming noticeably less calibrated after post-training [9]. For long free-form answers, sequence probability is a poor proxy, because it mixes "is the claim right" with "is this the likeliest phrasing".

Self-evaluation. Ask the model, after it answers, whether the answer is true, and read the probability of "True" from the logits. This P(True) approach was shown to be reasonably calibrated and to improve when the model sees several of its own samples first [8].

Verbalised confidence. Ask for a number. It works with any API, and models can be trained to produce calibrated verbal probabilities [11], but unprompted the numbers cluster at 80% to 95% and on multiples of five [6].

Consistency. Sample several answers and use the agreement rate as confidence. It is expensive, model-agnostic, and often among the stronger black-box signals [6].

Recalibration. Whatever the source, a monotone map fitted on held-out data (temperature scaling for log-odds [1], histogram binning or isotonic regression for any score) fixes the reliability term. It cannot add resolution. If two sources have similar AUROC, prefer the cheaper one and recalibrate it.

Selective prediction: turning confidence into abstention#

In regulated workflows you rarely need calibrated probabilities for their own sake. You need a rule: answer automatically when confidence is at least $\tau$, otherwise abstain and route to a person. Two quantities describe the rule [12]:

Coverage is the fraction of questions the system answers; selective risk is the error rate among those it answers. Lowering $\tau$ answers more questions and admits more errors. Sweeping $\tau$ traces the risk-coverage curve, and the area under it (AURC) summarises the whole trade-off in one number where lower is better.

Two practical points follow. First, the risk-coverage curve depends only on the ranking, so recalibration does not change what is achievable; it changes which number on the dial corresponds to which operating point. At $\tau =0.8$ the raw score answers 62.2% of questions at 16.4% risk, the temperature-scaled score answers 37.9% at 8.3% risk, and the verbalised score answers 77.5% at 25.0% risk. A threshold copied from one system, or from before a recalibration, means nothing. Second, the verbalised score's ties at the top make it impossible to select a small, safe subset: there is no threshold that separates the 286 "100% confident" answers from each other.

Choosing a threshold you can defend#

Suppose the requirement is "selective risk at most 10%". The naive procedure picks the lowest threshold whose empirical risk on a calibration set is at most 10%. That threshold is optimistic by construction: it stops exactly where noise happened to make risk look acceptable. We checked against a 400,000-question reference sample from the same simulated distribution, where the best achievable coverage at 10% risk with the temperature-scaled score is 37.0%. (On our 2,000-question evaluation split the same curve crosses 10% at 42.5% coverage, which is itself a reminder of how noisy these curves are.)

The alternative is to require an upper confidence bound on the risk to be below the target. We used a one-sided Clopper-Pearson bound at level $\delta =0.05$ and fixed-sequence testing: start from the most confident answers (the top 100, or the top tenth of the calibration set if larger), step down in pre-set increments, and stop at the first threshold whose bound exceeds 10%. Testing in a fixed order controls the error probability without a multiple-testing penalty, which is the idea behind the Learn-then-Test framework [13]; selective-guaranteed-risk procedures for classifiers work similarly [12].

The naive threshold breaks the 10% requirement in roughly half of all runs regardless of calibration-set size, because it is centred on the boundary. The bounded threshold breaks it in at most 2% of runs, as designed, but pays in coverage: with 250 labelled questions it almost never certifies anything, with 1,000 it answers 16.2% of traffic, and with 4,000 it answers 29.6% against the 37.0% achievable. The cost of a guarantee is labelled data. That trade-off belongs in the project plan, not in a footnote.

Code: the metrics in a few lines#

import numpy as np
from scipy.stats import beta

def ece(conf, correct, bins=10, equal_mass=False):
    conf, correct = np.asarray(conf, float), np.asarray(correct, float)
    if equal_mass:
        groups = np.array_split(np.argsort(conf, kind="stable"), bins)
    else:
        b = np.minimum((conf * bins).astype(int), bins - 1)
        groups = [np.where(b == k)[0] for k in range(bins)]
    return sum(len(g) / len(conf) * abs(conf[g].mean() - correct[g].mean())
               for g in groups if len(g))

def brier(conf, correct):
    return float(np.mean((np.asarray(conf) - np.asarray(correct)) ** 2))

def certified_threshold(conf, correct, target=0.10, delta=0.05):
    # fixed-sequence testing with one-sided Clopper-Pearson upper bounds
    o = np.argsort(-np.asarray(conf), kind="stable")
    c, err = np.asarray(conf)[o], np.cumsum(1 - np.asarray(correct)[o])
    n, tau = len(c), None
    for m in range(max(100, n // 10), n + 1, max(10, n // 40)):
        k = int(err[m - 1])
        ucb = 1.0 if k >= m else beta.ppf(1 - delta, k + 1, m - k)
        if ucb > target:
            break
        tau = c[m - 1]
    return tau  # None means: abstain on everything

What to report, and when#

Failure modes#

Calibrating on the wrong distribution. Temperature and thresholds are properties of a model on a distribution. A calibration set of easy benchmark questions will produce a threshold that is too permissive on hard production traffic. Sample the calibration set from production, stratified by the segments you report on.

Mixing answer extraction with confidence. If the answer parser fails, the confidence attached to the parsed answer is meaningless. Track parse failures separately and abstain on them.

Treating abstention as free. Every abstention costs a human review. Report coverage next to risk, and cost the review queue explicitly; a 0.5% risk at 5% coverage is rarely a useful system.

Re-using the evaluation set to choose the threshold. The risk you then report is the optimistic, naive one. Keep the threshold-selection split separate from the split you report on.

Group-level miscalibration. A model can be calibrated overall and badly miscalibrated on a subgroup (a language, a product line, a patient population). Compute the reliability diagram per segment that matters to the decision.

What this simulation does not capture#

The dataset is synthetic: correctness probabilities follow a logistic model, the raw confidence is a stretched and noisy version of the truth, and the verbalised score is a noisier, rounded transform. Real models have richer failure structure, including confidently wrong clusters that no monotone recalibration can fix. Questions are independent; clustered questions would widen every interval here, as clustered standard errors do for accuracy. The certified-threshold procedure assumes the calibration and deployment data are exchangeable. The formulas, the biases of ECE and the cost of guarantees carry over.

Frequently asked questions#

Sources#

1. Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. On Calibration of Modern Neural Networks, 2017. arxiv.org
2. Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, et al.. Measuring Calibration in Deep Learning, 2019. arxiv.org
3. Ananya Kumar, Percy Liang, Tengyu Ma. Verified Uncertainty Calibration, 2019. arxiv.org
4. Glenn W. Brier. Verification of forecasts expressed in terms of probability, 1950. doi.org
5. Allan H. Murphy. A new vector partition of the probability score, 1973. doi.org
6. Miao Xiong, Zhiyuan Hu, Xinyang Lu, et al.. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, 2023. arxiv.org
7. Katherine Tian, Eric Mitchell, Allan Zhou, et al.. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback, 2023. arxiv.org
8. Saurav Kadavath, Tom Conerly, Amanda Askell, et al.. Language Models (Mostly) Know What They Know, 2022. arxiv.org
9. OpenAI (Josh Achiam, Steven Adler, et al.). GPT-4 Technical Report, 2023. arxiv.org
10. Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, Michael C. Mozer. Mitigating Bias in Calibration Error Estimation, 2020. arxiv.org
11. Stephanie Lin, Jacob Hilton, Owain Evans. Teaching Models to Express Their Uncertainty in Words, 2022. arxiv.org
12. Yonatan Geifman, Ran El-Yaniv. Selective Classification for Deep Neural Networks, 2017. arxiv.org
13. Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, Lihua Lei. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control, 2021. arxiv.org