Expected calibration error (ECE) is a summary of how well a model's confidence matches its accuracy. A perfectly calibrated model that says 80% is right 80% of the time; ECE is the average size of the gap between confidence and accuracy, weighted by how many predictions fall at each confidence level.
Formula
Sort predictions into bins by confidence (often 10 or 15 equal-width bins). For bin , let be the fraction correct and the mean stated confidence. Then
Each bin contributes its calibration gap, weighted by its share of the data. ECE is 0 for perfect calibration and is expressed in the same units as accuracy.
Worked example
Illustrative numbers: a model answers 1,000 questions. 200 answers have mean confidence 0.95 and are 90% correct; 500 have confidence 0.80 and are 78% correct; 300 have confidence 0.60 and are 45% correct. The gaps are 0.05, 0.02 and 0.15, so
Most of the error comes from the low-confidence bin, where the model is overconfident by 15 points. A reliability diagram (accuracy against confidence per bin) would show this at a glance.
Pitfalls
- ECE depends on the binning scheme. Equal-mass bins and a stated bin count make numbers comparable.
- It is biased upward on small samples, because sampling noise alone creates gaps.
- Low ECE does not imply good discrimination: always predicting the base rate is calibrated but useless. Pair it with accuracy, AUROC or a proper scoring rule such as the Brier score.
- For language models, define confidence carefully: token probability, a verbalised percentage, and agreement across samples give different answers.
Why it matters
Calibration is what makes abstention and escalation work. If a system routes low-confidence answers to a person (human in the loop) or refuses below a threshold, those thresholds only mean something when confidence is calibrated. Post-training such as RLHF is known to change calibration, so ECE belongs in the eval suite before and after every training run. In high-stakes domains, an overconfident wrong answer is usually worse than an uncertain one.