Expert annotators cost several times more per label than generalists, so the question is never "are experts better?" but "where does an expert label change the outcome?". This note separates the two kinds of label error, random and systematic, shows why redundancy fixes the first and not the second, measures how expert adjudication and targeted routing perform on a cost-quality frontier, and traces the effect through to reward-model accuracy and evaluation rankings. All numbers come from a seeded simulation, with costs in relative units (a generalist label costs 1).
A worked example: a medication question three generalists agree on#
A preference dataset for a health assistant contains this pair. The prompt asks whether a common over-the-counter painkiller is safe to take alongside a prescribed anticoagulant. Response A is warm, fluent and reassuring, and recommends a typical dose. Response B is shorter, notes that the combination raises bleeding risk, and suggests checking with a pharmacist or prescriber. Three generalist annotators all prefer A: it reads better and sounds confident. A clinical pharmacist prefers B without hesitation.
This is the error that matters. It is not noise. Asking more generalists does not fix it, because all of them are applying the same surface cues (fluency, confidence, length) to a question whose answer depends on knowledge they do not have. Majority vote turns three individual mistakes into one confident wrong label. A reward model trained on thousands of such pairs learns to prefer reassuring fluency in exactly the domain where it is most dangerous.
Expert labels exist to fix this case. The engineering question is how to find these items without paying expert rates for the 75% of a dataset where a generalist does fine.
Two kinds of label error#
Write the label an annotator gives to item as the true label flipped with some probability. The classic model of Dawid and Skene [1] gives each annotator a confusion matrix and treats annotators as independent given the true label. Under that model, aggregating annotators with individual accuracy by majority vote gives an accuracy of
which rises towards 1 as grows. This is the logic of "get another label" [2] and of early crowdsourcing results, where a handful of non-expert labels matched an expert on several language tasks [3]. It is correct when errors are independent.
Specialist items break independence. If generalists share a misconception, the right model is a mixture: with probability the item triggers the shared wrong cue, and the annotators' errors are driven by it together. Then even infinitely many generalists converge to the wrong answer on that item, and the best majority vote can do is
where the shared cue is whatever surface feature generalists use in place of domain knowledge. In words: redundancy averages out noise, but it cannot average out a bias everyone has.
Simulation setup. Preference pairs are described by a 12-dimensional feature vector. On easy items (75%), the true preference follows one direction ; on specialist items (25%), it follows a different direction . Generalists judge easy items by with logistic noise (scale 0.5) and specialist items by a surface direction with cosine similarity 0.6 to , so the surface cue points the wrong way on 30% of specialist items. Experts judge every item by the true direction with less noise (scale 0.25). Expert labels cost 5, generalist labels 1. These are illustrative settings, not measurements of any annotator pool.
What each annotation strategy buys#
The adjudication result is the one to remember. Sending disagreements to an expert is a popular design: it spends expert time only where the crowd is unsure. In the simulation, generalists disagreed on 19.2% of easy items and 19.5% of specialist items: almost the same rate. Disagreement detects noise, and specialist errors are not noise. On specialist items, all three generalists were unanimously wrong 21% of the time, against 0.9% on easy items. Adjudication-by-disagreement never sees those items.
| Strategy | Relative cost per item | Error, easy items | Error, specialist items | Error, all |
|---|---|---|---|---|
| Generalist, one label | 1.0 | 8.8% | 30.4% | 14.2% |
| Three generalists, majority | 3.0 | 5.8% | 29.7% | 11.8% |
| Three generalists + expert on disagreement | 4.0 | 4.0% | 22.0% | 8.5% |
| Expert, one label | 5.0 | 4.6% | 4.5% | 4.6% |
Real evidence points the same way. GPQA was built so that questions are hard for skilled non-experts with web access: experts in the relevant field reached 65% accuracy, while highly skilled non-expert validators reached 34% despite spending on average over 30 minutes per question with unrestricted web access [4]. When a task depends on knowledge the annotator lacks, effort and search do not close the gap.
From label accuracy to reward-model accuracy#
Label accuracy is an intermediate quantity. What matters is the model trained on the labels. We trained a logistic reward model (a Bradley-Terry model on the pair's feature difference, with separate weights allowed for specialist items) on 6,000 labelled pairs and measured its accuracy against true preferences on 40,000 held-out pairs, averaging four seeds. The share of items given expert labels ran from 0 to 100%, allocated three ways: at random; to the items a pilot reward model trained on generalist labels was least sure about (uncertainty sampling); and to items routed as specialist by domain metadata.
Specialist-slice errors matter more than their share suggests, because a policy optimised against a reward model pushes hardest on exactly the directions the reward model gets wrong (see reward overoptimization). Three findings.
Random label noise is almost free; systematic noise is not. On easy items the reward model reached 98.6% accuracy from generalist labels with 8.8% error. Logistic regression on thousands of examples averages independent flips away. On specialist items it reached only 68.9% from generalist labels, because the flips were systematic and the model learned the surface direction faithfully.
Uncertainty sampling inherits the generalists' blind spot. A pilot model trained on generalist labels is confident on specialist items, because it learned the generalists' consistent (wrong) rule. Its low-margin items are the noisy easy ones. Uncertainty sampling therefore improved training-label accuracy faster than random allocation (90.2% against 88.0% at a 25% expert share) while doing nothing extra for the specialist slice. Active selection is only as good as the model doing the selecting. Settles' survey of active learning covers the general sampling-bias problem [5].
Routing is where the value is. Sending the 25% of items that need expertise to experts, and the rest to generalists, matched all-expert quality at 40% of the cost. The catch is that routing requires knowing which items need expertise. In practice that means domain metadata, a cheap triage classifier, or asking generalists to flag items they cannot judge, each with its own error rate that this simulation does not model.
What imperfect triage costs#
We re-ran the routed policy with a triage step that misses a share of specialist items, sending them to generalists. With perfect triage the reward model reached 96.3% on specialist items in this rerun (it uses a fresh test set, hence the small difference from Figure 2) at a relative cost of 2.0 per item. Missing 20% of specialist items dropped that to 91.5% (cost 1.8); missing 40% dropped it to 86.1% (cost 1.6). Specialist-item accuracy fell by about 2.5 points for every 10 points of triage recall lost, so triage recall is worth measuring directly: have experts review a random sample of items the triage step sent to generalists, and estimate how many needed them.
A useful summary is accuracy gained per unit of extra cost. Moving from all-generalist labels (cost 1.0) to routed experts (cost 2.0) added 27 points of specialist-item accuracy for one unit of cost. Spending the same unit on randomly placed expert labels added 6 points, and spending two units on extra generalist labels for majority vote lost a point. The marginal value of an expert label is not a property of the expert. It is a property of the item it is spent on.
Expertise in evaluation data#
Training data errors are partly averaged away. Evaluation data errors are not: they go straight into the number you report. With independent label errors at rate on a binary-graded test set, a model with true accuracy has measured accuracy
where is the accuracy against the noisy labels and the subscripts index two models. Independent errors compress differences between models by a factor , so detecting the same true difference takes about times as many items: 1.56 times at . Northcutt and colleagues found label errors in test sets of widely used benchmarks and showed that they could change which model ranks first [6].
Correlated errors are worse than compression. We simulated two models on 400 specialist items. Model A is right 85% of the time with errors unrelated to the generalist cue. Model B is right 80% of the time, and its errors are exactly the generalists' errors: it has learned the same surface rule, as a model trained on generalist preferences would. Against generalist labels, B scored 84.1% and A 63.7%: B ranked first in 100% of 4,000 simulated test sets. Against expert labels, A scored 81.8% and B 78.5%, and the ranking was wrong only 11% of the time, from sampling noise. A test set labelled by the same kind of annotator as the training data rewards models for sharing the annotators' mistakes.
Practice: spending an expert budget#
| Decision | Guidance |
|---|---|
| Where do expert labels go first? | An expert gold set for evaluation and auditing, then training items that need domain knowledge. |
| How do you find items that need expertise? | Domain metadata and triage, not crowd disagreement or pilot-model uncertainty, which miss shared misconceptions. |
| When is redundancy worth paying for? | On easy items where errors are independent and the labels feed evaluation directly; less so for large training sets. |
| How do you know the generalist pool has a blind spot? | Compare generalist and expert labels on a random audit sample stratified by domain; look for slices with high agreement among generalists and low agreement with experts. |
| How should expert time be used per item? | Rationales and rubric items, not just a preference bit: they let generalists and models handle similar items later. |
| How do you report data quality? | Error rate by slice against expert gold, with intervals, and the annotation design for each slice. |
The audit that detects a blind spot is a few lines:
import numpy as np
import pandas as pd
def blind_spot_report(df):
"""df: one row per audited item, columns slice, gen_labels (list of 0/1), expert_label (0/1)."""
rows = []
for s, g in df.groupby("slice"):
maj = g.gen_labels.apply(lambda v: int(np.mean(v) > 0.5))
unanimous = g.gen_labels.apply(lambda v: len(set(v)) == 1)
rows.append({"slice": s, "n": len(g),
"gen_vs_expert_error": (maj != g.expert_label).mean(),
"gen_disagreement": 1 - unanimous.mean(),
"unanimous_but_wrong": (unanimous & (maj != g.expert_label)).mean()})
# high unanimous_but_wrong with low gen_disagreement = systematic error: route this slice to experts
return pd.DataFrame(rows).sort_values("unanimous_but_wrong", ascending=False)For teams building systems that fall under the EU AI Act's high-risk category, Article 10 requires that training, validation and testing data sets be "relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose" [9]. Slice-level error rates against an expert gold set are one concrete way to document what "to the best extent possible" meant for a given dataset.
Failure modes#
Experts disagree too. In medicine, law and finance, expert disagreement is often real disagreement about practice, not error. Use two experts and an adjudication protocol on gold items, and record disagreements rather than forcing a single label where the domain has none.
The wrong expert. A cardiologist is a generalist on a tax question. Expertise is specific; route by sub-domain and verify credentials per domain.
Expert fatigue and drift. Expert error rates rise over long sessions, and expert time is often fragmented. Seed expert queues with known-answer items and track accuracy over time, as for any annotator.
Model-generated labels as a cheap substitute. LLM annotators can match or beat crowd workers on some text-annotation tasks [10], but on specialist items they tend to share the surface-cue failure, which is exactly the error expert labels exist to catch. Treat model labels as another generalist tier, and audit them the same way.
Routing errors. Items misrouted to generalists carry systematic errors with no flag. Audit a random sample of generalist-routed items with experts to estimate the miss rate.
Rationales that leak into evaluation. Expert rationales written for training items are valuable, but if the same experts write the gold set, their phrasing and their favourite examples end up on both sides. A model can then score well by echoing the experts' style rather than their judgement. Keep gold-set authors and training-data authors separate where the expert pool allows, or at least separate the topics they write about.
Ignoring the evaluation side. Spending the whole budget on training labels and evaluating against generalist labels hides the problem the training budget was meant to fix. The gold set comes first.
Scalable oversight research asks the long-run version of this question: how humans can supervise models on tasks where the humans are the non-experts [11]. For today's datasets the practical version is simpler: know which items need expertise, send them to experts, and keep an expert-labelled yardstick to check that you did.
Frequently asked questions#
Are expert annotators worth the extra cost?
Where the label depends on knowledge generalists lack, yes, and often only there. In our simulation, generalist labels were wrong on 30% of specialist items but only 9% of easy items, and the reward model averaged away the easy-item noise. Spending expert labels on specialist items gave nearly all of the benefit.
Does majority vote of several annotators replace an expert?
Only for independent errors. When generalists share a misconception, adding more of them converges on the wrong answer. Three-way majority vote cut easy-item error from 8.8% to 5.8% in our simulation but left specialist-item error at about 30%.
Should experts adjudicate annotator disagreements?
It helps with noise but misses systematic errors, because generalists usually agree when they share a misconception. In our simulation they disagreed equally often on easy and specialist items, and were unanimously wrong on 21% of specialist items.
Does active learning find the items that need experts?
Not if the selecting model was trained on generalist labels. It learns the generalists' consistent rule and is confident exactly where they are wrong. In our simulation uncertainty sampling did no better than random on specialist items.
Why do evaluation sets need expert labels?
Label errors compress measured differences between models and, when correlated with annotator biases, can reverse rankings. In our simulation a model that shared the generalists' errors ranked first on a generalist-labelled test set every time despite being 5 points less accurate.
Sources#
- A. P. Dawid, A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm, 1979. doi.org
- V. S. Sheng, F. Provost, P. G. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers, 2008. doi.org
- R. Snow, B. O'Connor, D. Jurafsky, A. Y. Ng. Cheap and Fast, But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks, 2008. aclanthology.org
- D. Rein, B. L. Hou, A. C. Stickland, J. Petty et al.. GPQA: A Graduate-Level Google-Proof Q&A Benchmark, 2023. arxiv.org
- B. Settles. Active Learning Literature Survey, 2009. burrsettles.com
- C. G. Northcutt, A. Athalye, J. Mueller. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, 2021. arxiv.org
- R. K. Arora, J. Wei, R. S. Hicks, P. Bowman et al.. HealthBench: Evaluating Large Language Models Towards Improved Human Health, 2025. arxiv.org
- K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi et al.. Large Language Models Encode Clinical Knowledge, 2022. arxiv.org
- European Parliament and Council of the European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 10, 2024. eur-lex.europa.eu
- F. Gilardi, M. Alizadeh, M. Kubli. ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks, 2023. arxiv.org
- S. R. Bowman, J. Hyun, E. Perez, E. Chen et al.. Measuring Progress on Scalable Oversight for Large Language Models, 2022. arxiv.org