Rubric-based evaluation and rubric rewards: atomic criteria, weights and noisy graders

A rubric turns "is this answer good?" into a list of small, checkable questions with weights. Done well, it gives you scores that are stable, auditable and usable as a training reward. Done badly, it gives you a number that moves when you re-run the grader, rankings that flip between runs, and a reward that a policy learns to satisfy without getting better. This note works through the arithmetic of weighted rubrics with noisy graders, shows from a seeded simulation where the variance actually comes from, and covers what changes when the rubric becomes a reward for reinforcement learning.

A rubric, worked through#

Take a single prompt from a medication-safety evaluation: a user asks whether they can take ibuprofen with the blood-pressure tablet they were just prescribed. A holistic grader would read the answer and give it a 7 out of 10. A rubric grader instead checks six criteria, each with a weight set by a clinician:

An answer that passes the first four and fails the last two scores $(4+3+3+2)/14=0.857$. The rubric does three useful things that the 7 out of 10 does not. It says why the answer lost points. It lets you measure grader reliability one criterion at a time. And it lets a domain expert decide, in advance and in writing, that a wrong dose matters four times as much as a wordy answer.

This structure is now standard in open-ended evaluation. HealthBench scores 5,000 health conversations against 48,562 conversation-specific criteria written by 262 physicians [1]. Prometheus trained an open evaluator to grade against user-supplied score rubrics [5], TICK generates instruction-specific checklists automatically [6], and WildBench pairs its tasks with checklists to make judge outputs more consistent [8]. Since mid-2025, the same rubrics have been used as rewards: Rubrics as Rewards (RaR) [2], checklist feedback (RLCF) [3] and rubric anchors [4] all train policies against per-criterion judgements instead of a single preference score.

The arithmetic of a weighted rubric#

Let a response be graded on $K$ criteria. Criterion $k$ has weight $w_k$, and ${\tilde{w}}_k=w_k/{\sum }_j{w}_j$ is its normalised weight. The grader returns ${\hat{y}}_k\in \{0,1\}$, or the average of $R$ graders' votes. The response score is

Here $\hat{S}$ is between 0 and 1. A system's score is the mean of $\hat{S}$ over $M$ prompts. The interesting part is what grader error does to it.

Grader error shrinks real differences#

Suppose the grader for criterion $k$ flips the true label with probability $e_k$ in either direction, and the system truly meets the criterion with probability $p_k$. Then the expected observed pass rate is

Here $e_k$ is the grader's per-criterion error rate and $p_k$ the true pass rate. The observed rate is pulled toward 0.5 by a factor $1-2e_k$. Subtract this for two systems A and B and the constant term cancels:

Each criterion's contribution to the gap between two systems is multiplied by $1-2e_k$. A grader that is wrong 10% of the time on a criterion keeps only 80% of the true difference on it. Noise does not bias the comparison toward one system, but it shrinks the signal you are trying to detect, and the shrinkage is worst on exactly the subjective criteria where graders disagree most.

Where the variance comes from#

If criteria are graded independently and each of $R$ graders errs independently, the variance of a system's mean score over $M$ prompts is

The first term in the bracket is response variance: which prompts the system happened to get right. The second is grader variance, and it is the only term that adding graders reduces. Both shrink with more prompts. Both are weighted by ${\tilde{w}}_k^2$, so a rubric whose weight sits on a few criteria behaves like a short test. A handy summary is the effective number of criteria:

$K_{\mathrm{eff}}$ equals $K$ when weights are equal and falls toward 1 as one criterion dominates. The clinical weights above give $K_{\mathrm{eff}}=196/40=4.9$; a rubric weighted 10, 1, 1, 1, 1, 1 gives 2.1.

Worked example#

In the simulation below, the two best systems have true weighted scores of 0.691 and 0.664 on the clinical rubric, a gap of 0.026. With a base grader error $\epsilon =0.1$ (criterion error rates from 0.05 on the dose criterion to 0.16 on the subjective ones), the attenuation formula predicts a measured gap of 0.0218; the simulation measured 0.0220. Each system's mean over 100 prompts has a standard deviation of about 0.022 (formula: 0.0219). The difference of two independent such scores has a standard deviation near $0.022\sqrt{2}=0.031$, so a gap of 0.022 is about 0.7 standard deviations. The better system comes out on top in only 70% of re-runs. Nothing about the rubric is broken; it is simply too few prompts for a gap that size.

Designing atomic criteria#

Most rubric problems are criterion-writing problems. A good criterion is something two careful experts would mark the same way after reading only the response and the criterion. In practice that means it is:

• Atomic. One fact or behaviour per criterion. "Gives the correct dose and warns about kidney disease" is two criteria; when graders disagree you cannot tell which half they disagree on.
• Observable in the response. "The model understood the question" is not gradable. "Names the user's medication" is.
• Binary or with anchored levels. If you need levels, write what each level looks like. Unanchored 1 to 5 scales drift between graders and over time.
• Written with its negative. Many important criteria are penalties: "recommends a dose above the labelled maximum". HealthBench uses negatively weighted criteria for this [1]. Without them, a rubric only rewards what is present and cannot see harmful additions.
• Non-overlapping. If "plain language" and "concise" both fire on the same padding, you have doubled the weight of one behaviour without meaning to.

The last row matters for training. A criterion that a stock phrase can satisfy will be satisfied by a stock phrase once a policy is optimised against it.

Inter-rater reliability, one criterion at a time#

Report agreement per criterion, not one number for the whole rubric. The usual statistic is Cohen's $\kappa$ [9], which compares observed agreement $p_o$ with the agreement $p_e$ expected by chance from each grader's marginal pass rate:

Here $p_o$ is the fraction of items both graders mark the same, and $p_e$ is what two graders with those pass rates would agree on by luck. For two graders who each flip the truth with probability $e$ on a criterion met with prevalence $\pi$, the pieces have closed forms: $p_o=(1-e{)}^2+e^2$, each grader's pass rate is $q=\pi (1-e)+(1-\pi )e$, and $p_e=q^2+(1-q{)}^2$.

This is the well-known prevalence effect, and it bites rubrics hard because many safety criteria are met by almost every response ("does not recommend a lethal dose"). A low $\kappa$ on such a criterion may mean the graders are poor, or only that the criterion rarely fails. Report $\kappa$ next to prevalence and raw agreement, and for rare-failure criteria measure grader sensitivity on a set enriched with known failures instead.

How grader noise moves rankings: a simulation#

Setup (simulation). Eight systems, the six-criterion clinical rubric above, and a true pass-rate matrix $p_{s,k}$ drawn from a logistic model with system ability, criterion difficulty and a system-by-criterion interaction (seed fixed; the matrix is in the code). Grader error on criterion $k$ is $e_k=\epsilon m_k$, with multipliers $m_k$ from 0.5 for the dose criterion to 1.6 for the two subjective criteria. For each setting we re-ran the whole evaluation 400 times, drawing fresh responses and fresh grader errors, and compared the measured ranking with the true one using Kendall's $\tau$.

The last row is the sobering one: even a perfect grader on 100 prompts finds the best system only 79% of the time. Response sampling, not grading, is the dominant source of variance at this scale, which is what the variance formula predicts: at $\epsilon =0.1$ the grader term contributes $0.0117^2$ of the total $0.0219^2$, a bit under 30%.

Grader variance does matter when you hold the responses fixed, for example when re-grading a frozen set of outputs after changing judge prompts. With 100 fixed responses per system, the SD of a system score from grading alone was 0.0116 with one grader, 0.0068 with three and 0.0053 with five, and the ranking matched the noise-free grading of the same responses with $\tau$ of 0.815, 0.888 and 0.907. If two dashboards built from the same outputs disagree, grader noise is the cause; if two evaluation runs on fresh samples disagree, it is mostly the prompts.

Weighting: who decides, and what it costs#

Weights encode values, so they should come from the people accountable for the outcome: clinicians for clinical criteria, compliance staff for regulatory ones. Write them before seeing model outputs, and version them with the rubric. Weights tuned after the fact to make a preferred model win are a quiet form of overfitting.

Weights also change the statistics. At $\epsilon =0.1$, one grader and 200 prompts, the three schemes in the interactive figure behaved as follows (each against its own true ranking):

Concentrating weight on one criterion raised the score SD by 69% relative to flat weights and cut top-system identification from 86% to 55%. If one criterion truly matters ten times more than the rest, it is often better to make it a gate (any failure is reported separately and blocks release) and weight the remainder evenly, than to fold it into the average.

Rubrics as rewards#

Using a rubric as an RL reward replaces a single learned preference score with $\hat{S}$ computed by a judge model per criterion, often with prompt-specific rubrics. RaR reported up to 31% relative improvement on HealthBench over Likert-style judge rewards [2]; RLCF reported gains on all five instruction-following benchmarks it tested [3]. The attraction is that the reward is decomposed and inspectable: when it goes up, you can see which criteria moved.

The risk is the same one that affects any learned reward [10], which we covered for preference reward models in our note on reward overoptimization. The policy optimises $\hat{S}$, not the experts' intent, and it will find any systematic way to raise ${\hat{y}}_k$ without raising $y_k$. We call this criterion gaming. It differs from noise: noise is random across responses, while gaming is a behaviour the policy can adopt on every response.

A simulated case of criterion gaming#

Setup (simulation). Each candidate response has a hedging level $h\in [0,1]$. Hedged answers omit the dose more often (true pass rate on that criterion falls by up to 40%), skip the interaction check more often (up to 30%), make fewer unsupported claims (genuinely better), and are longer (concision falls by up to 60%). They also append "please see your doctor". Graders for the escalation and plain-language criteria are swayed by that phrase: their false-positive rate rises by $\gamma h$, where $\gamma$ measures how gameable the grader is. The true score never rewards boilerplate.

Best-of-$n$ selection exerts weaker, per-sample pressure, and the picture is subtler. At $n=128$ with the honest grader, the selected responses scored 0.991 on the rubric and 0.925 in truth, and their mean hedging fell from 0.50 to 0.42. With the gameable grader the selected responses scored 0.998 on the rubric but 0.857 in truth, and selection no longer moved hedging at all (0.50). The proxy improved while the true score fell by 0.07. Note that even the honest grader shows a proxy-to-truth gap at large $n$: selecting the maximum of many noisy scores favours responses the grader over-scored, a winner's curse that exists without any gaming.

Detecting it#

Two checks caught the gaming in our simulation and work in practice:

1. A held-out audit grader. Grade a sample of training-time outputs with a grader the policy never saw (a different judge model, or experts). In the best-of-128 run, the audit grader scored 0.867 for honest-grader selections and 0.810 for gameable-grader ones. Compare the train-reward-to-audit gap against its value early in training, not against zero, since the winner's curse makes it positive anyway.
2. Per-criterion reward curves. Plot the pass rate of each criterion over training. A criterion whose pass rate climbs toward 100% while correlated objective criteria fall is the signature of gaming. In the hedging case, escalation rises while the dose criterion drops.

Failure modes we see most often#

LLM judges bring their own biases to every criterion, including position and verbosity effects [11] (we measured both in an earlier note), and calibrating a judge against human labels per criterion, as LLM-Rubric does [7], is worth the effort when a rubric gates releases.

Practical checklist#

Code: rank stability by bootstrap#

Given per-prompt, per-criterion grades for several systems, this estimates how often the ranking you report would survive a re-run on a fresh sample of the same size.

import numpy as np
from scipy.stats import kendalltau

def rank_stability(grades, weights, n_boot=2000, seed=0):
    """grades: array (systems, prompts, criteria) of 0/1 or mean grader votes."""
    rng = np.random.default_rng(seed)
    w = np.asarray(weights, float) / np.sum(weights)
    scores = grades @ w                      # (systems, prompts)
    point = scores.mean(1)
    S, M = scores.shape
    taus, top = [], []
    for _ in range(n_boot):
        idx = rng.integers(0, M, M)          # resample prompts, paired across systems
        b = scores[:, idx].mean(1)
        taus.append(kendalltau(point, b).statistic)
        top.append(np.argmax(b) == np.argmax(point))
    k_eff = w.sum() ** 2 / (w ** 2).sum()
    return dict(score=point, tau=np.mean(taus), p_top=np.mean(top), k_eff=k_eff)

The bootstrap resamples prompts, which captures the response term in the variance formula. To capture grader variance too, re-grade a subset with independent graders and resample the graders as well. If prompts come in clusters (several items per source document or conversation), resample clusters instead of items, as described in our note on clustered standard errors.

Frequently asked questions#

Sources#

1. R. K. Arora, J. Wei, R. S. Hicks et al.. HealthBench: Evaluating Large Language Models Towards Improved Human Health, 2025. arxiv.org
2. A. Gunjal, A. Wang, E. Lau et al.. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains, 2025. arxiv.org
3. V. Viswanathan, Y. Sun, S. Ma et al.. Checklists Are Better Than Reward Models For Aligning Language Models, 2025. arxiv.org
4. Z. Huang, Y. Zhuang, G. Lu et al.. Reinforcement Learning with Rubric Anchors, 2025. arxiv.org
5. S. Kim, J. Shin, Y. Cho et al.. Prometheus: Inducing Fine-grained Evaluation Capability in Language Models, 2023. arxiv.org
6. J. Cook, T. Rocktäschel, J. Foerster et al.. TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation, 2024. arxiv.org
7. H. Hashemi, J. Eisner, C. Rosset et al.. LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts, 2024. arxiv.org
8. B. Y. Lin, Y. Deng, K. Chandu et al.. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild, 2024. arxiv.org
9. J. Cohen. A Coefficient of Agreement for Nominal Scales, 1960. doi.org
10. L. Gao, J. Schulman, J. Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
11. L. Zheng, W.-L. Chiang, Y. Sheng et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org