Position and verbosity bias in LLM judges: swap-and-average and length-controlled win rates

An LLM judge that compares two answers is not only measuring quality. It also reacts to which answer it reads first and to how long each answer is. This note builds a small pairwise judge model with a position term and a length term, shows in a seeded simulation how a slightly worse model can post a 63.6% win rate, and walks through the two standard repairs: swap-and-average for position, and length-controlled win rates by regression adjustment for verbosity. Both repairs work, both have a price, and one of them can quietly overcorrect.

A worse model that wins 63.6% of the time#

Start with a concrete case. Model A is a candidate fine-tune; model B is the incumbent. On a pool of instructions, A is in fact slightly worse: if you asked a careful, unbiased grader with the same level of noise as your judge, A would win 46.4% of comparisons. A has also learned to write longer answers, about 1.6 times the length of B's on a typical prompt. Your pipeline puts the candidate in the first slot of the judge prompt, because that is how the template was written.

In our simulation, that pipeline reports a 63.6% win rate for A. Running every comparison in both orders and averaging brings it down to 53.7%. Fitting a length-adjusted regression brings it to 46.7%, within half a point of the 46.4% target. The judge never changed. Only the estimator did.

None of these effects are hypothetical. Zheng et al. measured position, verbosity and self-enhancement biases in strong LLM judges when introducing MT-Bench [1]. Wang et al. showed that simply altering the order of responses let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluator [2]. Saito et al. found that GPT-4 prefers longer answers more than human labellers do [3], and Dubois et al. built length-controlled AlpacaEval because the uncorrected benchmark was easy to move by prompting a model to be more verbose [4]. The rest of this note puts numbers on how those biases propagate into a headline win rate, and on what each repair costs.

A judge model with two nuisance terms#

We use a Bradley-Terry style model [5], which treats each pairwise verdict as a logistic function of a quality gap, and add two terms that should not be there.

Here $\sigma (x)=1/(1+e^{-x})$ is the logistic function; $q$ is the latent quality of an answer on the log-odds scale; $b$ is the position bias, the extra log-odds the judge gives to whichever answer it reads first; $\ell$ is the log of the answer's length, so ${\ell }_{\text{first}}-{\ell }_{\text{second}}$ is the log length ratio; and $\lambda$ is the verbosity bias, the extra log-odds per unit of log length ratio. With $b=\lambda =0$ the formula reduces to an ordinary pairwise model with a noisy but fair judge.

The intuition: quality, slot and length all push on the same log-odds scale, so a judge cannot tell you which of the three moved its verdict. Writing $d_i=q_{A,i}-q_{B,i}$ for the true quality gap on prompt $i$ and $x_i={\ell }_{A,i}-{\ell }_{B,i}$ for the log length ratio, the probability that A wins when A is shown first or second is

In the second order, the position term flips sign because B now holds the favoured slot. The length term does not flip, because A is still the longer answer wherever it sits. That asymmetry is the whole story of this note: order randomisation can cancel $b$ but can never cancel $\lambda$.

The target quantity is the win rate a bias-free judge of the same noise level would report, averaged over prompts:

Here $n$ is the number of prompts. Note what this target does and does not claim. It is not the probability that A is "really" better on each prompt; it is the judge's own preference once the two nuisance terms are switched off. That is the most an adjustment can recover from the judge's verdicts alone.

Simulation setup#

All numbers below come from a seeded simulation, not from a real judge. We draw 2,000 prompts. The quality gap is $d_i\sim \mathcal{N}(-0.2,1)$, so A is slightly worse on average and much better or worse on some prompts. The log length ratio is $x_i\sim \mathcal{N}(0.47,0.5^2)$, a median length ratio of 1.58, and by default it is independent of quality. The default biases are $b=0.5$ and $\lambda =0.8$. Expected verdict probabilities are computed exactly for each prompt, and a separate Monte Carlo section uses sampled binary verdicts on 805 prompts, the size of the AlpacaEval instruction set [4].

Position bias and swap-and-average#

Position bias is the easiest bias to fix because you control the order. Run every comparison twice, once in each order, and average:

Each prompt now contributes the mean of the A-first and A-second verdicts, so a judge that favours the first slot helps A exactly as often as it helps B.

Zheng et al. describe a "conservative" version: call the judge in both orders, declare a win only when the same answer is preferred both times, and score inconsistent pairs as ties [1]. A one-line calculation shows this has the same expectation as plain averaging. With independent calls, A wins both orders with probability $p^{(1)}{p}^{(2)}$ and loses both with probability $(1-p^{(1)})(1-p^{(2)})$; scoring the rest as half a win gives

So the conservative rule and averaging are the same estimator in expectation. Choose between them for reporting reasons, not accuracy: the tie count is a useful diagnostic, but it is not a measure of position bias by itself. In the default setting, 43% of prompts produce inconsistent verdicts across orders, yet with $b=0$ the rate is still 40%, because a noisy judge disagrees with itself on close calls whatever the order. The cleaner diagnostic is the first-slot win rate: across both orders, how often does the answer in the first position win? Without position bias this is 50% by symmetry. In the default setting it is 59.9%.

Swapping cancels the direction, not the damage#

Figure 1 sweeps the position bias with no verbosity bias. With A always shown first, the measured win rate climbs from 46.4% to 82.3% at $b=2$; with A always second it falls to 13.6%. The swap-averaged estimate stays close to the truth, but not on it: at $b=2$ it reads 47.9%. The reason is that the logistic curve is flat in the tails. A large $b$ pushes both orders into regions where quality differences barely move the verdict, so every prompt drifts towards a coin flip and the swapped average is pulled towards 50%. A strongly position-biased judge is not just biased; it is less informative, and averaging cannot restore the information it threw away.

This matches what Shi et al. report from over 150,000 real judge calls: position bias varies across judges and tasks, and it is strongly affected by the quality gap between the two answers [6]. In our model that is a direct consequence of the logistic form. When $|d_i|$ is large the position term barely changes $\sigma$; when $d_i\approx 0$ it decides the verdict. Position bias is concentrated in close calls, which are exactly the comparisons that matter when two strong models are close.

Verbosity bias survives swapping#

Turn the verbosity bias back on. Swapping removes the $b$ term but leaves $\lambda x_i$ untouched, because A's answer is longer in both orders. In the default setting the swap-averaged win rate is 53.7% against a target of 46.4%, about 7.3 points too high, and the bias moves with A's length, not with A's quality.

That second point makes verbosity bias worse than position bias for model development. Position bias is a fixed tax that can be cancelled by design. Verbosity bias is a gradient: any training process that is selected or rewarded by the judge will learn to write longer. Singhal et al. found that in several RLHF settings, improvements in reward were largely driven by increasing response length, and that a purely length-based reward reproduced most of the downstream improvement over the supervised baseline [7]. A judge with a nonzero $\lambda$ produces leaderboards that drift towards verbosity whenever it is also used to pick checkpoints.

Dubois et al. measured this directly on AlpacaEval by prompting the same models to be concise or verbose. The baseline model's uncorrected win rate ranged from 22.9% to 64.3% depending only on the verbosity instruction, while the length-controlled version ranged from 41.9% to 51.6% [4].

Length-controlled win rates by regression adjustment#

You cannot randomise length the way you randomise order, because length is a property of the model being evaluated. The fix is statistical: model the judge's verdicts as a function of the things you want to remove, then predict what it would have said if those things were neutral. Length-controlled AlpacaEval fits a logistic regression with a model term, a length term and an instruction-difficulty term, then sets the length term to zero [4]. A single-comparison version, with a position feature added, is

Here $y_{ij}$ is 1 when the judge prefers A on prompt $i$ in order $j$; $s_{ij}$ is $+1$ when A is shown first and $-1$ when it is shown second; $x_i$ is the log length ratio and $\mathrm{sd}(x)$ its standard deviation; $\theta$ is A's baseline log-odds of winning; $\varphi$ absorbs position bias; and $\gamma$ absorbs the length effect. The $\tanh$ gives diminishing returns to very large length gaps and keeps the term antisymmetric, so swapping A and B flips its sign. Dubois et al. normalise the raw length difference; we normalise the log length ratio, which suits a judge whose bias is on the ratio scale.

The length-controlled win rate keeps the fitted model and position terms, averages over the two orders, and sets the length feature to zero:

In words: the win rate A would have against B if their answers had the same length, with position cancelled by averaging. The adjustment is only as good as the fitted $\hat{\gamma }$, and $\hat{\gamma }$ is identified only because length varies across prompts. If every answer of A were exactly 1.6 times as long as B's, the length term would be collinear with the intercept and no regression could separate length from quality.

A worked example#

Take one prompt where A and B are equally good, $d=0$, and A's answer is 1.6 times longer, so $x=\ln 1.6=0.47$. With $b=0.5$ and $\lambda =0.8$:

• A shown first: $\sigma (0+0.5+0.8\times 0.47)=\sigma (0.876)=0.706$.
• A shown second: $\sigma (0-0.5+0.376)=\sigma (-0.124)=0.469$.
• Swap-averaged: $(0.706+0.469)/2=0.588$. The position term cancelled; the 0.376 log-odds of length did not.
• Length-controlled: remove the length term and average, $[\sigma (0.5)+\sigma (-0.5)]/2=0.5$, the correct answer for two equal responses.

On the full simulated pool, the regression recovers $\hat{\varphi }=0.41$ and $\hat{\gamma }=0.56$. Neither equals the true per-prompt coefficients (0.5 and 0.8 on different feature scales), because a single marginal logistic regression averages over heterogeneous prompts and attenuates every coefficient towards zero. That attenuation partly cancels in the prediction step, which is why the length-controlled estimate, 46.7%, still lands within 0.3 points of the 46.4% target.

Length gameability#

Figure 2 reproduces the gameability experiment in miniature. The same model, with the same quality on every prompt, is made concise, neutral, verbose or very verbose by shifting its length distribution. The uncorrected fixed-order win rate ranges from 49.9% to 69.8%; swap-averaging ranges from 40.2% to 60.2%. The length-controlled estimate stays between 44.2% and 46.7%. Measured as the standard deviation across the four verbosity settings divided by the mean, gameability falls from 15% for the swap-averaged win rate to 2% after length control.

The overshoot in the last group is a reminder that the regression is a model of the judge, and a misspecified one shows up exactly where the length gap is extreme. If you suspect your judge's length effect does not saturate, compare a linear and a $\tanh$ feature on held-out verdicts before trusting either.

Code#

The estimator is a few lines. This version takes per-comparison verdicts in both orders.

import numpy as np

def lc_win_rate(y, a_first, log_len_ratio, iters=50):
    """y: 1 if judge preferred A; a_first: 1 if A was shown first, else 0;
    log_len_ratio: log(len_A / len_B) for that comparison."""
    s = np.where(a_first == 1, 1.0, -1.0)
    t = np.tanh(log_len_ratio / log_len_ratio.std())
    X = np.column_stack([np.ones_like(s), s, t])
    beta = np.zeros(3)
    for _ in range(iters):                      # Newton-Raphson for logistic regression
        p = 1 / (1 + np.exp(-X @ beta))
        H = (X * (p * (1 - p))[:, None]).T @ X
        beta += np.linalg.solve(H, X.T @ (y - p))
    th, ph, _ = beta
    sig = lambda z: 1 / (1 + np.exp(-z))
    return 0.5 * (sig(th + ph) + sig(th - ph)), beta

For the interval, bootstrap prompts, not individual verdicts: resample prompt IDs with replacement, keep both orders of each sampled prompt together, refit, and take percentiles. The two verdicts on a prompt are correlated through $d_i$, so treating them as independent understates the standard error. The same clustering logic is covered in our note on clustered standard errors for benchmarks.

Interactive: corrupted and corrected win rates#

The figure below runs the same model as the simulation, in your browser, on the same 2,000 seeded prompts. Move the sliders to change the position bias, the verbosity bias, A's typical length relative to B, and the correlation between length and true quality. Every bar is recomputed from the formulas above, including a fresh logistic regression fit for the length-controlled estimate.

How often a biased judge declares a false win#

Expected values hide sampling noise, so we also ran a Monte Carlo with binary verdicts. Each replication draws 805 fresh prompts, samples one verdict per order per prompt from the judge model, and computes all three estimators. We ran 2,000 replications in each of two scenarios and applied a one-sided test at the 5% level for "A beats B".

Two findings stand out. First, with a judge this biased, swap-averaging alone declares a worse model the winner in 82.8% of replications. More prompts would not help; they only make the wrong answer more confident. Second, length control is not free. Its standard deviation across replications is 1.9 points against 1.3 for the swap-averaged estimate, roughly 2.1 times the variance, because the regression spends information separating length from quality. Plan sample sizes for the adjusted estimator, not the raw one, and compute its interval with a prompt-level bootstrap, as above, rather than a binomial formula.

Failure modes of the corrections#

When length is real quality#

Length control answers a counterfactual: what would the judge say if the two answers had the same length? That is the right question only if length is a nuisance. On some tasks longer answers are better because they are more complete: a step-by-step derivation, a full code listing, a differential diagnosis that covers the alternatives. Then length and quality are correlated, the regression attributes part of real quality to the length coefficient, and the adjustment removes it.

Figure 4 shows the size of the effect. We hold the judge biases at the default and let the log length ratio correlate with the true quality gap. The bias-free target does not move. The length-controlled estimate falls from 46.7% at zero correlation to 40.7% at 0.3, 33.6% at 0.6 and 28.5% at 0.8, an error larger than the verbosity bias it was meant to remove.

No adjustment on judge verdicts alone can solve this, because both stories predict the same verdicts. The way out is outside data: a small human-labelled set where you can estimate how much of the length effect humans share, or controlled pairs where the same content is padded or trimmed so that only length changes.

Other ways the corrections fail#

• Bias that varies by prompt. A single $\varphi$ and $\gamma$ assume one bias level everywhere. Shi et al. report that position bias varies by task and judge [6]. If your evaluation mixes coding, writing and safety prompts, fit the adjustment per category or include interaction terms.
• Mediators other than length. Judges can also respond to formatting and tone. Park et al. catalogue six bias types in judge models, including length bias and concreteness bias, a tendency to give more credit to answers with citations, numbers and technical terminology [8]. Length control adjusts only the feature you put in the regression; a model that learns markdown flourishes instead of padding is untouched.
• Collinearity. If one model is longer on every single prompt by a similar factor, $\gamma$ is poorly identified and the corrected estimate becomes noisy. Check the spread of the length feature before trusting the fit.
• Self-preference. A judge evaluating its own model family may prefer its own style [1]. That is a third nuisance term with no observable covariate, so neither swapping nor length control removes it. Use a judge from a different family, or several judges.
• Ties and scales. Some judges output a 1 to 10 score or allow ties. The algebra above carries over to expected scores, but position bias then also shows up as a shift in the score scale, and you should check for it on single-answer grading too.

Practical guidance#

A short checklist for a judge audit, before any number reaches a decision deck:

1. Randomise or swap order, and log which answer was in which slot for every call.
2. Log the token or character length of both answers for every comparison.
3. Report the first-slot win rate, the inconsistency rate and the length ratio distribution.
4. Fit the length-controlled regression and report it next to the raw and swap-averaged rates, with bootstrap intervals over prompts.
5. On a small human-labelled subset, check whether human preferences share the judge's length effect.
6. Re-run the audit when the judge model, judge prompt or candidate model family changes.

The broader lesson is the one that runs through all judge work: a judge is an instrument with known systematic errors, and an estimator that ignores them reports the instrument, not the model. Position bias you can design away. Verbosity bias you can only model, and a model of the judge is a claim that needs its own validation.

Frequently asked questions#

Sources#

1. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org
2. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, et al.. Large Language Models are not Fair Evaluators, 2023. arxiv.org
3. Keita Saito, Akifumi Wachi, Koki Wataoka, Youhei Akimoto. Verbosity Bias in Preference Labeling by Large Language Models, 2023. arxiv.org
4. Yann Dubois, Balázs Galambosi, Percy Liang, Tatsunori B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, 2024. arxiv.org
5. Ralph Allan Bradley, Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons, 1952. doi.org
6. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, et al.. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge, 2024. arxiv.org
7. Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett. A Long Way to Go: Investigating Length Correlations in RLHF, 2023. arxiv.org
8. Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, Sanghyuk Choi. OffsetBias: Leveraging Debiased Data for Tuning Evaluators, 2024. arxiv.org