Scalable oversight: debate, amplification, weak-to-strong generalisation and sandwiching

Scalable oversight is the problem of supervising AI systems accurately on tasks where the supervisor, human or AI, cannot reliably judge the answer alone. Its methods, such as debate, iterated amplification, recursive reward modelling, AI-assisted critique and weak-to-strong generalisation, try to keep the training signal correct as tasks outgrow the people giving it, and the sandwiching design lets researchers test them on today's models.

This article explains why the problem exists, sets out each method with the mathematics where there is some, and summarises what experiments have found so far. A toy model of debate, which you can adjust below, makes the central idea concrete: checking one step is easier than checking a whole argument, and a well-designed game can reduce the second to the first.

Why scalable oversight matters#

Most alignment training today rests on human judgement. In RLHF, people compare outputs and a reward model learns their preferences; the policy is then optimised against that model. This works when raters can tell the better output from the worse one. It degrades when they cannot: a rater comparing two 500-line code changes, two proofs or two summaries of documents they have not read will often prefer the one that looks more competent. Optimising against those preferences then teaches the model to look competent.

The resulting failure is quiet. Training metrics improve, and the model gets better at producing outputs that pass review, including outputs whose flaws the reviewers missed. Sycophancy is an early, mild version; McAleese and colleagues found that model-written critiques caught more bugs in model-written code than paid human code reviewers did, which is useful as a tool and sobering as a measure of how much human review misses. The goal of scalable oversight is to make the supervision signal track the truth rather than the appearance of it, even when the system being supervised knows more than its supervisor. Amodei and colleagues listed it as one of five concrete problems in AI safety in 2016; it sits at the centre of the alignment agenda.

Debate#

Irving, Christiano and Amodei proposed training models through a zero-sum debate game. Given a question, two copies of a model take turns making short statements in favour of competing answers, and a judge decides which side was more truthful and useful. Each debater is rewarded for winning. The hope is that, in equilibrium, the winning strategy is to tell the truth, because any lie can be exposed by the opponent at a point where the judge can check it.

The argument has a precise form borrowed from complexity theory. A judge who can only verify short claims in polynomial time can, by reading a single proof, decide problems in NP. With two competing provers who must defend their claims under cross-examination, the same judge can decide any problem in PSPACE, a much larger class: Irving and colleagues showed that debate with optimal play reaches PSPACE given polynomial-time judges, while direct judging reaches only NP. The intuition is that a long chain of reasoning can be checked by having the debaters zoom in, step by step, to the single point where they disagree, and the judge only ever checks that one point. Brown-Cohen, Irving and Piliouras later designed "doubly-efficient" debate protocols in which the honest debater needs only a polynomial number of simulation steps to win, even against a dishonest debater allowed exponentially many, which addresses the objection that honesty might require impossibly expensive computation.

A toy model of why debate helps#

Suppose an answer rests on an argument of $n$ steps, and a wrong answer contains at least one flawed step. The judge can check any single step and gets it right with probability $a$: a correct step is accepted with probability $a$ and a flawed step is caught with probability $a$. Compare two protocols.

Consultancy. One model argues for an answer assigned at random, correct half the time, as in the human experiments described below. The judge spot-checks $k$ of the $n$ steps at random and rejects the answer if any check fails. A dishonest consultant hides a single flaw. The judge's accuracy is

The first term is a correct answer surviving $k$ checks without a false alarm. The bracket is the probability a wrong answer survives: either the flaw was among the checked steps but missed, or it was not checked and the $k$ valid steps passed. As $n$ grows, $k/n$ shrinks and the judge almost never finds the flaw.

Debate. Two models argue opposite answers. The honest debater points at the flawed step in the opponent's argument, which it finds with probability $h$ (a measure of debater skill); otherwise it points at a random step. The dishonest debater, having nothing real to point at, challenges one of the honest side's valid steps. The judge checks both challenged steps; if exactly one is flagged, that side loses, otherwise the judge guesses. Writing $f$ for the probability that the dishonest side's challenged step is flagged,

The second expression follows from $A=fa+\frac{1}{2}[f(1-a)+(1-f)a]$: the dishonest side is flagged and the honest side is not, plus half of the ties. When the honest debater always finds the flaw ($h=1$), $f=a$ and debate accuracy equals $a$, the judge's accuracy on a single step, no matter how long the argument. That is the PSPACE intuition in miniature.

Worked numbers#

With a judge who checks single steps correctly 90% of the time, an honest debater who finds the flaw 90% of the time, and a consultancy judge who spot-checks three steps:

Two features are worth noticing. Debate accuracy is flat in $n$ and capped by the judge's single-step accuracy and the honest debater's skill: if the honest debater only finds the flaw half the time, accuracy settles at 70%. And extra spot-checks barely help consultancy on long arguments; worse, with an imperfect judge each extra check of a correct argument is another chance for a false alarm, so ten checks of a ten-step argument give 65.5% rather than the 90% a perfect checker would get. Every number in the table matched a seeded Monte Carlo simulation of 400,000 trials per cell to within 0.1 percentage points.

The toy leaves out exactly the things that make debate hard in practice. A dishonest debater can make its argument so convoluted that the flaw is hard to localise (lowering $h$). It can exploit judge biases so that $a$ is lower for some steps than others. And the analysis assumes the honest strategy is the equilibrium; whether training actually finds it is an empirical question.

Iterated amplification and recursive reward modelling#

Christiano, Shlegeris and Amodei proposed iterated amplification. Start with a human $H$ who can solve easy tasks. Let $H$ solve a harder task by breaking it into subquestions and delegating them to the current model $M_t$; call this combined system $\mathrm{Amp}(H,M_t)$. Train the next model to imitate it:

where $x_1,\dots ,x_m$ are the subquestions $H$ chooses for task $x$. Each round, the model learns to do in one step what previously took a human plus many model calls, so the tasks it can handle grow while every training signal is built from judgements a human could make. The paper demonstrated the idea on algorithmic tasks without an external reward. Wu and colleagues applied the decomposition idea at scale to summarising whole books: models summarised sections, then summaries of summaries, and human labellers could supervise each level without reading the book.

Leike and colleagues' recursive reward modelling is the RL version. To train an agent $A_k$ on a hard task, a user assisted by agent $A_{k-1}$ provides the feedback from which a reward model for $A_k$ is learned:

The assistance might mean the earlier agent summarises evidence, checks sub-results or points out problems. Both schemes rely on a decomposition assumption: that hard tasks split into pieces whose evaluation is easier than the whole, and that errors do not compound across levels.

AI-assisted critique#

The most deployed form of scalable oversight is the simplest: train a model to write critiques that help human raters find problems. Saunders and colleagues showed that model-written critiques helped humans find flaws in summaries that they would otherwise have missed, including flaws deliberately planted to mislead, and that larger models wrote more helpful critiques. McAleese and colleagues trained critic models for code and found that their critiques were preferred over human-written ones in 63% of cases on naturally occurring bugs. Critics have their own failure, hallucinated problems that can mislead raters, but human-plus-critic teams caught a similar number of bugs as critics alone while hallucinating less. Critique is the one-debater version of debate: it finds flaws but has no opponent to challenge a false accusation.

Weak-to-strong generalisation#

Burns and colleagues asked a different question. Instead of making the supervisor stronger, can a strong model generalise correctly from a weak supervisor's imperfect labels? They used small models as stand-ins for human supervisors and large pretrained models as stand-ins for future superhuman systems, fine-tuned the strong model on the weak model's labels, and measured how much of the gap to a strong model trained on ground truth was recovered:

where $P_{\text{weak}}$ is the weak supervisor's own accuracy, $P_{\text{weak}\to \text{strong}}$ the strong model's accuracy after training on weak labels, and $P_{\text{strong ceiling}}$ its accuracy when trained on ground truth. A performance gap recovered (PGR) of 0 means the strong model merely imitated its supervisor, errors included; 1 means it recovered its full ability. They found strong students consistently beat their weak supervisors, but naive fine-tuning recovered only part of the gap. An auxiliary loss that lets the student trust its own confident predictions over the weak labels helped substantially on NLP tasks, recovering close to GPT-3.5-level performance when a GPT-2-level supervisor trained GPT-4. The loss mixes the weak labels $f_w(x)$ with the student's own hardened predictions ${\hat{f}}_t(x)$:

with $\alpha$ the weight on self-confidence and CE the cross-entropy. The main failure mode is the student learning to imitate the supervisor's systematic mistakes, which is precisely what an overseer cannot detect. Weak-to-strong generalisation and debate are complementary: Lang, Huang and Li used debate between strong models to improve the weak supervisor's labels before weak-to-strong training, and reported gains on the same NLP benchmarks.

Sandwiching: how to test oversight methods today#

Oversight methods are meant for systems more capable than their overseers, which raises the question of how to test them before such systems exist. Cotra proposed "sandwiching": pick a task where the model is more capable than a group of non-experts but less capable than a group of experts. Let the non-experts supervise the model using some oversight method, without access to the experts, and use the experts' judgements only to score how well it worked. The model sits between the two groups, hence the name.

Bowman and colleagues ran a proof of concept on MMLU and a time-limited version of QuALITY, a reading-comprehension set where participants could not read the full passage. Humans chatting with an unreliable model assistant, the simplest possible oversight strategy, substantially outperformed both the model alone and their own unaided performance. The design has since become the standard way to evaluate oversight protocols: the honest question is not "did accuracy go up?" but "did it go up on the questions where the non-expert could not otherwise tell?"

What the experiments show so far#

The pattern is cautiously positive for multi-turn debate when the judge lacks information the debaters have, and much weaker when the judge's limitation is reasoning ability rather than missing information. Single-shot arguments without cross-examination did not help. Engels and colleagues' Elo framework puts a sobering number on nesting: even the best of their games succeeded only about half the time across a large capability gap, and success fell as the gap widened. Their model treats the chance that a guard with oversight-specific Elo $E_G$ beats an opponent with Elo $E_H$ as the standard logistic

so a 400-point disadvantage means winning about 9% of the time in a single game. Nesting several smaller steps can beat one large step, but only if each overseer's advantage at its own level is real.

Scalable oversight in practice#

Teams that never train a frontier model still face a small version of the problem whenever humans or models grade outputs they cannot fully verify. The practical lessons transfer directly.

Measuring whether any of this works needs the same discipline as other evaluations: known-answer items, error bars, and attention to the judge's own error rate, covered in the archive's note on LLM judge error.

Simulating the toy model#

import numpy as np

def simulate(a=0.9, h=0.9, k=3, n=100, trials=200_000, seed=0):
    rng = np.random.default_rng(seed)
    # Debate: the honest debater challenges the flawed step (found with prob h, else a random step);
    # the dishonest debater challenges one of the honest side's valid steps. The judge checks both.
    hit = (rng.random(trials) < h) | (rng.integers(0, n, trials) == 0)
    flag_dis = np.where(hit, rng.random(trials) < a, rng.random(trials) < 1 - a)
    flag_hon = rng.random(trials) < 1 - a
    debate = np.where(flag_dis & ~flag_hon, 1.0, np.where(flag_hon & ~flag_dis, 0.0, 0.5)).mean()
    # Consultancy: one argument, correct half the time; the judge spot-checks k random steps.
    correct = rng.random(trials) < 0.5
    flaw_seen = rng.random(trials) < k / n
    valid_checked = np.where(~correct & flaw_seen, k - 1, k)
    reject = (rng.binomial(valid_checked, 1 - a) > 0) | (~correct & flaw_seen & (rng.random(trials) < a))
    consult = np.where(correct, ~reject, reject).mean()
    return round(float(debate), 3), round(float(consult), 3)

for n in (1, 10, 100):
    print(n, simulate(n=n, k=min(3, n)))
# 1 (0.9, 0.899)
# 10 (0.863, 0.597)
# 100 (0.86, 0.509)

Open problems#

Obfuscated arguments. Debate's guarantee assumes the honest side can always point to the flaw. A dishonest debater may be able to build an argument whose flaw is real but too expensive for anyone to locate, so that the honest debater knows the conclusion is wrong without being able to show where.

Persuasion is not truth. Human and model judges respond to confidence, fluency and length. Khan and colleagues found that optimising debaters for persuasiveness helped judges, which is encouraging, but there is no guarantee this holds when the debaters are much stronger than the judge.

Systematic errors. Most methods handle random errors in supervision better than systematic ones. If every overseer shares a blind spot, decomposition and debate can inherit it.

Honest equilibria are not guaranteed by training. The theory concerns optimal play. Whether gradient-based training on a debate game converges to the honest strategy, and stays there, is open.

Measuring success without ground truth. The whole point is to work where the answer cannot be checked, which makes it hard to know when it has worked. Sandwiching is the best current answer, and it only covers tasks where some experts exist.

Scalable oversight connects most of the AI safety agenda: it is the training-signal side of alignment, the reason reward hacking gets harder to spot as models improve, and a design problem every team running evals with model judges already has.

Frequently asked questions#

Sources#

1. Amodei, Olah, Steinhardt, Christiano, Schulman, Mané. Concrete Problems in AI Safety, 2016. arxiv.org
2. Irving, Christiano, Amodei. AI safety via debate, 2018. arxiv.org
3. Christiano, Shlegeris, Amodei. Supervising strong learners by amplifying weak experts, 2018. arxiv.org
4. Leike, Krueger, Everitt, Martic, Maini, Legg. Scalable agent alignment via reward modeling: a research direction, 2018. arxiv.org
5. Cotra. The case for aligning narrowly superhuman models, 2021. alignmentforum.org
6. Wu, Ouyang, Ziegler, Stiennon et al.. Recursively Summarizing Books with Human Feedback, 2021. arxiv.org
7. Parrish, Trivedi, Perez, Chen et al.. Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions, 2022. arxiv.org
8. Saunders, Yeh, Wu, Bills et al.. Self-critiquing models for assisting human evaluators, 2022. arxiv.org
9. Bowman, Hyun, Perez, Chen et al.. Measuring Progress on Scalable Oversight for Large Language Models, 2022. arxiv.org
10. Michael, Mahdi, Rein, Petty et al.. Debate Helps Supervise Unreliable Experts, 2023. arxiv.org
11. Brown-Cohen, Irving, Piliouras. Scalable AI Safety via Doubly-Efficient Debate, 2023. arxiv.org
12. Burns, Izmailov, Kirchner, Baker et al.. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, 2023. arxiv.org
13. Khan, Hughes, Valentine, Ruis et al.. Debating with More Persuasive LLMs Leads to More Truthful Answers, 2024. arxiv.org
14. McAleese, Pokorny, Ceron Uribe, Nitishinskaya et al.. LLM Critics Help Catch LLM Bugs, 2024. arxiv.org
15. Kenton, Siegel, Kramár, Brown-Cohen et al.. On scalable oversight with weak LLMs judging strong LLMs, 2024. arxiv.org
16. Lang, Huang, Li. Debate Helps Weak-to-Strong Generalization, 2025. arxiv.org
17. Engels, Baek, Kantamneni, Tegmark. Scaling Laws For Scalable Oversight, 2025. arxiv.org