Benchmark contamination detection: what each detector misses

If a model saw your test questions during training, its score measures recall, not skill. Five families of detectors are in common use: n-gram overlap, likelihood scores such as Min-K% Prob, canary strings, guided completion and order-based tests. Each looks for a different fingerprint, and each one goes blind to a different kind of leak. This note explains how each works, then uses a seeded simulation to show the failure that matters most in practice: once leaked items are paraphrased, the score inflation survives and item-level detection mostly does not.

A concrete case: 2.7 points nobody can see#

Take a 2,000-question multiple-choice benchmark. Suppose 10% of the questions leaked into a training corpus, some as verbatim copies and some as rewrites. On clean questions the model answers 55% correctly. On the leaked ones it has partly memorised the answers and gets 82% right. The headline score becomes $0.9\times 55\%+0.1\times 82\%\approx 57.7\%$, a gain of 2.7 points in our simulation. That gap is larger than the difference between many adjacent entries on a public leaderboard, and it comes entirely from exposure rather than capability.

The natural response is to scan for contamination, flag the leaked items and report a clean score. Whether that works depends on how the questions leaked. If the copies are verbatim, a 13-gram scan against the training corpus finds essentially all of them. If the leaked copies were rewritten so that 30% of tokens changed, the same scan finds 2% of them at a 1% false alarm rate, while the score inflation drops only from 2.7 to 2.4 points. The rest of this note explains why, detector by detector, and what to do instead.

The five detector families#

Contamination detectors differ in what access they need (the training corpus, the model's token probabilities, or only its generated text) and in the granularity of their verdict (single item or whole benchmark). Sainz et al. argued that contamination has to be measured for each benchmark and each model rather than assumed away [14]; the methods below are the main ways to do that. The table summarises them, and the subsections explain the mechanics.

n-gram overlap#

The oldest method checks whether long exact token sequences from a test item appear in the training data. The GPT-3 paper flagged test examples that shared a 13-gram with the training set and reported results on the "clean" subset separately [1]. The Llama 2 report used a token-level variant, marking a sample as contaminated when a large share of its tokens sit inside long matching spans found in the pretraining corpus [2]. The method is cheap, interpretable and has a near-zero false positive rate when the threshold is a long span.

Its weakness is simple arithmetic. Suppose a leaked copy was rewritten so that each token is changed independently with probability $s$, the paraphrase strength. A particular $n$-gram survives only if all $n$ of its tokens survive:

Here $s$ is the share of tokens the rewrite changed and $n$ is the span length the scan looks for. Because the exponent is the span length, survival collapses quickly: long spans are exactly what makes the check precise, and exactly what makes it fragile.

A worked example with $n=13$ and a 40-token question, which contains 28 overlapping 13-grams. At $s=0.1$, each 13-gram survives with probability $0.9^{13}=0.254$, so a leaked item still shares about 7.1 of its 28 spans with the corpus, far above the 0.28 expected by chance at our assumed 1% background match rate. At $s=0.3$, survival is $0.7^{13}=0.0097$: about 0.55 matching spans against 0.28 by chance. Most leaked items then match zero or one span, exactly like clean items. Yang et al. showed this happens in practice. Paraphrased or translated copies of MMLU, GSM8K and HumanEval passed n-gram and embedding-based decontamination, and training on them pushed a 13B model to scores they describe as on par with GPT-4 [3].

Perplexity and Min-K% Prob#

Without the corpus, you can ask the model itself. Text a model has trained on tends to be more likely under it. Raw perplexity is a poor signal because simple, common text is also likely. Min-K% Prob [4] focuses on the hardest tokens instead: an unseen passage usually contains a few outlier tokens with very low probability, while a memorised one has fewer.

Here $x$ is the test item as a token sequence, $p_{\theta }(x_i\mid x_{<i})$ is the model's probability for token $i$ given the tokens before it, and $S_k$ is the set of the $k\%$ of tokens with the lowest probability (Shi et al. found $k=20$ worked well). A higher score means even the least likely tokens were fairly likely, which points to exposure. Min-K%++ [5] normalises each token's log-probability by the mean ${\mu }_{<i}$ and standard deviation ${\sigma }_{<i}$ of log-probabilities under the model's own next-token distribution at that position:

The intuition is that a training token tends to sit at a local peak of the model's distribution, so it should be likely relative to the alternatives, not just likely in absolute terms.

The catch is calibration. Duan et al. ran these attacks against models trained on the Pile and found they barely beat random guessing for most model sizes and domains [6]. Two reasons stand out: pretraining sees most documents roughly once, so the likelihood bump is small, and "member" and "non-member" text overlap heavily in content. They also showed that some earlier successes came from a distribution shift between members and non-members (for example, non-members drawn from a later time period) rather than from membership itself. For contamination this cuts both ways: a benchmark fine-tuned on for several epochs leaves a much larger bump than a document seen once in pretraining.

Canary strings#

BIG-bench ships a unique identifier string in its task files and asks that any document containing it be excluded from training corpora [7]. A canary does two jobs: corpus builders can filter on it, and auditors can prompt a model to complete the string. If a model reproduces the full identifier, it saw some file that carried it. The signal has essentially no false positives. It also has narrow coverage. The canary marks the file, not each item, so any copy that drops the header (a forum post quoting one question, a scraped table, a reformatted dataset on a mirror) carries no marker. A model that never reproduces the string tells you very little.

Guided completion#

Golchin and Surdeanu's "guided instruction" prompts a model with the dataset name, the split, and the opening of a test instance, and asks it to finish the instance [8]. An instance is flagged if the completion exactly or nearly matches the reference. A partition is flagged when completions under the guided prompt overlap the references (by ROUGE-L or BLEURT) significantly more than completions under a generic prompt without the dataset name, or when a few-shot classifier marks several completions as exact or near-exact matches. Deng et al.'s slot-guessing protocol is a close relative: mask a wrong answer option in a multiple-choice item and ask the model to supply it. Their abstract reports exact-match rates of 52% and 57% for two commercial models on MMLU, which is hard to explain without exposure [9]. These methods need only generated text, so they work on models behind an API. They depend on verbatim recall, which grows with repetition [10], and they do not fire when the model learned the content in a different wording.

Exchangeability tests#

Oren et al. use a property of the test file rather than of single items [11]. If a benchmark's examples are exchangeable, every ordering of them is equally plausible to a model that never saw the file. A model that trained on the file in its published order will assign that canonical order a higher likelihood than shuffled orders. Comparing the canonical log-likelihood ${\ell }_0$ against $M$ random permutations gives a p-value with a guaranteed false positive rate:

Here ${\ell }_j$ is the log-likelihood of the benchmark under the $j$-th shuffled order. If no shuffle beats the published order, $\hat{p}$ is as small as $1/(1+M)$. The authors report the test detects contamination in models as small as 1.4B parameters, on test sets of 1,000 examples, even when the dataset appears only a few times in pretraining. Its blind spot follows from its premise: it can only see contamination that preserved the file's order. Shuffled, sampled or rewritten copies leave no ordering signal.

A simulation of all four item-level detectors#

To compare detectors on equal terms we wrote a small generative model of a contaminated benchmark. It is deliberately simple, so every assumption can be stated in one line and changed. Each of the 2,000 items is contaminated with probability $c$. A contaminated item was seen as a rewrite with paraphrase strength $s\in [0,1]$. The four item-level scores are generated as follows:

The choices encode what the literature reports qualitatively. Exact-span methods depend on surface form. Likelihood scores keep a small residual lift (0.3 standard deviations) after heavy rewriting, because the content tokens are still familiar. Guided completion needs near-verbatim recall, so its lift decays fast. Canaries survive in a minority of leaked copies and vanish under rewriting. Crucially, the accuracy gain from contamination decays only mildly with paraphrasing, which matches Yang et al.'s finding that rephrased training data inflates scores about as well as verbatim data [3]. The interactive figure below runs exactly this model in your browser with the same random number generator and seed as our Python script, so the numbers in the text and the figure agree.

What the simulation shows#

With verbatim leaks, 13-gram overlap is perfect (AUC 1.00), Min-K% reaches 0.86, guided completion 0.79 and canaries 0.65. The ROC area hides an important detail: at a 1% false positive rate, Min-K% catches only 21% of contaminated items, while guided completion catches 45%. Guided completion has a steep start because when recall works it is unambiguous, but it does nothing for the 40% of items the model cannot reproduce. Min-K% shifts every contaminated item a little, which shows up as area under the curve but not as confident flags.

Paraphrasing reorders the ranking (Figure 2). By $s=0.2$ the 13-gram AUC is 0.86 and by $s=0.3$ it is 0.60. Min-K% degrades slowly, from 0.86 to 0.80 at $s=0.3$ and 0.74 at $s=0.5$, but it was never a confident detector. Guided completion falls to 0.66 at $s=0.3$. The canary is gone almost immediately.

True positive rate at a 1% false positive rate per detector (simulation, 10% contamination, 40,000 items). Inflation is observed accuracy minus accuracy on clean items, in percentage points.

Inflation survives filtering#

The practical question is not "what is the AUC" but "if I drop every flagged item, how much of the inflation is left?" We combined the detectors into a single filter: flag an item if any of the three continuous scores exceeds its 99.67th clean-item percentile, or if the canary fires. That gives an overall false positive rate of 0.89%. With verbatim leaks the filter catches 100% of contaminated items and removes all of the inflation. At $s=0.2$ it catches 39%; at $s=0.3$, 11%, and 89% of the inflation remains in the "decontaminated" score (2.1 of 2.4 points).

This is the core result. A team that runs a decontamination pass, sees few flags, and reports "no significant contamination" may simply be looking for the wrong fingerprint.

Base rates: why flags mislead at low contamination#

ROC curves do not depend on how many items are contaminated, but the meaning of a flag does. The share of flagged items that are truly contaminated (precision, or positive predictive value) is

Here $c$ is the contamination rate, TPR the share of contaminated items flagged and FPR the share of clean items flagged. When $c$ is small, the false positives from the large clean majority dominate. Worked example: with $c=2\%$, a detector catching half of contaminated items at a 1% false positive rate gives $\text{PPV}=0.01/(0.01+0.0098)=0.51$. Half of the flags are wrong. In the simulation at 2% contamination with no paraphrasing, Min-K% at a 1% false positive rate flags 33 items, of which 39% are truly contaminated. Item-level lists from likelihood detectors should be read as leads to verify, not as a verdict.

Benchmark-level tests have more power#

Item-level detection is hard because a small shift per item is invisible against item-to-item noise. Averaged over a whole benchmark, the same small shift becomes visible, provided you have a reference set to compare against. The cleanest reference is a fresh set written to match the benchmark's style and difficulty, as Zhang et al. did with GSM1k for GSM8K [12]. They found accuracy drops on the fresh set for several model families, and a positive relationship between a model's likelihood of generating GSM8K examples and its GSM8K-to-GSM1k gap, while many frontier models showed minimal signs of overfitting.

We simulated two benchmark-level tests under the same model at $s=0.3$ (heavy paraphrasing, where item-level detection has mostly failed), with the 2,000-item benchmark and a 1,000-item fresh matched set, 4,000 replications per point. The first test compares accuracy between the two sets. The second compares the mean Min-K% score. Both use a one-sided test at the 5% level.

At 10% contamination the mean Min-K% comparison detects contamination in 89% of replications, against 33% for the accuracy comparison; at 5% contamination the figures are 42% and 16%. The likelihood test is more powerful because its per-item shift (1.14 standard deviations) is large relative to its noise, while a binary correct-or-not outcome is noisy and the accuracy gain is diluted by items the model would have answered correctly anyway. Neither test says which items leaked. Both depend on the fresh set truly matching the original; a fresh set that is slightly harder produces the same signature as contamination. If items in either set share passages or templates, the standard error in the accuracy test must account for that clustering; see our note on clustered standard errors for benchmarks.

What each detector misses#

Two misses deserve emphasis. First, fine-tuning data is the most likely route for benchmark-adjacent text, and it is often synthetic. Yang et al. found test overlap in synthetic data generated by strong models, which means contamination can arrive without anyone copying a benchmark [3]. Second, training on the benchmark's own training split, or on closely related public data, raises scores in ways no detector here was designed to see. Zhou et al. showed how much even small amounts of benchmark-related data can move results [13].

Practice: a detection plan that matches the threat#

Choose detectors by the access you have and the leak you fear. A short decision guide:

A checklist we apply before trusting a public benchmark number:

1. Decide which leak you are testing for. "No n-gram hits" rules out verbatim copies in a corpus you searched. It says nothing about paraphrases or about corpora you did not search.
2. Report precision, not only AUC. At low contamination rates most item-level flags are false. Verify a sample of flags by hand before excluding items.
3. Prefer aggregate tests for the yes-or-no question. Benchmark-level comparisons against a matched reference have far more power than item-level flags.
4. Keep a private held-out set. It is the only control that does not depend on knowing how the leak happened. Rotate it once it has been used for public claims.
5. Calibrate on known members. Fine-tune a small model on a random half of a held-out set and check that your detector separates the halves before you trust it on a model you cannot inspect.

A minimal implementation of the two scores you can compute from token log-probabilities:

import numpy as np

def min_k_prob(token_logprobs, k=0.2):
    """Mean log-probability of the k fraction of least likely tokens."""
    lp = np.sort(np.asarray(token_logprobs))
    n = max(1, int(len(lp) * k))
    return lp[:n].mean()

def ngram_survival(s, n=13):
    """Chance an n-gram survives a rewrite that changes each token w.p. s."""
    return (1 - s) ** n

# A leaked item rewritten at s=0.3 keeps ~1% of its 13-grams:
print(round(ngram_survival(0.3), 4))   # 0.0097

Failure modes in contamination audits#

• Reference sets from a different time or source. Comparing likelihoods of benchmark items against newer web text conflates contamination with temporal drift, the confound Duan et al. identified [6].
• Thresholds tuned on the model under audit. Any threshold chosen to make flags look reasonable on the same data invalidates the false positive rate.
• Treating a clean scan as a clean bill. In our simulation at $s=0.3$ and 10% contamination, only 58% of the combined filter's flags are truly contaminated, and removing them leaves 89% of the inflation in place.
• Normalisation gaps. Whitespace, casing, LaTeX and option-letter formatting differences defeat exact matching as effectively as paraphrase.
• Ignoring the answer key. A model can memorise answers without memorising questions. Leaked answer keys or solution write-ups inflate scores while leaving the question text unseen.

Frequently asked questions#

Sources#

1. T. B. Brown et al.. Language Models are Few-Shot Learners, 2020. arxiv.org
2. H. Touvron et al.. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. arxiv.org
3. S. Yang, W.-L. Chiang, L. Zheng, J. E. Gonzalez, I. Stoica. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, 2023. arxiv.org
4. W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, L. Zettlemoyer. Detecting Pretraining Data from Large Language Models, 2023. arxiv.org
5. J. Zhang et al.. Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models, 2024. arxiv.org
6. M. Duan, A. Suri, N. Mireshghallah, S. Min et al.. Do Membership Inference Attacks Work on Large Language Models?, 2024. arxiv.org
7. A. Srivastava et al.. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, 2022. arxiv.org
8. S. Golchin, M. Surdeanu. Time Travel in LLMs: Tracing Data Contamination in Large Language Models, 2023. arxiv.org
9. C. Deng, Y. Zhao, X. Tang, M. Gerstein, A. Cohan. Investigating Data Contamination in Modern Benchmarks for Large Language Models, 2023. arxiv.org
10. N. Carlini, F. Tramer, E. Wallace, M. Jagielski et al.. Extracting Training Data from Large Language Models, 2020. arxiv.org
11. Y. Oren, N. Meister, N. Chatterji, F. Ladhak, T. B. Hashimoto. Proving Test Set Contamination in Black Box Language Models, 2023. arxiv.org
12. H. Zhang et al.. A Careful Examination of Large Language Model Performance on Grade School Arithmetic, 2024. arxiv.org
13. K. Zhou et al.. Don't Make Your LLM an Evaluation Benchmark Cheater, 2023. arxiv.org
14. O. Sainz, J. A. Campos, I. Garcia-Ferrero, J. Etxaniz, O. Lopez de Lacalle, E. Agirre. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark, 2023. arxiv.org