Long-context evaluation beyond needle-in-a-haystack: tasks, position effects and effective context length

A model that finds one planted sentence in 128,000 tokens has passed the needle-in-a-haystack test. It has not shown that it can use 128,000 tokens. Retrieval of one literal fact is the easiest thing a long context asks for; finding several facts, combining them, or reasoning across facts scattered through a document degrades much sooner. This note sets out a taxonomy of long-context tasks, a simple model of how length and position interact, precise definitions of effective context length, and how many samples it takes to measure that length without fooling yourself. The figures come from a stylised, seeded simulation, labelled as such, built to show the mechanics.

What the needle test measures#

The needle-in-a-haystack test, popularised in 2023 [1], hides a sentence such as "The special magic number for Lisbon is 4829173" at a chosen depth in long filler text and asks for it back. Sweeping length and depth gives a heatmap, and most recent long-context models produce one that is almost entirely green.

That is useful as a smoke test and weak as an evaluation, for three reasons:

• Literal matching. The question repeats the needle's key words, so the model can find it by surface overlap. NoLiMa removed that overlap and found that 11 of 13 models fell below half of their short-context score by 32K tokens [7].
• One fact. Real tasks need several facts, often of the same type, and errors compound.
• No reasoning. Retrieval is not use. Asking a model to combine two facts it can each retrieve reliably is a different, harder task, and performance on it drops with input length even when the facts are easy to find [10].

RULER made this concrete: across 17 models, almost all nearly aced the vanilla needle test, yet only about half of those claiming 32K or more held satisfactory performance at 32K once tasks included multiple needles, multi-hop tracing and aggregation [3].

A taxonomy of long-context tasks#

Synthetic tasks give control over length, depth and difficulty, which makes them good for measuring a curve. Realistic tasks tell you whether the curve matters for your use case. HELMET argued that synthetic needle tasks correlate poorly with downstream long-context performance and recommended a mix across categories [8]. Use both.

A simple model of length and position#

To reason about these tasks it helps to write down how retrieval of one needle depends on context length $L$ and relative depth $d$ (0 is the start of the context, 1 the end). A convenient form is

Here $\sigma$ is the logistic function, $\ell$ counts doublings beyond 1K tokens (7 at 128K), $a$ sets short-context ability, $b$ is the general cost of each doubling, $c$ is a middle-of-context penalty whose size grows with length, and $r$ gives a small bonus to the most recent tokens. The term $4d(1-d)$ is zero at both ends and one in the middle, which produces the U-shape reported in "Lost in the Middle" [2]: information at the start or end of a long input is used better than information in the middle.

Tasks that need several pieces of information compound this. If a task needs $k$ needles retrieved independently, its accuracy is roughly

the product of the per-needle probabilities at each needle's depth $d_i$. A per-needle rate of 0.95 sounds high; for ten items it gives $0.95^{10}=0.60$. Aggregation tasks are fragile for exactly this reason. Multi-hop tasks add a further factor for combining the facts, which itself declines with length.

Setup (simulation). We set $a=4.2$, $b=0.30$, $c=1.9$, $r=0.6$, and defined five tasks: a single needle; one needle among three distractors (base logit lowered by $0.9+0.12\ell$); retrieving all four values of a key placed close together; aggregating ten items spread uniformly; and a two-hop question whose two facts sit at different depths, with a reasoning factor $0.92e^{-0.06\ell }$. These parameters are illustrative. They were chosen to reproduce the qualitative patterns in the cited benchmarks, not fitted to any model, and the absolute lengths below should not be read as claims about real systems.

Worked example#

At 128K tokens the single-needle probability is 0.89 at the start, 0.94 at the end and 0.55 in the middle. Averaged over depth it is 0.71, so a depth-averaged heatmap reports a respectable number while one fact in two placed mid-document is lost. For the four-value task at 16K, each needle placed mid-document is retrieved with probability 0.83, so all four come back together only $0.83^4\approx 0.48$ of the time. Near the start of the context the per-needle rate is 0.94 and the joint rate 0.76; averaged over depth, the task scores 0.62.

Position effects grow with length#

The middle penalty in the model is scaled by $\ell /7$, so it is negligible at short lengths and dominant at long ones. Liu and colleagues observed this pattern in multi-document question answering and key-value retrieval: accuracy was highest with the relevant information at the start or end of the input and dropped when it sat in the middle, including for models built for long inputs [2].

For evaluation design, this means the depth grid matters as much as the length grid. It also means the position of the question matters: placing the question after the context puts the answer-relevant instruction in the most recent, best-attended position, which is how most production prompts are built and is usually the kinder test.

Why aggregation collapses first#

Inverting the product rule gives the per-item reliability a task needs. For $k$ items to be retrieved together with probability at least $\tau$, each must be retrieved with probability at least

Here $\tau$ is the target task accuracy and $k$ the number of independent items the task needs. With $\tau =0.85$: one item needs 0.85, two need 0.922, four need 0.960, ten need 0.984, and twenty need 0.992. Each extra item pushes the requirement closer to perfect retrieval, and retrieval falls with length, so the length at which the task still passes shrinks quickly as $k$ grows. This is the arithmetic behind the gap between needle scores and aggregation scores, and it is why a model's effective length on a counting or summarisation task is often a small fraction of its needle length. In practice item errors are not fully independent (a model that misses one mid-document item is more likely to miss its neighbour), which can soften or worsen the collapse; the product is a first approximation, not a law.

Defining effective context length#

A model's advertised context length is the longest input it accepts. Its effective context length is the longest input at which it still performs a given task acceptably. Written out,

Here $\overline{\mathrm{Acc}}(L)$ is accuracy at length $L$ averaged over depths, and $\tau$ is the acceptance threshold. Requiring every shorter length to pass as well stops a lucky long-length result from extending the estimate. Three choices hide inside this definition, and each changes the answer:

1. The task. There is no task-free effective length. RULER reports it over a mix of 13 tasks [3].
2. The threshold. RULER uses an absolute threshold, the score of a reference model at 4K (85.6% for Llama 2 7B) [3]. NoLiMa uses a relative one, 85% of the model's own short-context score [7]. A relative threshold makes weak models look long-context capable; an absolute one makes strong short-context models look better than their length handling deserves.
3. Averaging over depth. Averaging hides the middle dip. Using the worst depth instead is stricter and closer to the guarantee a user wants: "wherever the fact is, it will be found".

The same simulated model has an effective context length anywhere from under 1K to 32K depending on the task and the definition. A report that gives one number without naming the task, threshold and depth treatment is not a measurement.

Measuring it with finite samples#

Effective length is a threshold crossing, and threshold crossings are sensitive to noise. If each length and depth cell gets $n$ trials and there are $D$ depths, the standard error of depth-averaged accuracy near the threshold is about

With $\tau =0.85$ and 11 depths, that is 0.048 at 5 trials per cell, 0.024 at 20 and 0.012 at 80. When accuracy declines slowly with length, an error of 0.02 to 0.05 is enough to move the crossing by a whole doubling.

Setup (simulation). For three tasks we drew binomial outcomes at every length and depth cell from the model's true accuracies, estimated the depth-averaged effective length, and repeated this 2,000 times per sample size.

At five trials per cell, which is common in quick needle sweeps, the estimate lands on the true length only a little over half the time, and is off by a factor of two in most of the rest. Even 20 trials per cell misses by a doubling about a quarter of the time. Two consequences follow. Comparisons of effective length between two models need many trials per cell or a statistical test on the accuracy curves themselves, not on the derived lengths. And because the estimate errs in both directions, a vendor-reported effective length from a small sweep can be optimistic as easily as pessimistic.

A real check on a small open model#

To see how the task families behave outside a model, we ran a small probe on Qwen2.5-0.5B-Instruct, which supports a 32K context [13], on a laptop GPU. Haystacks were contiguous passages of a public-domain book (Darwin's On the Origin of Species), with needles inserted at sentence boundaries at depths 0, 0.5 and 1. Each needle value was a fresh random seven-digit number; decoding was greedy. Every cell has 8 trials, so each figure below rests on 24 trials and carries a 95% interval of roughly ±10 to ±20 points.

A single-needle run at 8K, completed before we stopped the probe for time, also scored 24/24. Two observations. First, even a 0.5B model saturates the single-needle test at these lengths, while the same model returns all four values of a key only 42% of the time at 4K (it recovered 74% of individual values). The aggregation arithmetic above is visible at small scale.

Second, our two-hop task looked easy for the wrong reason. Its haystack contained exactly one sentence of the form "X's locker number is N", so the model could answer by matching the words "locker number" and never use the first hop. This is the literal-matching shortcut that NoLiMa was built to remove [7], and we built it into our own test by accident. A correct version adds distractor facts of the same form for other people, so that only the chain through the first fact identifies the right one. We report the flawed result because it is the most common way multi-hop tests go wrong.

Designing a long-context evaluation#

Failure modes in long-context evaluation#

• Reporting the needle test as the result. It is the easiest task in the family and saturates first.
• Averaging over depth only. A model that loses half of mid-document facts at long lengths can still show a high average.
• Single-number effective length. Without task, threshold and depth treatment it cannot be compared or reproduced.
• Too few trials per cell. The estimate jumps a doubling between runs.
• Confusing retrieval with use. Aggregation and multi-hop accuracy can be poor at lengths where retrieval is near perfect.
• Ignoring the cost side. Latency and cost grow with input length; a retrieval pipeline over a shorter context may beat a long-context call on both accuracy and cost for your task.

Code: effective length with a bootstrap interval#

Given trial outcomes per length and depth, this estimates the depth-averaged and worst-depth effective length and how often each value recurs under resampling.

import numpy as np
from collections import Counter

def effective_length(lengths, correct, tau=0.85, worst=False):
    """lengths: sorted list. correct[i][j]: 0/1 array of trials at length i, depth j."""
    best = 0
    for L, row in zip(lengths, correct):
        accs = [np.mean(c) for c in row]
        a = min(accs) if worst else np.mean(accs)
        if a < tau:
            break
        best = L
    return best

def bootstrap_eff(lengths, correct, tau=0.85, worst=False, n_boot=2000, seed=0):
    rng = np.random.default_rng(seed)
    draws = []
    for _ in range(n_boot):
        res = [[c[rng.integers(0, len(c), len(c))] for c in row] for row in correct]
        draws.append(effective_length(lengths, res, tau, worst))
    point = effective_length(lengths, correct, tau, worst)
    return point, {L: k / n_boot for L, k in sorted(Counter(draws).items())}

If the bootstrap puts substantial mass on two neighbouring lengths, report both. That is the honest answer at your sample size.

Frequently asked questions#

Sources#

1. G. Kamradt. LLMTest_NeedleInAHaystack (needle-in-a-haystack pressure test), 2023. github.com
2. N. F. Liu, K. Lin, J. Hewitt et al.. Lost in the Middle: How Language Models Use Long Contexts, 2023. arxiv.org
3. C.-P. Hsieh, S. Sun, S. Kriman et al.. RULER: What's the Real Context Size of Your Long-Context Language Models?, 2024. arxiv.org
4. Y. Bai, X. Lv, J. Zhang et al.. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding, 2023. arxiv.org
5. X. Zhang, Y. Chen, S. Hu et al.. ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens, 2024. arxiv.org
6. Y. Kuratov, A. Bulatov, P. Anokhin et al.. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack, 2024. arxiv.org
7. A. Modarressi, H. Deilamsalehy, F. Dernoncourt et al.. NoLiMa: Long-Context Evaluation Beyond Literal Matching, 2025. arxiv.org
8. H. Yen, T. Gao, M. Hou et al.. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly, 2024. arxiv.org
9. K. Vodrahalli, S. Ontanon, N. Tripuraneni et al.. Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries, 2024. arxiv.org
10. M. Levy, A. Jacoby, Y. Goldberg. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models, 2024. arxiv.org
11. Y. Bai, S. Tu, J. Zhang et al.. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks, 2024. arxiv.org
12. M. Karpinska, K. Thai, K. Lo et al.. One Thousand and One Pairs: A "novel" challenge for long-context language models, 2024. arxiv.org
13. Qwen team: A. Yang et al.. Qwen2.5 Technical Report, 2024. arxiv.org