How to evaluate reinforcement learning agents: seeds, IQM and confidence intervals

To evaluate a reinforcement learning agent, train it from many independent random seeds, score each final policy on enough evaluation episodes, normalise scores per task, and report an aggregate such as the interquartile mean (IQM) with a stratified bootstrap confidence interval, alongside a performance profile that shows the whole distribution of runs. A mean over three seeds, with no interval, is not an evaluation: it is a single noisy draw from a wide distribution.

This page explains why RL results are so noisy, what each of the recommended tools does, and how many seeds you need. Every number comes from an experiment we ran: three tabular Q-learning variants on five small tasks, 100 seeds each, 1,500 training runs in total. You can resample those runs yourself in the seed explorer below.

Why are RL results so noisy?#

A supervised model trained twice with different seeds usually lands within a fraction of a point of itself. An RL agent can land anywhere. The seed controls network initialisation, the environment's random transitions and, most importantly, the agent's own exploration. Early luck compounds: an agent that stumbles on a reward in its first few hundred steps builds its value estimates around that discovery, while an identical agent that does not may never leave a poor local behaviour. Reinforcement learning learns from data it collects itself, so the noise feeds back into the data.

Henderson et al. showed in 2017 that two sets of five seeds of the same algorithm, differing only in seed, could produce learning curves different enough to look like two different algorithms. Agarwal et al. later re-examined the Atari 100k benchmark, where papers typically reported only a handful of runs per game, and found that several published rankings did not survive a proper treatment of uncertainty. Their paper, Deep Reinforcement Learning at the Edge of the Statistical Precipice, is the standard reference for the tools on this page, and its open-source library rliable implements them.

Two levels of randomness: seeds and episodes#

If you train $N$ seeds and evaluate each final policy on $m$ episodes, the variance of the average score is

where ${\sigma }_{\text{seed}}^2$ is the variance of a policy's true expected return across training seeds and ${\sigma }_{\text{ep}}^2$ is the variance of returns across evaluation episodes for a fixed policy. The second term shrinks with more evaluation episodes, which are cheap. The first term only shrinks with more training runs, which are expensive. In practice the first term dominates, and adding evaluation episodes to a three-seed study buys almost nothing. Our experiment removes the second term entirely: we computed each final policy's expected return exactly by solving the Bellman equations for that policy, so all the variance you see comes from training.

Our experiment: 1,500 seeded Q-learning runs#

Setup (real runs, not simulated scores). We wrote five small tabular tasks in numpy: an 8×8 gridworld with a single rewarding goal, a 10×10 gridworld where moves slip in a random direction 20% of the time, an 8×8 cliff-walking grid, and two RiverSwim-style chains of length 6 and 8 where a small reward sits at the left end and a large one at the far right, behind a current that pushes the agent back. Discount $\gamma =0.95$. We trained three variants of Q-learning with a fixed budget of environment steps per task:

• A: constant ε-greedy exploration, ε = 0.2, step size α = 0.2.
• B: ε decays linearly from 1.0 to 0.05 over the first half of training, α = 0.2.
• C: the same schedule as B with a larger step size, α = 0.5.

Each variant ran with 100 independent seeds per task. For every final greedy policy we computed its exact expected discounted return from the start state, then normalised it so that 0 is the uniform random policy and 1 is the optimal policy found by value iteration:

where $G$ is the policy's expected return, $G_{\text{rand}}$ the random policy's and $G_{\text{opt}}$ the optimal policy's. Normalisation puts tasks with very different reward scales on one axis, which is what makes aggregation across tasks meaningful. Atari papers do the same with human and random reference scores.

Two facts stand out. First, A is clearly worse: constant exploration at ε = 0.2 never reaches the far end of either chain. Second, B and C are, for practical purposes, tied: 0.784 against 0.787 with 100 seeds per task, and their 95% intervals (0.758 to 0.811 for B, 0.760 to 0.815 for C) overlap almost completely. That tie is the interesting case, because it is exactly the situation in which a paper with three seeds per task will report a winner anyway.

The individual runs are far from normally distributed. On the 8×8 grid, about a fifth of runs never find the goal and score near 0 while the rest score exactly 1. On Chain 8, B's runs split almost evenly between "found the big reward" (53% of runs above 0.99) and "settled for the small one". A mean over three such runs depends mostly on how many of the three happened to land in each mode.

Mean, median or IQM: which aggregate should you report?#

Suppose you have $M$ tasks and $N$ runs per task, so $MN$ normalised scores $x_{m,n}$. Three aggregates are common.

• Mean of all $MN$ scores. Uses every run, but a few extreme runs move it a lot, and on unbounded scales one task can dominate.
• Median of per-task means, long the headline number on Atari. Insensitive to one outlying task, but with few tasks it is decided by whichever task sits in the middle, and it ignores how well the algorithm does on every other task.
• Interquartile mean (IQM): pool all $MN$ scores, sort them, discard the bottom and top 25%, and average the middle half.

where $n=MN$ is the total number of runs and $x_{(i)}$ is the $i$-th smallest score. The IQM sits between the mean and the median: it ignores the extreme quarter at each end, so a handful of lucky or collapsed runs cannot move it, yet it averages half of all runs rather than relying on one middle value, so it has far lower variance than the median.

Worked example: IQM from 15 runs#

Take the first three seeds of algorithm B on each of the five tasks. Sorted, the 15 normalised scores are −0.020, −0.008, 0.002, 0.770, 0.770, 0.807, 0.868, 0.978, 1, 1, 1, 1, 1, 1, 1. With $k=\lfloor 15/4\rfloor =3$, drop the three lowest and three highest and average the middle nine: the IQM is 0.910. The mean of all 15 is 0.745, pulled down by the three failed runs. The median of the five task means is 0.667. Three summaries of the same data differ by almost a quarter of the scale, which is why a paper must say which one it reports.

The median trap with few tasks#

With all 100 seeds, B and C are tied on the mean (0.784 against 0.787) and on the IQM (0.945 against 0.949), but the median of task means says B leads by 9.7 points (0.790 against 0.693). Nothing about the runs is noisy here; the median simply picks a different middle task for each algorithm, the 8×8 grid for B and the slippery grid for C, and ignores the chains where C is ahead. With five tasks, the median is a statement about one task. Agarwal et al. make the same point for Atari: the median is a poor summary when per-task performance is uneven, which it almost always is.

How to put a confidence interval on an aggregate#

Aggregates like the IQM have no convenient closed-form standard error, so we use Efron's bootstrap. The stratified bootstrap respects the structure of the experiment: tasks are fixed, runs within a task are the random part. One bootstrap replicate resamples $N$ runs with replacement within each task, recomputes the aggregate, and repeats this a few thousand times. The 2.5th and 97.5th percentiles of the replicates form a 95% percentile interval.

import numpy as np

def iqm(x):
    x = np.sort(np.ravel(x)); k = len(x) // 4
    return x[k:len(x) - k].mean()

def stratified_bootstrap_ci(scores, stat=iqm, reps=2000, seed=0):
    """scores: array of shape (tasks, runs). Resample runs within each task."""
    rng = np.random.default_rng(seed)
    T, N = scores.shape
    vals = np.empty(reps)
    for b in range(reps):
        idx = rng.integers(0, N, size=(T, N))
        vals[b] = stat(np.take_along_axis(scores, idx, axis=1))
    return np.percentile(vals, [2.5, 97.5])

# scores_B has shape (5, 100) in our experiment
# stratified_bootstrap_ci(scores_B)  ->  approx [0.929, 0.958] for the IQM

A caveat that is easy to miss: bootstrap intervals from very few runs are too narrow. We drew 300 random subsets of B's runs and checked how often the 95% IQM interval contained the 100-seed IQM. Coverage was 84% with 3 seeds per task, 90% with 5 and 94% with 10. The percentile bootstrap cannot see variability that three runs never happened to sample. Agarwal et al. note the same limitation. With fewer than five runs per task, read every interval as optimistic.

What happens with 3, 5, 10 or 20 seeds?#

We resampled the comparison of B and C 3,000 times at each seed count, drawing seeds without replacement from the 100 we had. Recall that their true gap is 0.3 points on the mean and 0.4 on the IQM.

The standard deviation of B's aggregate score tells the same story. For the mean it was 8.3, 6.3, 4.3 and 2.9 points at 3, 5, 10 and 20 seeds. For the IQM it was 7.9, 4.4, 2.5 and 1.5 points. The IQM's advantage grows with the number of seeds here because our run distributions are bimodal: once there are enough runs, the middle half sits inside the "success" mode and stops moving. Meanwhile, the direction of the comparison was a coin flip under every metric and seed count; C "won" on the mean in 51% to 52% of draws. A result like "C beats B by 6 points (3 seeds)" is not a weak finding. It is no finding.

When the true gap is large, few seeds are enough to see it. Comparing A against B, both the mean and the IQM picked B in 99.8% of draws with three seeds. The median of task means picked B in only 81%, because on three of five tasks A and B are close. Few seeds are not always fatal; they are fatal for the small differences most papers report.

Performance profiles: show the distribution, not just a number#

Any single aggregate hides something. The IQM, by construction, ignores the worst quarter of runs, and in our experiment the worst quarter is where the failures live. A performance profile (also called a run-score distribution) shows everything: for each threshold $\tau$, the fraction of all runs whose score exceeds it.

where $1[\cdot ]$ is 1 when the condition holds and 0 otherwise. The idea comes from Dolan and Moré's performance profiles for benchmarking optimisation software. Reading it is simple: a curve that lies above another everywhere is better at every level of ambition. In our runs, 84% of B's and C's runs score above 0.1 against 54% of A's, and 62% to 63% of B's and C's runs score above 0.9 against 34% of A's. The curves for B and C cross and recross; that is what a tie looks like when you draw the whole distribution.

The profile also answers a question the IQM cannot: how often does training fail? In our data, 46% of A's runs and 16% of B's and C's scored below 0.1. If you are deploying an RL-trained policy rather than writing a paper, that failure rate may matter more than any average, and it is the quantity a safety-minded reviewer will ask for.

Probability of improvement#

A different question is how likely a randomly chosen run of one algorithm beats a randomly chosen run of another on the same task. Agarwal et al. call this the probability of improvement:

where $x_{m,i}$ and $y_{m,j}$ are runs of the two algorithms on task $m$, and $S$ is 1 if the first is larger, ½ if they tie and 0 otherwise. It is the Mann-Whitney statistic averaged over tasks, so it does not care about score scale or outliers. For our runs, $P(C>B)=0.502$, a dead heat, and $P(B>A)=0.604$. The second number may look modest given how much better B is on the chains, and that is the point: on three of the five tasks the two are close, and the probability of improvement weighs every task equally. Report it with a bootstrap interval, and treat a lower bound above 0.5 as evidence of a real improvement.

How many seeds do you need?#

Colas, Sigaud and Oudeyer framed seed count as a power calculation. For a two-sided test at level $\alpha$ with power $1-\beta$, detecting a true difference $\delta$ between two algorithms whose per-seed scores have standard deviation $\sigma$ needs roughly

seeds per algorithm, where $z$ values are standard normal quantiles (1.96 and 0.84 for 5% and 80%). Plugging in our own numbers: the per-seed standard deviation of the five-task mean is about 0.14 (8.3 points at 3 seeds times $\sqrt{3}$). Detecting a 5-point difference with 80% power then takes about 125 seeds. The IQM's per-seed spread at 10 seeds was about 0.08, giving roughly 40 seeds. Those numbers are large because our tasks are bimodal, which is typical of exploration-heavy problems; on smooth tasks the requirement is lower. The general lessons hold: halving the detectable difference quadruples the seed count, and outlier-resistant aggregates buy you a smaller $\sigma$.

If you cannot afford the seeds, say so and claim less: report intervals, describe a gap as "not resolved at this budget", and prefer fewer tasks with more seeds over many tasks with three seeds each when the goal is a comparison. For the broader statistics of comparing two systems, including paired tests and multiple comparisons, see statistical significance for evals.

Evaluation protocol details that change results#

Aggregation is only the last step. Several protocol choices move scores as much as the algorithm does, and each should be reported.

Patterson, Neumann, White and White's Empirical Design in Reinforcement Learning covers these at length, and Jordan et al. and Chan et al. propose further measures of reliability across runs, such as dispersion across seeds and risk across runs, worth adding when stability is the claim.

A reporting checklist#

1. Number of seeds per task and algorithm, and how seeds were chosen.
2. Normalisation used, with the reference scores.
3. An aggregate (IQM is a good default) with a 95% stratified bootstrap interval.
4. A performance profile for each method.
5. The failure rate: fraction of runs below a stated low threshold.
6. Probability of improvement for the key comparison, with its interval.
7. Protocol settings from the table above, and the hyperparameter budget.
8. Per-task results in an appendix, so readers can see where differences come from.

The same problem in language-model RL and agent evals#

None of this is specific to gridworlds. RL fine-tuning of language models has the same seed sensitivity, and because runs are expensive, papers often report one. When a training run with verifiable rewards or a reward model gains two points on a benchmark, the first question is how much a second seed of the baseline would have moved. Agent benchmarks raise the episode-level version of the question: an agent's success on a task is itself random, and how you summarise repeated attempts changes the conclusion, as our note on pass@k and pass^k shows, and as the agent trajectory evals study explores for multi-step runs. The overview of AI evals places these within a full evaluation pipeline, and the page on RL environments for agents covers how the tasks themselves are built.

Frequently asked questions#

Sources#

1. R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, M. G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice, 2021. arxiv.org
2. R. Agarwal et al.. rliable: reliable evaluation on reinforcement learning and machine learning benchmarks (library), 2021. github.com
3. P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, D. Meger. Deep Reinforcement Learning that Matters, 2017. arxiv.org
4. C. Colas, O. Sigaud, P.-Y. Oudeyer. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments, 2018. arxiv.org
5. C. Colas, O. Sigaud, P.-Y. Oudeyer. A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms, 2019. arxiv.org
6. M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, M. Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents, 2017. arxiv.org
7. S. C. Y. Chan, S. Fishman, J. Canny, A. Korattikara, S. Guadarrama. Measuring the Reliability of Reinforcement Learning Algorithms, 2019. arxiv.org
8. S. M. Jordan, Y. Chandak, D. Cohen, M. Zhang, P. S. Thomas. Evaluating the Performance of Reinforcement Learning Algorithms, 2020. arxiv.org
9. A. Patterson, S. Neumann, M. White, A. White. Empirical Design in Reinforcement Learning, 2023. arxiv.org
10. E. D. Dolan, J. J. Moré. Benchmarking optimization software with performance profiles, 2002. doi.org
11. B. Efron. Bootstrap Methods: Another Look at the Jackknife, 1979. doi.org
12. R. S. Sutton, A. G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net