Offline reinforcement learning: learning from logged data without falling off the cliff

Offline reinforcement learning (also called batch RL) learns a decision-making policy entirely from a fixed dataset of logged interactions, with no further trial and error in the environment. The central difficulty is distribution shift: a learned value function is only trustworthy for actions that appear in the data, and the maximisation at the heart of RL actively seeks out the actions where it is wrong.

This guide explains why that happens, using a toy problem small enough to solve exactly and an interactive figure you can break yourself. It then walks through the three families of fixes (behaviour regularisation, conservatism and in-sample learning), how to evaluate an offline policy before deploying it, and why the same ideas show up in preference-based training of language models. Every number below comes from a seeded numpy simulation that we ran; the figure runs the same code in your browser.

A concrete example: the cliff the data never saw#

Imagine a controller with one knob, an action $a$ between $-1$ and $1$. The best setting is $a=0.4$, which earns a reward of 1 per step. Push the knob past $\pm 0.75$ and you fall off a cliff: the reward drops linearly, to about $-0.75$ at the extremes. The task repeats forever with discount factor $\gamma =0.9$, so the best achievable value is $1/(1-0.9)=10$.

You do not get to experiment. You get a log of 200 past decisions made by a cautious operator who kept the knob between $-0.5$ and $0.1$, with the reward observed each time (plus a little noise). The operator never tried the good setting and never went near the cliff. You fit a smooth model of $Q(a)$, the long-run value of each knob setting, to this log with fitted Q iteration (the batch version of Q-learning), then pick the setting with the highest $Q$.

What happens, for the dataset with seed 1: the fitted model is accurate inside the logged range, but a cubic curve has to do something outside it, and here it bends upwards at $a=-1$. The greedy policy picks $a=-1$, the critic reports a value of 20.3, twice the true optimum, and the policy actually earns $-7.5$. It drove off the cliff with full confidence. Across 50 random datasets of the same shape, naive fitted Q iteration chose an action outside the logged range every single time, landed in the cliff zone in 49 of them and ended up with a negative value in 47, while its median value estimate was 19.2. The operator it was supposed to improve on earned 1.7.

Nothing went wrong with the regression; it went wrong with the max.

Offline RL vs online RL, off-policy RL and imitation learning#

In ordinary reinforcement learning the agent acts, sees what happens and corrects itself. If its value function overrates an action, it tries the action, gets a low reward and the error disappears. Offline RL removes that feedback loop. Levine, Kumar, Tucker and Fu's tutorial frames this as the main reason offline RL is hard and the main reason it is worth doing: in robotics, healthcare, recommendation and dialogue, exploration is expensive or unsafe, but logs are plentiful [1].

The difference between off-policy and offline is subtler than it looks. Off-policy algorithms can in principle learn from any data, but Fujimoto, Meger and Precup showed that standard off-policy deep RL agents fail when trained purely on a fixed batch, even a batch generated by an expert, because nothing ever corrects their value estimates for actions the batch does not contain. They named the effect extrapolation error [2]. Behaviour cloning, at the other end, is perfectly stable offline but cannot use the reward to do better than the data. Offline RL sits between them: it uses rewards to improve on the behaviour policy while staying close enough to the data that its value estimates mean something. If you only have demonstrations and no rewards, the relevant tools are imitation and inverse reinforcement learning instead.

Why distribution shift breaks Q-learning#

Fitted Q iteration repeats one regression. Given the current estimate ${\hat{Q}}_k$, compute a target for every logged transition and fit ${\hat{Q}}_{k+1}$ to those targets:

Here $(s_i,a_i,r_i,s_i^{\prime })$ is the $i$-th logged transition, $\gamma$ is the discount factor and the maximum runs over every action available in the next state $s_i^{\prime }$. This is the sample version of the Bellman optimality equation, and it is exactly the update Ernst, Geurts and Wehenkel studied as batch-mode RL two decades ago [11].

The regression only ever sees $Q$ at logged pairs $(s_i,a_i)$. The target, though, asks for $Q$ at the maximising action $a^{\prime }$, which can be any action at all. Two things then compound. First, selection: if the estimates at unseen actions are noisy, the maximum picks out the ones with the most positive error, because the expected maximum of noisy estimates exceeds the maximum of their expectations. Second, bootstrapping: the inflated value becomes part of the next target, so the error is passed back through every state that can reach it. In our one-state example the fixed point of fitted Q iteration is $\hat{V}={\max }_a\hat{r}(a)/(1-\gamma )$, so a reward overestimate of 2.03 at $a=-1$ becomes a value overestimate of 20.3 once multiplied by $1/(1-\gamma )=10$.

Online, the next rollout punishes this optimism. Offline, there is no next rollout. Kumar et al. call the same mechanism bootstrapping error and show that it accumulates through the backups rather than averaging out [3].

See it happen: dataset coverage vs extrapolation error#

Setup. True reward $r(a)=e^{-((a-0.4)/0.4{)}^2}-3\max (0,|a|-0.75)$; behaviour actions uniform on $[-0.2-w,-0.2+w]$, clipped to $[-1,1]$; 200 transitions with reward noise SD 0.1; critic $\hat{Q}$ linear in the first three Legendre polynomials plus an intercept; $\gamma =0.9$. The coverage half-width $w$ controls how much of the action space the log covers. We compare five learners (formulas in the next section): naive fitted Q iteration, a support-constrained maximum in the spirit of BCQ (only actions within 0.05 of a logged action are eligible), a CQL-style penalty ($\alpha =0.1$), an IQL-style expectile learner ($\tau =0.9$, advantage weighting $\beta =10$), and behaviour cloning. For each we record what its own critic claims the policy is worth and what the policy really earns. Seeds 1 to 50 for every coverage level; this is a simulation, not a benchmark.

The full sweep, averaged over 50 datasets per coverage level:

Four things are worth noticing. The naive critic's claimed value is worst exactly when its policy is worst: overestimation and failure are the same event, so you cannot use the critic to catch it. With full coverage ($w=1$) the naive method is excellent; offline RL is not hard because of the data size but because of the data's support. The support-constrained learner can slightly beat the best logged action at low coverage because it may step 0.05 beyond the data, and it underperforms at $w=0.8$ because the cubic critic is misspecified inside the support, a reminder that staying in-distribution does not fix a bad function class. And behaviour cloning is safe but stuck at the operator's level, around 2 to 3, because the operator was mediocre.

How offline RL algorithms fix it#

Every serious offline RL method is a way of keeping the maximisation away from actions the data cannot vouch for. They differ in where the constraint lives.

Behaviour regularisation and policy constraints#

The most direct fix constrains the policy. BCQ trains a generative model of the behaviour policy and only lets the learned policy choose among (slightly perturbed) actions that the model considers likely, so the maximisation in the target never reaches far outside the data [2]. BEAR relaxes this from matching the behaviour distribution to staying within its support, using a kernel distance between action samples [3]. Wu, Tucker and Nachum's behaviour-regularised actor-critic framework compared many of these variants and found that much of the added machinery was unnecessary for good performance [4]. TD3+BC took that to its conclusion. It adds one behaviour cloning term to an ordinary actor update:

Here $\pi (s)$ is the deterministic policy's action, $a$ is the logged action in the same state, and $\lambda$ normalises the value term by the average magnitude of $Q$ so a single $\alpha$ (2.5 in the paper) works across tasks. The policy is pulled towards higher value but pays a squared penalty for straying from what was logged. With this and state normalisation, Fujimoto and Gu matched the state of the art of the time while more than halving run time [7]. Policy constraints are easy to reason about, but their quality is capped by how well you can estimate the behaviour policy, and in high-dimensional action spaces that estimate is itself a hard problem. Our support-max learner is the idealised version with a perfect support estimate, which is why it looks so good in one dimension.

Conservatism: Conservative Q-Learning#

CQL moves the constraint from the policy to the critic. It adds a penalty that pushes $Q$ down on actions the current policy (or a soft maximum over all actions) favours, and up on actions in the data [5]:

The log-sum-exp term is a soft version of ${\max }_{a}Q(s,a)$, the second term inside the bracket is the average $Q$ on logged actions, $\hat{\mathcal{B}}\hat{Q}$ is the usual Bellman target and $\alpha$ sets the strength. Intuitively: any action whose value is propped up only by extrapolation gets dragged down, while logged actions are anchored by real rewards. Kumar et al. prove that with an appropriate $\alpha$, the learned $Q$ lower-bounds the true value of the policy in expectation, so policy improvement on it is pessimistic rather than optimistic [5]. You can see the shape in Figure 1: the CQL critic is roughly correct inside the data and falls off steeply outside it.

The price is a hyperparameter that matters a great deal. In our toy, at coverage 0.6:

Too little conservatism lets some datasets through to the cliff; too much and the policy hugs the behaviour mean and gives back most of the possible improvement. The value lower bound is a statement about tabular or well-specified critics; with our misspecified cubic critic the CQL estimates were still somewhat optimistic about the actions it chose (3.7 claimed against 2.0 earned for seed 1 at $w=0.3$), just far less so than the naive critic.

In-sample learning: Implicit Q-Learning#

IQL avoids the problem at its root by never evaluating an action outside the dataset [6]. Instead of a maximum over actions, it estimates the value of the best actions in the data with an upper expectile:

$V(s)$ is a state value, $\tau \in (0.5,1)$ is the expectile level and $L_2^{\tau }$ is an asymmetric squared loss that penalises underestimates more than overestimates, so $V$ settles near the upper end of the values of logged actions; as $\tau \to 1$ it approaches the maximum over actions that appear in the data. The policy is then extracted by advantage-weighted regression: copy logged actions, weighted by $\exp (\beta (Q(s,a)-V(s)))$. In the toy, IQL never chose the cliff at any coverage level and reached 9.7 at $w=0.6$. Its cost shows at full coverage, where it earned 8.6 against CQL's 10.0: weighting logged actions only averages over good ones, it cannot put all its mass on the single best one unless $\beta$ is very large.

Other approaches#

Two more ideas are worth knowing. With a large and diverse enough dataset, plain off-policy methods can work: Agarwal, Schuurmans and Norouzi trained agents on the full replay logs of an Atari agent and found that recent off-policy algorithms trained only on the logs outperformed the agent that produced them [9]. This is our $w\ge 1$ regime: when coverage is good, extrapolation error is small. Decision Transformer recasts offline RL as sequence modelling, predicting actions conditioned on a desired return [10]; it is simple and stable, but it learns from trajectory-level patterns and is less able to stitch together the good halves of different trajectories, which value-based methods do naturally.

How do you evaluate an offline RL policy?#

The awkward truth is that the reason you are doing offline RL (you cannot run the policy) is also the reason you cannot easily check it. Three practices help.

Do not trust the training critic. Figure 1 makes the point: the critic that produced the policy is the most biased judge of it. Use off-policy evaluation (OPE) with a separately fitted estimator, such as fitted Q evaluation of the fixed final policy, importance sampling when the behaviour policy's probabilities are logged, or doubly robust combinations of the two. Each has known failure modes, high variance for importance sampling over long horizons and the same support problem for model-based estimates.

Select hyperparameters offline, carefully. Many published results tune hyperparameters by running policies in the simulator, which quietly breaks the offline assumption. Paine et al. found that offline RL algorithms are sensitive to hyperparameter choices, that the choice of algorithm and Q estimator strongly affects how well offline selection works, and that with those factors controlled, policies can be ranked reliably from logged data alone [12]. Our CQL table above is a small instance of the first finding.

Use standard datasets for method comparisons. The D4RL suite provides fixed datasets with deliberately varied properties: narrow expert data, noisy data, mixtures and data that require stitching [8]. Results on one data regime rarely transfer to another. Report results with seeds and intervals, as in our guide to evaluating RL agents.

Where some online interaction is possible, a common pattern is offline pretraining followed by careful online fine-tuning; methods such as AWAC were designed for that hand-off [13].

Offline RL and language models#

Preference-based fine-tuning of language models has an offline flavour. A preference dataset is a fixed log: prompts, pairs of responses sampled from some earlier model, and a human or model judgment of which is better. Direct preference optimisation trains the policy on that fixed dataset without sampling new responses during training [14], which makes it closer to offline RL than to the online loop of classic RLHF. The same coverage question applies: the dataset only says something about responses that resemble the ones that were compared, and the KL term towards a reference model (see KL-regularised RL) plays a role similar to behaviour regularisation, keeping the policy near the region where the data is informative. When a reward model is trained offline and then optimised online, its errors on unfamiliar responses are what the policy finds, which is the overoptimisation pattern analysed in our research note on reward model overoptimization. And because the log is fixed, its label quality caps everything downstream, as discussed in our note on preference label noise. The wider picture is in reinforcement learning for LLMs.

In practice: a checklist#

The core of the toy fits in a few lines. This reproduces the naive and support-constrained fixed points for one dataset:

import numpy as np

rng = np.random.default_rng(0)
gamma, grid = 0.9, np.linspace(-1, 1, 201)
r_true = lambda a: np.exp(-((a - 0.4) / 0.4) ** 2) - 3 * np.maximum(0, np.abs(a) - 0.75)
feats = lambda a: np.stack([np.ones_like(a), a, a**2, a**3], -1)

a = rng.uniform(-0.5, 0.1, 200)                      # the logged actions
r = r_true(a) + 0.1 * rng.standard_normal(200)        # the logged rewards
theta = np.linalg.lstsq(feats(a), r, rcond=None)[0]   # critic's reward model
r_hat = feats(grid) @ theta

# fixed point of fitted Q iteration in a one-state MDP: Q(a) = r_hat(a) + gamma * V
for name, allowed in [("naive", np.ones_like(grid, bool)),
                      ("in-support", np.abs(grid[:, None] - a).min(1) <= 0.05)]:
    k = np.flatnonzero(allowed)[r_hat[allowed].argmax()]
    print(f"{name:10s} action {grid[k]:+.2f}  claims {r_hat[k] / (1 - gamma):6.2f}"
          f"  earns {r_true(grid[k]) / (1 - gamma):6.2f}")

Failure modes#

Trusting the critic's value estimate. In the toy, the naive critic's estimate was highest exactly when its policy was worst. Overestimation is the failure, not a symptom you can monitor with the same model.

Confusing data volume with coverage. Doubling the log from the same cautious operator adds no information about the cliff. Coverage of the relevant actions matters more than size.

Over-conservatism. A large penalty or tight constraint is safe and useless: CQL with $\alpha =1$ earned 3.3 at coverage 0.6, where 10 was available inside the data.

Misspecified critics inside the support. Constraining the policy to the data does not fix a function class that cannot fit the data, as the support-max learner's drop at $w=0.8$ shows.

Confounded logs. If the behaviour policy acted on information that is not in the logged state (a clinician who saw the patient, an operator who saw the weather), the data's action-outcome relationships are biased in ways no offline RL algorithm in this article corrects.

Tuning online and calling it offline. Hyperparameters selected by running policies in the environment produce optimistic results that will not hold when you truly cannot run them [12].

What the toy does not show#

Our example has one state, one continuous action and a four-parameter critic, so every failure is visible. Real offline RL has many states, neural critics that extrapolate less predictably than a cubic, and long horizons over which bootstrapping error compounds across the state space. The CQL and IQL learners here use the papers' objectives in their simplest form (a uniform grid for the log-sum-exp, a single expectile), not the full algorithms. Treat the toy as an illustration of mechanisms; for algorithm rankings, the D4RL literature is the place to look [8].

Frequently asked questions#

Sources#

1. Sergey Levine, Aviral Kumar, George Tucker, Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020. arxiv.org
2. Scott Fujimoto, David Meger, Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration, 2018. arxiv.org
3. Aviral Kumar, Justin Fu, George Tucker, Sergey Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, 2019. arxiv.org
4. Yifan Wu, George Tucker, Ofir Nachum. Behavior Regularized Offline Reinforcement Learning, 2019. arxiv.org
5. Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning, 2020. arxiv.org
6. Ilya Kostrikov, Ashvin Nair, Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning, 2021. arxiv.org
7. Scott Fujimoto, Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement Learning, 2021. arxiv.org
8. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, 2020. arxiv.org
9. Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi. An Optimistic Perspective on Offline Reinforcement Learning, 2019. arxiv.org
10. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling, 2021. arxiv.org
11. Damien Ernst, Pierre Geurts, Louis Wehenkel. Tree-Based Batch Mode Reinforcement Learning, 2005. jmlr.org
12. Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, Nando de Freitas. Hyperparameter Selection for Offline Reinforcement Learning, 2020. arxiv.org
13. Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, 2020. arxiv.org
14. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. arxiv.org