Offline reinforcement learning: Definition and Examples

Offline reinforcement learning (also called batch RL) learns a policy from a fixed dataset of previously collected transitions, without interacting with the environment during training. It promises RL for settings where exploration is expensive or dangerous, such as medicine, driving or industrial control, but it must cope with the fact that the data only covers what the old behaviour policy did.

The setup

The learner gets a dataset $D = {(s_{i}, a_{i}, r_{i}, s_{i}^{'})}$ collected by one or more unknown behaviour policies $μ$ . The goal is a policy $π$ with high return in the real environment. Unlike imitation learning, the data may be mediocre and the learner should do better than it; unlike online off-policy RL, it can never collect new data to check its guesses.

Why naive off-policy methods fail

A standard Q-learning target, $r + γ max_{a^{'}} Q (s^{'}, a^{'})$ , queries actions that may never appear in the data. Their Q-values are pure extrapolation, and the max picks out whichever one is most overestimated. The error then propagates through bootstrapping, and the policy heads straight for the actions the data says least about.

Worked example

In some state the dataset contains only action $a_{1}$ , whose true and estimated value is 3. The network has never been trained on $a_{2}$ and happens to output 5 for it, while its true value is 0. A greedy policy picks $a_{2}$ , and the backed-up target for the previous state inherits the phantom value of 5. Online, one try would reveal the mistake; offline, nothing ever corrects it.

Main families of methods

Policy constraints: keep $π$ close to the behaviour policy, for example with a KL or support constraint, or with a behaviour cloning term in the loss.
Conservative value estimates: conservative Q-learning (CQL) adds a penalty that pushes down Q-values on actions sampled from the learned policy and pushes up those in the data, yielding a lower bound on value:

Q min α (E_{s \sim D, a \sim π} [Q (s, a)] - E_{(s, a) \sim D} [Q (s, a)]) + \frac{1}{2} E_{D} [(Q - B Q^)^{2}],

where $α$ sets the strength of the penalty and $B Q^$ is the usual Bellman target.

In-sample learning: methods such as implicit Q-learning avoid querying unseen actions at all.
Model-based: learn a dynamics model and penalise rollouts where the model is uncertain.

Why it matters

Most high-stakes domains have logs but cannot tolerate trial and error, so offline RL is often the only realistic way to apply RL there. It also forces a hard question that matters for evaluation: how do you estimate a new policy's performance from data it did not generate? Off-policy evaluation with importance sampling is the usual answer, and its uncertainty is usually larger than people assume.

Sources

Sergey Levine, Aviral Kumar, George Tucker, Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020. arxiv.org
Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning, 2020. arxiv.org