Inverse reinforcement learning: inferring the reward behind the behaviour

Inverse reinforcement learning (IRL) infers the reward function an agent is optimising from observations of its behaviour, instead of learning behaviour from a given reward. The catch is that the problem is ill-posed: many different rewards, including the trivial zero reward, explain the same demonstrations perfectly, and they can disagree completely about what to do in situations the demonstrations never covered.

This guide shows that ambiguity concretely in a small gridworld you can explore, then covers how the classic algorithms deal with it (max-margin IRL, feature matching and maximum entropy IRL, with a full derivation), how adversarial methods scale it up, and how IRL connects to the preference-based reward learning used to train language models. All numbers come from seeded numpy runs; the interactive figures run the same code in your browser.

A concrete example: what did the expert want?#

Here is a 7 by 7 grid. There is a goal on the right edge, a wall of water down the middle column with gaps at the top and bottom, and a patch of grass along the bottom. An expert starts in the top-left corner and walks nine steps: right along the top row, through the gap, down and across to the goal. Your job is to say what the expert was optimising, so that a robot can do the same thing from anywhere on the grid.

The reward we used to generate the demonstration is: $-1$ for stepping onto grass, $-5$ for stepping into water, $+10$ for reaching the goal, with discount 0.95. But look at what the demonstration actually shows. It never goes near grass or water, so it is equally well explained by a reward that pays $+10$ at the goal and nothing else. It is also explained by the true reward multiplied by 0.1, by the true reward plus a shaping term that makes every individual number look completely different, and by a reward of zero everywhere, under which every action is optimal and so is the expert's.

All five rewards make every one of the expert's nine actions optimal. Now start the robot in the middle of the left edge, directly across the water from the goal. Under the true reward it detours through the top gap, twelve steps, for a discounted return of 5.69. Under the goal-only reward it walks straight through the water in six steps and earns 3.23 by the true reward. Under the zero reward it has no reason to move at all.

That table is the whole difficulty of IRL. The first three rows are harmless ambiguity: different numbers, same behaviour everywhere. The last two are the dangerous kind: same behaviour where you looked, different behaviour where you did not.

IRL vs RL vs behaviour cloning#

Behaviour cloning treats demonstrations as labelled data and fits $\pi (a\mid s)$ directly. It is simple and often a strong baseline, but small errors take it into states the demonstrator never visited, where its predictions are unreliable and errors compound; Ross, Gordon and Bagnell analysed this and proposed querying the expert on the learner's own states (DAgger) as a fix [12]. IRL's bet is different: the reward is a more compact and transferable description of the task than the policy, so if you recover it you can plan in new states, new start positions or even changed dynamics. Ng and Russell put the premise plainly: reinforcement learning "is founded on the presupposition that the reward function, rather than the policy, is the most succinct, robust, and transferrable definition of the task" [1]. The rest of this article is about when that bet pays off.

Why IRL is ill-posed: reward ambiguity#

Ng and Russell gave the first formal treatment [1]. For a finite MDP with known transitions, a policy $\pi$ is optimal for reward $R$ if and only if the reward satisfies a set of linear inequalities: no action does better than the demonstrated one in any state. Their first observation was uncomfortable: $R=0$ always satisfies them. So do many other rewards. There are three families of ambiguity worth separating.

Scaling. Multiplying $R$ by any positive constant changes no decisions. This one is harmless and usually removed by a norm constraint.

Potential-based shaping. Ng, Harada and Russell proved that adding

to the reward, for any function $\Phi$ of the state, leaves the optimal policy unchanged in every MDP [2]. Here $s$ is the current state, $s^{\prime }$ the next state and $\gamma$ the discount. The intuition is that the shaping terms telescope along any trajectory: the total added reward is $-\Phi (s_0)$ plus a discounted $\Phi$ of the final state, which no choice of actions can influence (with $\Phi =0$ at terminal states). Our shaped candidate uses a random $\Phi$ with standard deviation 4, so its per-step rewards range from about $-16$ to $+15$ and bear no visible resemblance to the true reward, yet its optimal policy matches the true one on all 43 cells. Shaping is the subject of its own guide; for IRL it means that the numbers you recover are only meaningful up to a potential.

Unobserved trade-offs. This is the one that matters. If the demonstrations never face a choice that depends on some feature, they say nothing about how the expert values it. The goal-only reward differs from the truth only on water and grass, which the expert never needed to avoid. Skalse et al. formalise this as partial identifiability: even with infinite data, demonstrations and comparisons determine the reward only up to a set of transformations that depends on the data source, and some of those transformations change the optimal policy in new environments [9]. Cao, Cohen and Szpruch show that in entropy-regularised settings the reward can be pinned down up to a constant if you observe the same expert under two different discount factors or sufficiently different environments [10], which is a precise version of the intuition that diverse demonstrations are what resolve ambiguity.

Classic IRL algorithms#

Max-margin IRL#

Ng and Russell's practical answer to ambiguity was a tie-breaking heuristic: among rewards that make the demonstrated policy optimal, prefer the one that makes it optimal by the largest margin over the next-best action, with a penalty that favours simple, small rewards [1]. That turns IRL into a linear program. It rules out $R=0$ (which has zero margin) but not the goal-only reward, which also has a positive margin on every demonstrated step.

Apprenticeship learning by feature matching#

Abbeel and Ng changed the goal from recovering the true reward to performing as well as the expert under it [3]. Assume the reward is linear in known features, $R(s)=w^{\top }\varphi (s)$, and define a policy's feature expectations $\mu (\pi )=\mathbb{E}[{\sum }_t{\gamma }^t\varphi (s_t)\mid \pi ]$. Then a policy's value is $w^{\top }\mu (\pi )$, and

Here ${\mu }_E$ is the expert's empirical feature expectation and $\tilde{\pi }$ the learner's policy. If you visit the features as often as the expert does, you score the same under every possible weighting, so you do not need to know which one is true. The guarantee is clean but local: it holds for the expert's start-state distribution. From our new start, where the expert never walked, matching feature counts on the demonstration says nothing.

Maximum entropy IRL#

Feature matching has its own ambiguity: many policies, and mixtures of policies, match the same feature counts. Ziebart et al. resolved it with the principle of maximum entropy: among all distributions over trajectories that match the expert's feature counts, choose the one with the most entropy, the one that commits to nothing beyond what the data requires [4]. For deterministic dynamics that distribution has an exponential-family form:

Here $\tau$ is a trajectory, $f_{\tau }$ its total feature count and $Z(w)$ normalises over all trajectories from the same start. Trajectories with higher reward are exponentially more likely, but every trajectory has some probability, so the model expects a noisy, nearly rational expert rather than a perfect one. That one assumption removes the zero-reward solution: $w=0$ makes all trajectories equally likely, which fits a purposeful expert badly.

The weights are fitted by maximum likelihood. Take the log of $P$ for $N$ demonstrations and differentiate:

The gradient of $\log Z$ is the model's expected feature count, a standard exponential-family identity. So the likelihood is concave, and at its maximum the model visits each feature exactly as often as the expert: feature matching falls out as the optimality condition. The expected counts come from a backward pass of soft value iteration, $Q_t(s,a)=R(s^{\prime })+V_{t+1}(s^{\prime })$ and $V_t(s)=\log {\sum }_a\exp Q_t(s,a)$, which defines the stochastic policy ${\pi }_t(a\mid s)=\exp (Q_t(s,a)-V_t(s))$, followed by a forward pass that propagates the start-state distribution through that policy. This is the same soft-optimal policy that appears in KL-regularised RL, with a uniform reference policy.

MaxEnt IRL in action: what the demonstrations can and cannot pin down#

Setup. Same grid, now episodic with horizon 20; reward $w^{\top }\varphi (s^{\prime })$ on entering a cell, with features step (always 1), grass, water and goal; entering the goal ends the episode. The expert is the soft-optimal policy for step $-2$, grass $-3$, water $-12$, goal $+20$. We treat the goal reward as known and learn the other three by gradient ascent on the log-likelihood (learning rate 0.03, 2,000 steps, L2 penalty 0.001). Demonstrations start either in the three top-left cells or anywhere on dry land. We measure how often the learned reward's soft-optimal policy steps into water from the left-edge start, where the expert does so with probability 0.2 percent. Twenty seeds per row.

Mean $\pm$ standard deviation over 20 seeds. Three lessons. First, MaxEnt IRL recovers what the demonstrations exercise: the step cost is right in every row, because every demonstration trades distance against reward. Second, more demonstrations from the same place make you more confident, not more correct. From the corner, the water weight converges tightly to about $-4.5$, which is merely "bad enough not to wander into from up here". Without the small L2 penalty it would keep drifting downward without converging, because the data only says the expert avoided water, not by how much. Third, diversity fixes it. Demonstrations from anywhere include states where water is genuinely tempting; across 200 of them the expert stepped into water only 1.6 times on average, yet the many near-misses where it chose to walk around were enough to push the water weight to about $-9$ and cut the new-start water rate from 66 percent to 2.6 percent. The remaining gap from $-12$ reflects how little information there is about a cost the expert almost never pays.

Bayesian, deep and adversarial IRL#

Bayesian IRL makes the ambiguity explicit instead of breaking ties: put a prior on rewards, model the expert as Boltzmann-rational, and sample from the posterior with MCMC [5]. The posterior is wide exactly where our corner-only demonstrations are silent, which is the information a cautious planner needs.

To scale beyond tabular problems, Finn, Levine and Abbeel's guided cost learning learns a neural-network cost with MaxEnt IRL, estimating the partition function by sampling from a policy that is optimised alongside the cost [6]. Ho and Ermon's generative adversarial imitation learning (GAIL) then showed that MaxEnt IRL followed by RL is equivalent to matching the expert's state-action occupancy measure, and that this can be done directly with a discriminator, skipping the reward [7]. GAIL is an imitation method: it produces a policy, not a transferable reward. Fu, Luo and Levine's adversarial IRL (AIRL) restores the reward by constraining the discriminator's form so that the learned reward is disentangled from the dynamics, and showed that such rewards transfer when the environment changes, where GAIL policies do not [8]. AIRL's structure, a state-only reward plus a learned shaping term, is a direct engineering response to the shaping ambiguity above.

From IRL to reward learning from preferences#

Demonstrations are one kind of evidence about a reward. Comparisons are another: show a person two trajectories and ask which is better. Christiano et al. trained deep RL agents on a reward model fitted to such comparisons, using a Bradley-Terry likelihood that has the same Boltzmann form as MaxEnt IRL, applied to pairs instead of all trajectories [11]. Comparisons have a practical advantage over demonstrations: people can judge behaviour they could not perform. Jeon, Milli and Dragan's reward-rational choice framework puts demonstrations, comparisons, corrections and even turning the robot off into one model, each read as a noisily rational choice among options [13]. That lineage leads directly to reward models for language models and the RLHF pipeline. The identifiability lessons carry over: a reward model trained on comparisons is only informative about the kinds of responses it was shown, which is one root of the overoptimisation pattern in our research note, and its labels are noisy choices, as analysed in our note on preference label noise.

Why IRL matters for AI safety#

Learning values from behaviour is one of the oldest proposals for aligning AI systems: rather than writing down a reward that will be gamed (see specification gaming), infer what people want from what they do. Hadfield-Menell et al.'s cooperative IRL frames this as a game in which the robot is uncertain about the human's reward and acts to learn it, which makes behaviours such as asking and deferring rational [14]. Two results temper the optimism. The first is the one our gridworld shows: rewards inferred from limited behaviour can be confidently wrong about situations the behaviour never covered, and the policy will act on them. The second is deeper. Armstrong and Mindermann show that you cannot uniquely decompose behaviour into a reward and a planning algorithm: a human who acts suboptimally could have a simple reward and a bad planner, or a strange reward and a perfect one, and no amount of observation separates the two without normative assumptions about human rationality [15]. Every IRL method makes such an assumption, Boltzmann rationality being the usual one, and it is exactly where systematic human biases leak into the inferred values.

In practice: a checklist#

A compact MaxEnt IRL for deterministic tabular problems, the same algorithm as the figure without episode termination:

import numpy as np

def maxent_irl(nxt, phi, trajs, T, lr=0.03, iters=2000, l2=1e-3):
    """MaxEnt IRL for deterministic dynamics.
    nxt[s, a]: next state; phi[s]: feature vector of state s (reward r = phi @ w on entering s);
    trajs: list of state sequences, all of length T + 1."""
    S, A = nxt.shape
    mu_demo = np.mean([phi[tr[1:]].sum(0) for tr in trajs], 0)   # empirical feature counts
    d0 = np.bincount([tr[0] for tr in trajs], minlength=S) / len(trajs)
    w = np.zeros(phi.shape[1])
    for _ in range(iters):
        r, V, pis = phi @ w, np.zeros(S), []
        for t in range(T):                       # backward pass: soft value iteration
            Q = r[nxt] + V[nxt]
            V = np.logaddexp.reduce(Q, axis=1)
            pis.append(np.exp(Q - V[:, None]))
        d, mu_model = d0, np.zeros_like(w)
        for pi in reversed(pis):                 # forward pass: expected visitation
            d = np.bincount(nxt.ravel(), (d[:, None] * pi).ravel(), minlength=S)
            mu_model += d @ phi
        w += lr * (mu_demo - mu_model - l2 * w)  # log-likelihood gradient
    return w

# tiny check: a 5-state chain where the expert likes state 4
nxt = np.array([[max(s - 1, 0), min(s + 1, 4), s] for s in range(5)])
phi = np.eye(5)[:, [3, 4]]                       # features: "in state 3", "in state 4"
trajs = [[0, 1, 2, 3, 4, 4, 4]] * 20 + [[0, 1, 2, 3, 4, 4, 3]] * 2
print(maxent_irl(nxt, phi, trajs, T=6).round(2))  # [1.51 4.29]

Failure modes#

Mistaking consistency for correctness. Every row of our first table fits the demonstration perfectly. A reward that explains the data is a hypothesis, not a finding.

Reading meaning into raw reward values. Because of shaping and scaling, a learned reward of $+3$ in one state and $-2$ in another says little by itself. Compare rewards by the policies they induce, or use shaping-invariant distances.

Confident extrapolation. Tight error bars on a weight learned from narrow demonstrations (our $-4.52\pm 0.02$) measure sampling noise, not identifiability.

Wrong rationality model. An expert who is systematically myopic or risk-averse will be read as having a different reward. No algorithm detects this from behaviour alone [15].

Feature misspecification. If what the expert cares about is not in $\varphi$, MaxEnt IRL will attribute their behaviour to whatever features are present.

What the toy does not show#

The grid has 49 states, known deterministic dynamics and a linear reward in four hand-chosen features, and the expert really is Boltzmann-rational for a reward in that family, so MaxEnt IRL is perfectly specified. Real IRL has unknown or stochastic dynamics, high-dimensional states, neural reward models and human experts whose departures from the model are systematic. Those make every problem shown here worse, not better.

Frequently asked questions#

Sources#

1. Andrew Y. Ng, Stuart Russell. Algorithms for Inverse Reinforcement Learning, 2000. aima.eecs.berkeley.edu
2. Andrew Y. Ng, Daishi Harada, Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping, 1999. people.eecs.berkeley.edu
3. Pieter Abbeel, Andrew Y. Ng. Apprenticeship Learning via Inverse Reinforcement Learning, 2004. mlanthology.org
4. Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, Anind K. Dey. Maximum Entropy Inverse Reinforcement Learning, 2008. cdn.aaai.org
5. Deepak Ramachandran, Eyal Amir. Bayesian Inverse Reinforcement Learning, 2007. ijcai.org
6. Chelsea Finn, Sergey Levine, Pieter Abbeel. Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization, 2016. arxiv.org
7. Jonathan Ho, Stefano Ermon. Generative Adversarial Imitation Learning, 2016. arxiv.org
8. Justin Fu, Katie Luo, Sergey Levine. Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, 2017. arxiv.org
9. Joar Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, Adam Gleave. Invariance in Policy Optimisation and Partial Identifiability in Reward Learning, 2022. arxiv.org
10. Haoyang Cao, Samuel N. Cohen, Lukasz Szpruch. Identifiability in inverse reinforcement learning, 2021. arxiv.org
11. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei. Deep reinforcement learning from human preferences, 2017. arxiv.org
12. Stephane Ross, Geoffrey J. Gordon, J. Andrew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2010. arxiv.org
13. Hong Jun Jeon, Smitha Milli, Anca D. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning, 2020. arxiv.org
14. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell. Cooperative Inverse Reinforcement Learning, 2016. arxiv.org
15. Stuart Armstrong, Sören Mindermann. Occam's razor is insufficient to infer the preferences of irrational agents, 2017. arxiv.org