Inverse reinforcement learning: Definition and Examples

Inverse reinforcement learning (IRL) runs reinforcement learning backwards: given demonstrations of an expert acting in an environment, it infers a reward function under which that behaviour is optimal or near-optimal. The learned reward can then be optimised to produce a policy, or inspected to understand what the expert was trying to achieve.

The problem and its ambiguity

Standard RL maps a reward $R$ to a policy. IRL takes a Markov decision process without a reward, plus expert trajectories, and searches for $R$ . Ng and Russell (2000) pointed out that the problem is ill-posed: the zero reward makes every policy optimal, and many different rewards explain the same behaviour. Practical IRL methods therefore add a principle to choose among them, such as maximising the margin between the expert and alternatives, or maximum entropy.

Maximum entropy IRL

Ziebart and colleagues (2008) model the expert as choosing trajectories with probability exponential in their reward:

P (τ ∣ θ) = \frac{exp ( R _{θ} ( τ ) )}{Z ( θ )}, R_{θ} (τ) = t \sum θ^{⊤} ϕ (s_{t}),

where $ϕ (s)$ is a feature vector of state $s$ , $θ$ holds the reward weights, and $Z (θ)$ normalises over all trajectories. Fitting $θ$ by maximum likelihood has a clean gradient: the expert's average feature counts minus the feature counts expected under the current model. Learning stops when the model visits features as often as the expert does.

Worked example

Two candidate routes have rewards 2 and 1 under the current $θ$ . The model predicts the first with probability $e^{2} / (e^{2} + e^{1}) \approx 0.73$ . If the expert takes the first route 90 percent of the time, the gradient increases the weight on features that distinguish it, until the predicted share matches 0.9.

Relation to other methods

Behaviour cloning copies actions directly and needs no reward, but generalises poorly outside the demonstrated states.
Adversarial imitation methods learn a discriminator that acts as an implicit reward, closely related to IRL.
Reward modelling from preferences, as in RLHF, is a cousin of IRL: it infers a reward from human comparisons instead of demonstrations, using a similar exponential choice model (the Bradley-Terry model).

Why it matters

IRL is one of the oldest formal answers to the alignment question of how a system can learn what people want from what they do. Its central lesson carries over directly to reward models for language models: behaviour underdetermines intent, so any inferred reward embeds assumptions about how rational and consistent the demonstrator was, and optimising that reward hard exposes where those assumptions fail.

Sources

Andrew Y. Ng, Stuart Russell. Algorithms for Inverse Reinforcement Learning, 2000. people.eecs.berkeley.edu
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, Anind K. Dey. Maximum Entropy Inverse Reinforcement Learning, 2008. cdn.aaai.org