Imitation learning: Definition and Examples

Imitation learning is the family of methods that train a policy from demonstrations of an expert instead of from a reward signal. The goal is to act like the demonstrator, either by copying its actions directly or by recovering the objective it seems to be pursuing and optimising that.

Main approaches

Behaviour cloning: supervised learning from states to the expert's actions. Simple and often effective, but sensitive to compounding errors.
Interactive imitation (DAgger): run the learner, ask the expert to label the states the learner actually visits, aggregate the data, and retrain.
Inverse reinforcement learning: infer a reward from the demonstrations, then run RL on it.
Adversarial imitation: train a discriminator to tell expert from learner behaviour, and use it as a reward, so the learner matches the expert's state-action distribution.

The compounding error problem

A cloned policy is trained only on states the expert visited. Once it makes a small mistake it drifts into states it has never seen, where it is more likely to make further mistakes. Ross and Bagnell (2010) showed that a cloned policy with per-step error rate $ϵ$ over a horizon of $T$ steps can incur expected cost that grows with $T^{2} ϵ$ , while interactive methods such as DAgger achieve a bound linear in $T$ :

J (π^_{BC}) \leq J (π^{*}) + T^{2} ϵ, J (π^_{DAgger}) \leq J (π^{*}) + O (T ϵ),

where $J$ here is the expected total cost over the horizon (lower is better) and $π^{*}$ is the expert policy. Both bounds are simplified here: the original results carry constants and, for DAgger, assumptions about how errors translate into cost.

Worked example

With $ϵ = 0.01$ and $T = 50$ , the behaviour cloning bound allows an extra cost of $5 0^{2} \times 0.01 = 25$ , while a linear bound allows about $50 \times 0.01 = 0.5$ . The gap widens with the horizon, which is why cloning works well for short tasks and struggles with long ones unless the data includes recoveries from mistakes.

Imitation versus reinforcement learning

Imitation needs no reward design and is far more sample efficient, but it cannot exceed the demonstrator and inherits its mistakes. Reinforcement learning can exceed the demonstrator but needs a reward, which opens the door to specification gaming. Many practical systems do both: pretrain by imitation, then fine-tune with RL.

Why it matters

Supervised fine-tuning of language models on human-written responses is imitation learning, and pretraining on text is imitation at scale. That framing explains familiar behaviour: models copy demonstrators' habits, including errors, and can degrade on long agentic tasks where their own outputs push them into situations no demonstration covered.

Sources

Stéphane Ross, J. Andrew Bagnell. Efficient Reductions for Imitation Learning, 2010. proceedings.mlr.press
Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2010. arxiv.org
Jonathan Ho, Stefano Ermon. Generative Adversarial Imitation Learning, 2016. arxiv.org