Behaviour cloning is the simplest form of imitation learning: collect states paired with the actions an expert took, then fit a policy to predict those actions with ordinary supervised learning. There is no reward, no environment interaction during training and no planning.
The objective
Given a dataset of expert demonstrations, behaviour cloning maximises the log-likelihood of the expert's actions:
For discrete actions this is cross-entropy classification; for continuous actions with a Gaussian policy it reduces to mean squared error regression. Supervised fine-tuning of a language model is exactly this objective, with prompts and earlier tokens as the state and the next token as the action.
An early example
ALVINN (1988) trained a small neural network to steer a vehicle from camera images by imitating a human driver. Its author found that a network trained only on good driving did not know how to recover once it drifted off-centre, because the human never drifted, and addressed it by synthesising shifted views labelled with corrective steering. That fix anticipates the main weakness of the method.
Worked example: compounding errors
Suppose a cloned policy matches the expert at each step with probability 0.99 and errors are independent. Over a 100-step task, the chance of a perfect run is . Worse, after the first error the policy is in a state the expert never demonstrated, so its per-step accuracy there is typically lower than 0.99, and errors cluster. This distribution shift between training states and visited states is why the error of behaviour cloning can grow with the square of the horizon.
Remedies
- Collect corrective data on the learner's own states, as in DAgger.
- Add noise to the expert during data collection so demonstrations include recoveries.
- Use expressive policy classes (mixtures, diffusion or transformer policies) when demonstrations are multimodal; a unimodal regressor averaging "swerve left" and "swerve right" goes straight.
- Follow with RL, or use cloning as a regulariser inside offline RL.
Why it matters
Behaviour cloning is cheap, stable and frequently a strong baseline that more complex methods fail to beat. It is also the first stage of nearly every language model post-training pipeline. Knowing its failure mode, confident behaviour in states the data never covered, tells you where to aim evaluations: long horizons, recovery from the model's own mistakes, and inputs unlike the demonstrations.
Related terms
- Imitation learning
- Offline reinforcement learning
- Inverse reinforcement learning
- Policy
- RLHF (reinforcement learning from human feedback)
Sources
- Dean A. Pomerleau. ALVINN: An Autonomous Land Vehicle in a Neural Network, 1988. proceedings.neurips.cc
- Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2010. arxiv.org