Model-based reinforcement learning uses a model of the environment, a function that predicts the next state and reward from the current state and action, to plan or to generate extra training experience. Model-free methods such as Q-learning skip the model and learn values or policies directly from real interactions.
What the model is
A learned model approximates the transition and reward functions of the MDP:
It is fitted by supervised learning on observed transitions . Some methods predict raw observations; others, such as latent world models, predict in a learned compact state space and only need to predict what matters for reward and value.
Three ways to use a model
- Planning at decision time: search ahead through the model before each action, as in model predictive control or tree search. A learned model that predicts only rewards, values and policies was enough to plan at superhuman level in board games and Atari.
- Generating synthetic experience: Dyna-style methods mix real transitions with imagined ones from the model to train a model-free learner, multiplying the value of each real sample.
- Training in imagination: world-model agents roll out entire trajectories inside the model and train an actor and critic on them.
The compounding-error problem
Model errors accumulate over a rollout. If a model's one-step prediction is "good enough" 99% of the time, the chance that a 50-step rollout contains no bad step is . Worse, a policy optimised against the model will seek out states where the model is wrong in its favour, a form of exploitation closely related to reward hacking. Practical methods use short rollouts branched from real states, ensembles of models to estimate uncertainty, and penalties for leaving the data the model was trained on.
Why it matters
The main payoff is sample efficiency: a model lets an agent learn from far fewer real interactions, which matters when interactions are slow, expensive or risky, as in robotics or medicine. Models also support safe exploration, since dangerous actions can be tried in simulation first. The cost is that the policy is only as trustworthy as the model in the states it actually visits, so evaluating model error where the policy goes, not just on average, is part of evaluating the agent.
Related terms
Sources
- Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting, 1991. doi.org
- Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine. When to Trust Your Model: Model-Based Policy Optimization, 2019. arxiv.org
- Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, et al.. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, 2019. arxiv.org