On-policy vs off-policy

An on-policy method learns the value or improvement of the same policy that is generating its experience. An off-policy method learns about a target policy $\pi$ using data collected by a different behaviour policy $b$, such as an older version of the agent, an exploratory policy or a human demonstrator.

Canonical examples

• SARSA (on-policy) updates toward $R+\gamma Q(S^{\prime },A^{\prime })$, where $A^{\prime }$ is the action the agent actually takes next, exploration included.
• Q-learning (off-policy) updates toward $R+\gamma {\max }_{a^{\prime }}Q(S^{\prime },a^{\prime })$, the value of the greedy target policy, regardless of what the behaviour policy does next.
• REINFORCE and PPO are on-policy in spirit: each batch is collected by the current policy and mostly discarded after a few updates. Deep Q-networks with replay buffers are off-policy.

Correcting for the mismatch

When an off-policy method estimates expectations under $\pi$ from samples drawn under $b$, it can reweight each sample by the importance sampling ratio

This requires coverage: $b(a\mid s)>0$ wherever $\pi (a\mid s)>0$. Over a multi-step trajectory the ratios multiply. If $\pi$ assigns probability 0.9 to an action that $b$ chose with probability 0.3, the ratio is 3; three such steps in a row give a weight of $3^3=27$, which is why off-policy Monte Carlo estimates can have enormous variance. One-step Q-learning avoids the products entirely because its target does not depend on the next action taken.

Worked contrast

In the cliff-walking gridworld from Sutton and Barto, an agent exploring with $ϵ=0.1$ must walk from start to goal beside a cliff. Q-learning learns the optimal path along the edge, but its exploratory steps occasionally fall off, so its online returns are worse. SARSA learns the value of the exploring policy it actually runs, so it prefers a safer path further from the edge. Neither is wrong; they answer different questions.

Why it matters

The distinction decides what data you can reuse. Off-policy learning lets an agent learn from logs, replay buffers and demonstrations, which is the foundation of offline RL, but brings variance and stability problems. On-policy learning is simpler and more stable but sample hungry. In safety terms, the cliff example generalises: an on-policy agent accounts for its own mistakes, while an off-policy agent may learn a plan that is only safe if executed perfectly.

Related terms

• Policy
• Q-learning
• Importance sampling
• Offline reinforcement learning
• PPO (Proximal Policy Optimization)
• REINFORCE

Sources

1. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net