Reward shaping

Reward shaping is the practice of adding extra reward signals to a reinforcement learning problem so the agent learns faster, typically by rewarding progress toward a goal rather than only the goal itself. Done carelessly it changes what the agent optimises; done in potential-based form it provably does not.

Potential-based shaping

Ng, Harada and Russell (1999) showed that if the shaping term has the form

for some potential function $\Phi$ over states, then the optimal policies of the shaped problem $R+F$ are exactly those of the original problem $R$. Here $\gamma$ is the discount factor. The reason is that the shaping terms telescope along any trajectory: the total added reward depends only on the start state (and the end state, for finite episodes), so it cannot make any action sequence relatively more attractive. They also showed that, without further knowledge of the environment, potential-based shaping is the only form with this guarantee.

Worked example

In a gridworld, let $\Phi (s)=-d(s)$, where $d$ is the distance to the goal, and $\gamma =0.9$. Stepping from distance 4 to 3 earns $F=0.9\times (-3)-(-4)=1.3$. Stepping away, from 4 to 5, earns $0.9\times (-5)+4=-0.5$. The agent gets dense feedback about direction, yet walking in a loop back to where it started earns nothing extra when $\gamma =1$ and slightly less than nothing when $\gamma <1$.

Contrast a naive bonus of $+1$ for every step that reduces distance, with no penalty for moving away. Then oscillating toward and away from the goal earns $+1$ every two steps forever, which can beat reaching the goal. Ng and colleagues cite a bicycle-riding agent that learned to ride in circles for exactly this kind of reason.

In practice

• A good potential approximates the optimal value function; if $\Phi =V^{\ast }$, learning becomes nearly trivial.
• Shaping that is not potential-based, such as auxiliary bonuses for format or length in language model training, should be treated as changing the objective and evaluated as such.
• Shaping helps most with sparse rewards and long horizons, where the agent otherwise rarely sees any signal.

Why it matters

Hand-designed reward terms are one of the most common sources of specification gaming and reward hacking. The potential-based result gives a precise test: if a bonus cannot be written as a difference of potentials, assume the agent will find a way to collect it that you did not intend.

Related terms

• Sparse reward
• Specification gaming
• Reward hacking
• Credit assignment
• Value function

Sources

1. Andrew Y. Ng, Daishi Harada, Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping, 1999. people.eecs.berkeley.edu
2. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net