Reward Shaping: Potential-Based Shaping Explained

Q: What is reward shaping in reinforcement learning?

Reward shaping adds an extra term to the environment's reward to give the agent more frequent feedback, for example a bonus for getting closer to a goal. It can make sparse-reward problems much faster to learn, but unless the extra term has a specific form it can change which policy is optimal, leading the agent to farm the bonus instead of solving the task.

Q: What is potential-based reward shaping?

It is shaping where the extra reward for a transition from s to s' is gamma times Phi(s') minus Phi(s), for some function Phi over states, with Phi equal to zero at terminal states. The shaping terms telescope along any trajectory, so every policy's return is shifted by the same state-dependent constant and the optimal policy is unchanged.

Q: Why does potential-based shaping preserve the optimal policy?

Summed along a trajectory, the shaping terms collapse to minus the potential of the start state, plus a terminal term that is zero by construction. So the shaped action values equal the original ones minus Phi(s), which is the same for every action in a state. Shifting all actions by the same amount cannot change which one is best.

Q: Can reward shaping change the optimal policy?

Yes, if it is not potential-based. Ng, Harada and Russell proved that for any non-potential shaping function there are environments where it changes the optimal policy. In our gridworld, a bonus for each step that reduced the distance to the goal made the optimal policy loop forever once the bonus reached 0.115, about 12% of the goal reward.

Q: Does potential-based shaping always speed up learning?

No. It preserves the optimal policy, not the learning dynamics. For Q-learning it is equivalent to initializing the action values to the potential, so a potential with the wrong scale or sign acts as a bad initialization. In our tests a value-like potential cut total steps 2.7-fold, while a negative-distance potential multiplied them by 7.6.

Q: What is a good potential function?

An estimate of the optimal value function is ideal, because shaping with it makes the shaped values close to zero and the learning problem nearly trivial. Even a rough heuristic on the right scale helps: a Manhattan-distance value estimate that ignored a wall still cut Q-learning steps 2.2-fold in our gridworld.

Q: How is reward shaping related to reward hacking?

A shaping bonus is a proxy for progress. When the proxy can be collected without making progress, for example by circling near a goal, an optimizer will find that behaviour. This is the same mechanism as reward hacking and specification gaming. Potential-based shaping removes the possibility because no route can earn more shaping reward than another.

Reward shaping means adding extra reward to a reinforcement learning problem to guide the agent toward good behaviour faster. It is powerful and dangerous in equal measure: a bonus that seems to point the right way can create a new optimal policy that collects the bonus forever and never finishes the task. Ng, Harada and Russell proved in 1999 that one form is safe in general. If the extra reward is a difference of potentials, $F (s, s^{'}) = γ Φ (s^{'}) - Φ (s)$ , the optimal policy is unchanged for every environment, and no other additive form guarantees that.

This page explains why, with a gridworld you can play with. In it, a bonus of +0.12 per step toward the goal (12% of the goal's reward) makes the optimal policy loop forever next to a wall, while potential-based shaping with any scale leaves it intact. We also show the part that the theorem does not cover: potential-based shaping preserves the optimum but can still make learning much slower. A good potential cut Q-learning's total steps by a factor of 2.7; a badly scaled one multiplied them by 7.6. All numbers come from seeded runs we did for this article.

A gridworld where shaping helps and hurts#

An 8-by-8 grid has a wall down column 4, open only in the top row. The agent starts in the bottom-left corner and the goal, worth +1, is in the bottom-right corner, with discount $γ = 0.95$ . The goal is only 7 steps away in Manhattan distance, but the shortest path goes up to the top row, across the gap and back down: 21 steps. That gap between a natural heuristic and the true distance is the point of the example, and it is typical: real shaping signals are always somewhat wrong.

With the sparse reward alone, tabular Q-learning ( $ε$ -greedy with $ε = 0.1$ , step size 0.5, values initialized to zero) wanders for hundreds of steps per episode until it stumbles on the goal. Averaged over 20 seeds, it needed 19,294 environment steps over 300 episodes, and the greedy policy first became optimal after a median of 40 episodes. Shaping is meant to fix exactly this. (If tabular Q-learning is new to you, the Q-learning page covers it, and value iteration is how we compute exact optimal policies below.)

How a reasonable bonus breaks the optimal policy#

The obvious shaping reward is a bonus for progress: + $k$ whenever a step reduces the Manhattan distance to the goal, nothing otherwise. It rewards the right thing locally. We computed the exact optimal policy of the shaped problem by value iteration for a range of $k$ :

For $k \leq 0.11$ , the optimal policy is unchanged and still takes the 21-step path.
From $k = 0.115$ upward, the optimal policy never reaches the goal. It walks right along the bottom row to the cell beside the wall, then steps up and down forever between two cells. Every step down reduces the Manhattan distance and collects $k$ ; every step up costs nothing.

The arithmetic is simple. A loop that earns $k$ every second step is worth $k / (1 - γ^{2}) \approx 10.3 k$ at $γ = 0.95$ . The honest route requires moving away from the goal for seven steps to climb the column, collects some bonuses on the way down, and ends with a +1 discounted by 21 steps ( $0.9 5^{20} \approx 0.36$ ). Once $k$ passes about 0.115, farming the bonus is simply worth more. The agent is not malfunctioning. It is optimizing the reward it was given.

This is the classic bicycle example from Randløv and Alstrøm's work on learning to ride with reinforcement learning [2], as recounted by Ng and colleagues when motivating their theorem [1]: a shaping reward for progress toward the goal, with no matching penalty for moving away, led the simulated rider to loop in circles instead of reaching the goal. The same pattern, a shaping term or proxy that pays for something correlated with the goal rather than the goal, is the core of specification gaming and reward hacking, and Amodei and colleagues list it among the concrete problems of AI safety [6].

The reason is structural: a bonus that depends only on the transition can be collected around a cycle of states. If the sum of bonuses around some cycle is positive, a long enough discount horizon will make the cycle attractive. Any shaping scheme that is safe in general must make every cycle's total bonus zero, and that is exactly what a potential difference does.

Potential-based reward shaping#

Choose any function $Φ$ over states, set $Φ = 0$ at terminal states, and add

F (s, a, s^{'}) = γ Φ (s^{'}) - Φ (s)

to every transition. Here $Φ (s)$ is the potential of state $s$ , and $γ$ is the same discount factor as the task. Intuitively, the agent is paid for climbing the potential and charged for descending it, and the accounting is arranged so that no route can be more profitable than another in total: as the proof below shows, the discounted shaping payments along any trajectory add up to the same amount, $- Φ (s_{0})$ . A loop can no longer earn anything that the direct route does not.

Why the optimal policy is unchanged#

Add up the shaping terms along any trajectory starting at $s_{0}$ . The sum telescopes:

t = 0 \sum T - 1 γ^{t} (γ Φ (s_{t + 1}) - Φ (s_{t})) = γ^{T} Φ (s_{T}) - Φ (s_{0}) .

If the episode ends in a terminal state with $Φ (s_{T}) = 0$ , or runs forever so that $γ^{T} Φ (s_{T}) \to 0$ for bounded $Φ$ , every trajectory's shaped return equals its original return minus $Φ (s_{0})$ , a constant that depends only on where it started. Taking expectations, the shaped action values are the original ones shifted by a state-dependent constant:

Q^{' *} (s, a) = Q^{*} (s, a) - Φ (s), V^{' *} (s) = V^{*} (s) - Φ (s) .

Within a state, every action is shifted by the same amount, so the ranking of actions and therefore the set of optimal policies is identical [1]. The same argument works for the value of any fixed policy, so policy evaluation is also preserved up to the shift. Ng and colleagues also proved the converse: if $F$ is not of this form, there exist transition and reward functions for which shaping with $F$ changes the optimal policy. You might get away with a non-potential bonus in a particular environment, as with $k \leq 0.11$ in our grid, but you cannot know that without solving the problem.

The terminal-state detail#

The telescoped sum leaves $γ^{T} Φ (s_{T})$ behind. If terminal states have nonzero potential, that term depends on when and where the episode ends, which changes the agent's preferences between short and long episodes and between different exits. Grześ analysed this for episodic tasks and showed that setting the potential of every terminal state to zero is required for the guarantee [4]. In code, this is the line that is easiest to get wrong: when the next state is terminal, use $Φ (s^{'}) = 0$ in the shaping term even if your heuristic would assign it a value. Time-limit truncations are different: the next state is not terminal, so its real potential should be used.

Checking it on the gridworld#

We solved the shaped problem by value iteration with three potentials at many scales: $Φ (s) = k γ^{d (s) - 1}$ , with $d$ the true shortest-path distance (this is exactly $k$ times the optimal value function); the same with Manhattan distance, which ignores the wall; and $Φ (s) = - k \cdot Manhattan (s)$ . In every case, for every scale we tried up to $k = 100$ , the optimal policy took the original 21-step path and chose an optimal action in every state.

Invariance is not speed: what shaping does to learning#

The theorem is about the optimum, not about the path a learning algorithm takes to it. Wiewiora showed that, for Q-learning and similar methods, potential-based shaping with zero-initialized values behaves exactly like unshaped learning with the action values initialized to $Φ$ [3]. So the question "is this a good potential?" is the same as "would $Q_{0} (s, a) = Φ (s)$ be a good initialization?". We ran Q-learning with the same settings as above, 20 seeds and 300 episodes per configuration, capping episodes at 500 steps.

Figure 1. Total environment steps over 300 Q-learning episodes on the gridworld (mean over 20 seeds; log scale). Potentials that look like the value function help; negative-distance potentials, which also preserve the optimal policy, make learning far slower; non-potential progress bonuses trap the learner in loops (every episode hit the 500-step cap).

Shaping	Preserves optimal policy?	Mean steps per episode, first 20	Total steps, 300 episodes	Runs whose greedy policy was optimal by episode 300
None	Yes	392.5	19,294	20 of 20 (median episode 40)
$Φ = γ^{d_{true} - 1}$ (optimal values)	Yes	33.2	7,232	20 of 20
$Φ = γ^{d_{Manhattan} - 1}$	Yes	96.5	8,725	20 of 20 (median episode 35)
$Φ = - 0.1 d_{Manhattan}$	Yes	500 (capped)	146,984	0 of 20
$Φ = - 0.1 d_{true}$	Yes	467.7	101,549	8 of 20
Progress bonus, $k = 0.05$	Yes, in this grid	500 (capped)	150,000	0 of 20
Progress bonus, $k = 0.1$	Yes, in this grid	500 (capped)	150,000	0 of 20

Three findings, each worth remembering:

The best potential is an estimate of the value function. With $Φ$ equal to the optimal values, the first 20 episodes averaged 33 steps instead of 393, and total steps fell 2.7-fold. Even the Manhattan version, wrong about the wall, cut total steps 2.2-fold, because its values have the right scale and ordering almost everywhere.
A policy-preserving potential can still wreck learning. $Φ = - 0.1 \times$ distance encodes the same ordering of states but with the wrong scale and sign. By Wiewiora's equivalence it is a pessimistic initialization: every untried action looks worse than whatever the agent has already done, so exploration stalls. It also pays a small positive reward for bumping into walls far from the goal, since $γ Φ (s) - Φ (s) = 0.1 (1 - γ) d (s) > 0$ . Total steps rose 7.6-fold and no run found the optimal policy in 300 episodes. Invariance of the optimum gave no protection.
A non-potential bonus that is technically harmless can still trap learning. At $k = 0.05$ and $0.1$ the exact optimal policy is unchanged, yet every Q-learning run looped until the step cap in every episode. A near-optimal loop is enough to capture an $ε$ -greedy learner long before the small difference in value is learned.

Try it: shaping that breaks vs preserves the optimal policy#

The left panel shows the greedy policy of the exact optimal solution under the chosen shaped reward, with the path from the start in green if it reaches the goal and red if it loops. The right panel runs Q-learning in your browser with and without the shaping (8 seeds, 150 episodes, so totals are smaller than in the table). Pick a shaping type and move its weight.

Shaping Weight k0.20

Figure 2 (interactive). Exact greedy policy for the shaped reward (value iteration) and Q-learning steps per episode, both computed live with the article's formulas. The static version shows the progress bonus at

k = 0.2

: the optimal policy loops between two cells beside the wall, and Q-learning never finishes an episode within the cap. Try the progress bonus at 0.11 and 0.12, then any potential at any weight.

Shaping in policy gradient methods#

The same algebra has a tidy interpretation for policy gradients. Since potential-based shaping shifts $Q$ and $V$ by the same $Φ (s)$ , the advantage $A = Q - V$ is unchanged, so the expected policy gradient is unchanged as well. In a sampled reward-to-go, the shaping terms from step $t$ onward telescope to $- Φ (s_{t})$ (plus a vanishing terminal term), so potential-based shaping acts exactly like subtracting the baseline $Φ (s_{t})$ . A good potential reduces variance the way a good baseline does, and a poor one adds variance without biasing anything. With a learned critic, as in actor-critic methods, the critic absorbs the shift, and the main effect is on how fast it learns. Non-potential bonuses, by contrast, change the advantage itself, and the policy gradient will faithfully climb toward the bonus-farming behaviour.

Why this matters for alignment and safety#

Most reward functions used in practice contain shaping terms that are not potential-based: format bonuses and length penalties in RL with verifiable rewards, partial credit for intermediate steps, auxiliary rewards for "looking" safe. Each one is a small change to the objective, and the gridworld shows how little it takes. A bonus worth 12% of the true reward changed the optimal behaviour completely, and much smaller ones captured the learning dynamics. Two practical lessons follow.

Prefer shaping you can write as a potential. Progress toward a goal can usually be expressed as a potential (a distance or value estimate) rather than a one-sided bonus. Then the worst case is slow learning, not a different objective.
Evaluate on the true objective. If you must use non-potential terms, evaluate trained policies on the unshaped reward, with held-out checks for loops and degenerate behaviour, as you would for any proxy. Our research notes on reward hacking in RLVR and reward model overoptimization show the same failure at scale, and safe RL covers constraints as an alternative to bonus terms.

Code: a shaping wrapper that keeps the guarantee#

def shaped_reward(r, s, s_next, terminated, phi, gamma):
    """Potential-based shaping (Ng, Harada and Russell, 1999).
    r: original reward for the transition s -> s_next.
    terminated: True only for true terminal states, not for time-limit truncation.
    phi: function from state to float, ideally an estimate of the optimal value."""
    phi_next = 0.0 if terminated else phi(s_next)   # potential of terminal states must be 0
    return r + gamma * phi_next - phi(s)

# Example potential for a gridworld goal at distance d(s): an optimistic value estimate,
# not a negative distance (see the learning-speed results above).
# phi = lambda s: goal_reward * gamma ** (distance_to_goal(s) - 1)

Use the same $γ$ as the learner. Using $Φ (s^{'}) - Φ (s)$ without the discount, a common slip, is not potential-based when $γ < 1$ : it adds a term $(1 - γ) Φ (s^{'})$ that rewards being in high-potential states, which is an ordinary state reward and can change the optimal policy.

A checklist for reward shaping#

Write every shaping term as $γ Φ (s^{'}) - Φ (s)$ if you can. If you cannot, treat the term as a change to the task, not a hint.
Set $Φ = 0$ at terminal states and use the learner's $γ$ .
Choose $Φ$ on the scale of the value function. An estimate of $V^{*}$ , even a rough one, is the ideal potential; potentials with the wrong sign or scale act as bad initializations.
Check for profitable cycles. For any non-potential bonus, ask what an agent could earn by going around a loop forever.
Measure learning on the unshaped return, across several seeds, as described in how to evaluate RL agents.
Anneal non-potential bonuses to zero if you use them for early exploration, and confirm the final policy is evaluated without them.

Shaping is one way to inject knowledge into an agent; learning the reward from demonstrations is another, covered in inverse reinforcement learning, and the general framework is in what is reinforcement learning. Sutton and Barto discuss shaping in the context of designing reward signals [5].

Frequently asked questions#

What is reward shaping in reinforcement learning?