PPO (Proximal Policy Optimization): Definition and Examples

Proximal Policy Optimization (PPO) is a policy gradient algorithm that approximates a trust region with a simple clipped objective. It lets you run several epochs of minibatch gradient ascent on each batch of collected data without the policy drifting too far from the one that collected it.

The clipped objective

Define the probability ratio $r_{t} (θ) = π_{θ} (a_{t} ∣ s_{t}) / π_{θ_{old}} (a_{t} ∣ s_{t})$ . PPO maximises

L^{CLIP} (θ) = E_{t} [min (r_{t} (θ) A^_{t}, clip (r_{t} (θ), 1 - ϵ, 1 + ϵ) A^_{t})],

where $A^_{t}$ is an advantage estimate (usually from GAE) and $ϵ$ , typically 0.1 to 0.2, sets the clip range. Taking the minimum makes the objective a pessimistic bound: the policy gets no extra credit for pushing the ratio beyond $1 \pm ϵ$ in the direction the advantage favours, but it is still penalised fully if the ratio moves the wrong way.

Worked example

Let $ϵ = 0.2$ .

$A^= + 2$ , ratio 1.5: unclipped term $3.0$ , clipped term $1.2 \times 2 = 2.4$ ; the minimum is 2.4 and its gradient with respect to $θ$ is zero, so this sample stops pushing.
$A^= - 2$ , ratio 0.5: unclipped $- 1.0$ , clipped $0.8 \times (- 2) = - 1.6$ ; the minimum is $- 1.6$ , again flat, so the action is not pushed down further.
$A^= - 2$ , ratio 1.5: unclipped $- 3.0$ , clipped $- 2.4$ ; the minimum is $- 3.0$ , unclipped, so the gradient still corrects the mistake.

The full training loop

In practice PPO is an actor-critic method. The loss combines the clipped policy term, a value-function regression term, and often an entropy bonus. Each iteration collects a batch with the current policy, computes advantages, then runs a few epochs of minibatch updates. Implementation details such as advantage normalisation, value clipping, learning-rate annealing and gradient clipping have a measurable effect on results.

PPO for language models

In RLHF, the policy is a language model, each token is an action, and the reward comes from a reward model plus a KL penalty to a reference model. The critic is a value head of similar size, which is the main memory cost that critic-free methods such as GRPO remove.

Why it matters

PPO became the default deep RL algorithm because it is stable, simple to implement and reuses data. Knowing what the clip does, and what it does not do, helps diagnose training: clipping limits movement per update, but it does not stop the policy drifting over many updates toward exploiting a flawed reward, which is why reward hacking still needs its own defences.

Sources

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. Proximal Policy Optimization Algorithms, 2017. arxiv.org
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel. Trust Region Policy Optimization, 2015. arxiv.org