PPO in reinforcement learning, from first principles: trust regions, natural gradients and clipping

Proximal policy optimization (PPO) is a policy gradient method that reuses each batch of experience for several gradient steps on a clipped surrogate objective. The objective weights each action's advantage by the probability ratio $r={\pi }_{\theta }(a\mid s)/{\pi }_{\text{old}}(a\mid s)$ and stops rewarding changes once $r$ leaves $[1-\epsilon ,1+\epsilon ]$. That approximates the trust region of its predecessor, TRPO, which limits the KL divergence between the old and new policy explicitly, using nothing more than first-order optimization.

This page builds PPO from reinforcement learning foundations rather than as a recipe: why raw policy gradient steps are hard to size, where the surrogate objective comes from, what a trust region guarantees, how the natural gradient implements one, and what clipping does and does not do. A seeded gridworld experiment measures the per-update KL divergence of each approach. At the same learning rate and number of epochs, optimizing the unclipped surrogate produced single updates with a KL of 20 nats, while clipping kept the worst update below 1 and TRPO held it at its 0.015 target. All numbers come from runs we did for this article.

Why plain policy gradient steps are hard to size#

The policy gradient [1] tells you which direction improves the policy at the current parameters. It says nothing about how far you can go. Two facts make that a real problem.

• Distance in parameters is not distance in behaviour. For a softmax policy, moving a logit from 0 to 1 changes action probabilities a lot; moving it from 10 to 11 changes almost nothing. A fixed step size in $\theta$ is too timid in some places and reckless in others.
• A bad step corrupts your next batch. In supervised learning, an overshoot is fixed by the next gradient on the same data. In reinforcement learning, the policy generates its own data. Step to a policy that never visits the useful states, and the next batches contain no information to undo it. The REINFORCE article shows this collapse directly: past a step-size threshold, runs fall into deterministic policies that never reach the goal.

There is also an efficiency problem. A policy gradient estimate is only valid at the parameters that collected the data, so vanilla methods take one step per batch and discard it. Sampling is usually the expensive part, so we would like to take several steps per batch without losing correctness. Both problems have the same solution: an objective that is a good local model of the true return, and a limit on how far to trust it.

The surrogate objective#

Kakade and Langford's performance difference lemma [5] states exactly how much better a new policy ${\pi }^{\prime }$ is than an old one $\pi$:

where $J$ is expected discounted return, $d^{{\pi }^{\prime }}$ is the normalized discounted state distribution of the new policy and $A^{\pi }=Q^{\pi }-V^{\pi }$ is the advantage under the old one. In words: the improvement is the old policy's advantage of the new policy's actions, averaged over the states the new policy visits. The trouble is $d^{{\pi }^{\prime }}$: we only have samples from $d^{\pi }$. Replace it with the old distribution and rewrite the action expectation with an importance ratio, and you get the surrogate

which can be estimated entirely from the old batch. Two properties make it useful. At ${\pi }^{\prime }=\pi$ it equals zero, matching the true improvement, and its gradient with respect to the new policy's parameters there equals the policy gradient, because ${\nabla }_{\theta }\frac{{\pi }_{\theta }}{\pi }{|}_{\theta ={\theta }_{\text{old}}}={\nabla }_{\theta }\log {\pi }_{\theta }$. So $L$ is a first-order-accurate local model of $J$. Its error comes from the state-distribution swap, which grows with how different ${\pi }^{\prime }$ is from $\pi$. Maximize $L$ without limits and you will exploit exactly the regions where the model is wrong.

Trust regions: TRPO#

Schulman and colleagues bounded the error of the surrogate [3]. Writing $\eta$ for expected return, their result implies

where $D_{\mathrm{KL}}^{\max }$ is the largest KL divergence between the two policies' action distributions over all states and $ϵ={\max }_{s,a}|A^{\pi }(s,a)|$ (with $L_{\pi }$ as defined above, which matches the paper's definition). Maximizing the right-hand side each iteration guarantees that the true return never decreases: a minorize-maximize scheme. The constant $C$ is enormous for realistic $\gamma$, so the penalty version takes tiny steps. TRPO keeps the idea and replaces the penalty with a constraint on the average KL over visited states:

The hat denotes an average over the batch, and $\delta$ (0.01 in the paper's experiments) is the trust-region radius in nats. The constraint is on behaviour, not parameters, which fixes the first problem above, and the monotonic-improvement argument, though no longer a guarantee once you switch to average KL and sampled advantages, explains why it fixes the second.

The natural gradient inside TRPO#

To solve the constrained problem, approximate the objective to first order and the KL to second order. Near ${\theta }_{\text{old}}$, $D_{\mathrm{KL}}\approx \frac{1}{2}\Delta {\theta }^{\top }F\Delta \theta$, where $F=\mathbb{E}[\nabla \log {\pi }_{\theta }\nabla \log {\pi }_{\theta }^{\top }]$ is the Fisher information matrix of the policy. Maximizing $g^{\top }\Delta \theta$ subject to $\frac{1}{2}\Delta {\theta }^{\top }F\Delta \theta \le \delta$ gives

where $g$ is the policy gradient. The direction $F^{-1}g$ is the natural gradient, introduced for learning by Amari [7] and for policies by Kakade [6]: the steepest-ascent direction when distance is measured by KL divergence between action distributions rather than Euclidean distance between parameters. It is invariant to how you parameterize the policy. TRPO computes $F^{-1}g$ with conjugate gradient using Fisher-vector products, so $F$ is never formed, then backtracks along the direction until the sampled KL is within $\delta$ and the surrogate actually improves. That line search is what makes TRPO dependable and also what makes it awkward: it needs second-order machinery and does not combine easily with architectures that share parameters between policy and value function.

What is PPO? The clipped objective#

PPO keeps the surrogate and replaces the constraint with a change to the objective itself [4]. For each sample, with $r_t(\theta )={\pi }_{\theta }(a_t\mid s_t)/{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)$,

where $\epsilon$ (0.2 in the paper) sets the width of the region in which the ratio may move freely. The minimum makes the objective a pessimistic bound on the unclipped surrogate: it takes the clipped value when clipping would make the objective look better and the unclipped value when it would look worse. Working through the cases shows what that means for the gradient:

Note what is not clipped. If a good action has become less likely ($\hat{A}>0$, $r<1-\epsilon$), the full gradient remains, so the optimizer can undo a move in the wrong direction. Clipping only removes the incentive to keep moving in the direction the advantage asks for once the ratio has moved far enough. The PPO paper also proposes an adaptive KL-penalty variant, ${\hat{\mathbb{E}}}_t[r_t{\hat{A}}_t-\beta D_{\mathrm{KL}}]$, doubling or halving $\beta$ when the measured KL drifts from a target; it reported clipping as the better performer. A complete PPO loss adds a value-function term (PPO is an actor-critic method, usually with GAE advantages [2]) and an entropy bonus.

Rotate the surface and two features stand out. The flat red plateaus sit on opposite sides for positive and negative advantages, so the clip is asymmetric in $r$ but symmetric in intent. And the plateaus are flat, not walls: nothing in the objective pulls $r$ back once it is outside the range. Clipping stops the push; it does not enforce the bound.

Measuring trust: a gridworld experiment#

Setup (simulation). A 5-by-5 gridworld with a goal worth +1 in the far corner, three pits worth −1, 10% chance that a move goes in a random other direction, and $\gamma =0.95$. The policy is a tabular softmax (100 logits), starting uniform with a true return of −0.505; the optimal return is 0.600. Each iteration collects 16 episodes, computes advantages as the Monte Carlo return minus the exact value of the current policy (to isolate the optimizer from critic error), and updates with one of: a single vanilla policy gradient step; 10 full-batch gradient epochs on the unclipped surrogate; 10 epochs on the PPO clipped surrogate with $\epsilon =0.2$; or a TRPO step (natural gradient from the sample Fisher matrix, then backtracking until the sample KL is below $\delta$ and the surrogate improves). We ran 40 iterations for 10 seeds each and measured the exact KL divergence of every update, averaged over the old policy's discounted state distribution, and the exact return after every update.

What the numbers say, without overselling them:

• Clipping bounds the update size, roughly. At learning rate 128 the unclipped surrogate made an update with KL 20 nats and a ratio of 454 on some sample: the policy jumped to near-deterministic choices from 16 episodes of evidence, and its final return (0.137) was the worst in the table. With the same optimizer settings, the clipped objective kept every update below 1 nat and finished at 0.348.
• Clipping does not keep the ratio inside $[0.8,1.2]$. Even at the gentler learning rate the median batch contained a sample with ratio 3.1. The flat plateaus remove the gradient for samples already outside the range, but other samples in the same state still move the shared logits, and every sample's ratio moves with them. Wang and colleagues document the same effect in deep networks [9].
• A small problem forgives aggressive steps. On this tabular task the unclipped surrogate at learning rate 32 reached the highest return after 40 updates (0.545). Large, risky updates that happen to land well are a real phenomenon in small problems, and it is exactly why the safety argument for trust regions should be judged on worst cases and on larger function approximators, not on one average.
• TRPO hits its target precisely. Its mean and maximum KL are close to $\delta$, and with $\delta =0.05$ it was the fastest learner after 10 updates. The cost is per-step computation: here a 100-by-100 linear solve, in deep networks dozens of conjugate-gradient iterations and a line search.

The share of updates that lowered the return (13% to 16% for the aggressive settings, including PPO) is a reminder that none of these methods guarantees improvement once advantages are sampled. Engstrom and colleagues found that much of PPO's advantage over TRPO in standard deep RL benchmarks came from code-level choices such as value-function clipping, reward scaling, learning-rate annealing and network initialization rather than from the clipped objective itself, and that PPO's clipping did not by itself keep updates within the trust region [8]. Andrychowicz and colleagues' large on-policy study recommends the PPO loss with $\epsilon$ around 0.25 as a starting point and tuning it per task [10].

TRPO vs PPO#

The PPO clipped gradient in numpy#

For a tabular softmax policy, the gradient of the clipped objective is the importance-weighted policy gradient with clipped samples masked out. This is the function used in the experiment.

import numpy as np

def softmax(logits):
    e = np.exp(logits - logits.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

def ppo_clip_grad(theta, theta_old, states, actions, adv, eps=0.2):
    """theta, theta_old: (S, A) logits. states, actions: (N,) ints. adv: (N,) advantages.
    Returns the gradient of mean(min(r*A, clip(r, 1-eps, 1+eps)*A)) w.r.t. theta, and the ratios."""
    pi, pi_old = softmax(theta), softmax(theta_old)
    r = pi[states, actions] / pi_old[states, actions]
    clipped = ((adv > 0) & (r > 1 + eps)) | ((adv < 0) & (r < 1 - eps))
    coef = np.where(clipped, 0.0, r * adv)          # d(r*A)/d log pi = r*A where not clipped
    grad = np.zeros_like(theta)
    np.add.at(grad, (states, actions), coef)        # d log pi(a|s) / d theta[s, a] = 1 - pi(a|s) ...
    np.add.at(grad, states, -coef[:, None] * pi[states])  # ... and -pi(b|s) for every b
    return grad / len(states), r

# one PPO update: several epochs of ascent on the same batch
# for epoch in range(10):
#     g, r = ppo_clip_grad(theta, theta_old, S, A, adv)
#     theta += lr * g

Two monitoring quantities are worth logging from this function in any implementation: the clip fraction (share of samples with clipped true) and an estimate of the KL between the old and new policy. Stopping the epochs early when KL exceeds a threshold, a common addition, brings PPO closer to an actual trust region.

Using PPO well#

1. Start from $\epsilon =0.2$, 4 to 10 epochs, GAE with $\lambda =0.95$. These defaults come from the original paper and survive most re-examinations; tune $\epsilon$ per task.
2. Log approximate KL and clip fraction every update. A KL that climbs from update to update, or a clip fraction that keeps growing, signals that the learning rate or epoch count is too high for the task.
3. Use KL early stopping as a seatbelt. Our experiment shows clipping alone allowed single updates of nearly 1 nat at a high learning rate.
4. Normalize advantages and observations; anneal the learning rate. These code-level details changed results as much as the objective in published comparisons [8].
5. Get the advantages right first. PPO inherits every bias of its critic; see actor-critic and GAE and baselines.
6. Compare across seeds. Differences between PPO variants are often smaller than seed variance; see how to evaluate RL agents.

A note on language models#

PPO was the optimizer in early RLHF pipelines, where the policy is a language model, an action is a token and a reward model scores whole responses. Everything above carries over, with one addition: a KL penalty to a fixed reference model is usually added to the reward to keep the policy close to its starting point. That KL serves a different purpose from TRPO's trust region (it limits drift over the whole of training, not per update) and is covered in KL-regularized RL. The wider context is in reinforcement learning for LLMs.

Failure modes#

• Too many epochs or too high a learning rate. The ratio escapes the clip range through shared parameters and the policy drifts far from the data, as in our learning-rate 128 runs.
• Entropy collapse. Repeated confident updates can make the policy near-deterministic before it has explored; an entropy bonus or a smaller $\epsilon$ helps.
• Stale advantages. Advantages are computed once per batch with the old critic. Reusing a batch for many epochs optimizes against increasingly outdated estimates.
• Mistaking the surrogate for progress. A rising $L^{\text{CLIP}}$ says the policy moved toward the batch's advantages, not that the return improved; one in seven of our aggressive updates lowered the true return.

Frequently asked questions#

Sources#

1. R. S. Sutton, D. McAllester, S. Singh, Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation, 1999. proceedings.neurips.cc
2. J. Schulman, P. Moritz, S. Levine, M. Jordan, P. Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation, 2015. arxiv.org
3. J. Schulman, S. Levine, P. Moritz, M. I. Jordan, P. Abbeel. Trust Region Policy Optimization, 2015. arxiv.org
4. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov. Proximal Policy Optimization Algorithms, 2017. arxiv.org
5. S. Kakade, J. Langford. Approximately Optimal Approximate Reinforcement Learning, 2002. people.eecs.berkeley.edu
6. S. M. Kakade. A Natural Policy Gradient, 2001. proceedings.neurips.cc
7. S. Amari. Natural Gradient Works Efficiently in Learning, 1998. doi.org
8. L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, A. Madry. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO, 2020. arxiv.org
9. Y. Wang, H. He, C. Wen, X. Tan. Truly Proximal Policy Optimization, 2019. arxiv.org
10. M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, O. Bachem. What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study, 2020. arxiv.org