Trust region (TRPO): Definition and Examples

A trust region is a limit on how far a single update may move the policy, inside which a local approximation of the objective is reliable. Trust Region Policy Optimization (TRPO) applies this idea to policy gradients by maximising a surrogate objective subject to a bound on the average KL divergence between the old and new policies.

The problem it solves

Plain policy gradient steps are measured in parameter space, but a small change in parameters can cause a large change in behaviour. One bad step can collapse a policy, and because the next batch of data is collected with that broken policy, recovery can be hard. A trust region measures step size in policy space instead.

The TRPO problem

θ max E_{s, a \sim π_{θ_{old}}} [\frac{π _{θ} ( a ∣ s )}{π _{θ_{old}} ( a ∣ s )} A^(s, a)] s.t. E_{s} [D_{KL} (π_{θ_{old}} (\cdot ∣ s) ∥ π_{θ} (\cdot ∣ s))] \leq δ .

The objective is an importance-weighted estimate of how much the new policy improves on the old one, using advantages $A^$ computed under the old policy. The constraint keeps the average KL divergence below a small $δ$ , often 0.01. Schulman and colleagues derived it from a bound showing that the surrogate approximates true improvement only when the policies are close. TRPO solves the problem approximately with a conjugate-gradient step on the Fisher information matrix, the curvature of the KL, followed by a line search that rejects steps violating the constraint.

Worked example

A one-state Bernoulli policy currently picks action 1 with probability 0.5. Moving to 0.6 gives $D_{KL} = 0.5 ln (0.5/0.6) + 0.5 ln (0.5/0.4) \approx 0.020$ , which exceeds $δ = 0.01$ . Moving to 0.55 gives about 0.005, which is allowed. The constraint therefore caps this update somewhere between 0.55 and 0.6, regardless of how large the gradient is.

Relation to other methods

The natural policy gradient takes the same Fisher-preconditioned direction without the line search.
PPO replaces the hard constraint with a clipped objective or an adaptive KL penalty, which keeps most of the stability with first-order optimisation.
The KL penalty in RLHF is a different idea: it ties the policy to a fixed reference model for the whole run, not just to the previous iterate.

Why it matters

The trust-region idea, that policy updates should be small in behaviour space, underlies nearly every stable modern policy optimiser. It is why practitioners monitor KL divergence per update and treat a sudden spike as a sign that training is about to go wrong.

Sources

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel. Trust Region Policy Optimization, 2015. arxiv.org