Trust Region and Proximal Methods

These methods cap how far a policy may move in one update, measured in distribution space rather than weight space, so one bad gradient step cannot wreck the policy.

A safe policy shift gains 0.125 inside the KL bound of 0.01 while a greedy shift promising 0.45 lands at KL 0.298 and is rejected

Why Does This Exist?

In supervised learning a bad gradient step just raises the loss for one batch; the dataset sits still while later batches repair the damage. In RL the policy generates its own training data. One oversized step can push the agent into a region where it only collects terrible trajectories, and then it trains on terrible trajectories forever. That is catastrophic policy collapse: the data source and the student fail together.

The trap underneath is geometry. Euclidean distance between weight vectors says nothing about how much the behavior changed. In a sensitive layer, a nudge of $0.001$ can flip the action distribution from 99 percent A to 99 percent B; in a dead layer, a shove of $1.0$ changes nothing. Plain policy gradients optimize in weight space and cannot see this. Trust region and proximal methods optimize in distribution space instead, bounding each update by KL divergence so behavior moves a little even when weights move a lot. TRPO enforces the bound exactly and gets its own page at Trust Region Policy Optimization; PPO approximates it cheaply alongside Proximal Policy Optimization. This page covers the shared idea behind both.

Think of It Like This

The safety tether and the elastic harness

You hike a knife-edge ridge in dense fog. The slope under your boots (the local gradient) says uphill is that way, but five meters ahead the ridge may end in a cliff you cannot see.

With a steel tether (TRPO) you clip into your last confirmed footing with a fixed maximum radius. You may roam anywhere inside the circle, never outside it. Because your mental map is accurate within that radius, every tethered step is safe by construction.

With an elastic harness (PPO) there is no hard ring. Walk normally and it does nothing; lunge past the marked zone and it takes up the slack, removing any reward for going further. Same safety, no surveying equipment.

Where the analogy stops: a tether measures meters. A trust region measures statistical divergence between action distributions, averaged over every state the policy visits, not distance in any physical space.

How It Actually Works

The surrogate and the guarantee

Let $\eta(\pi)$ be the expected discounted return of a policy. A classic identity relates a new policy to the old one through the old policy's advantage function $A^{\text{old}}(s, a)$ (how much better action $a$ is than average in state $s$ ):

$\eta(\pi_{\text{new}}) = \eta(\pi_{\text{old}}) + \mathbb{E}_{s \sim \rho_{\text{new}},\, a \sim \pi_{\text{new}}}[A^{\text{old}}(s, a)]$

where $\rho$ is the discounted state visitation distribution. The catch: the expectation needs states from the new policy, which does not exist yet. Swap in the old policy's states and correct the actions with a probability ratio, and you get the surrogate objective, an estimate built entirely from data you already hold:

$L(\theta) = \mathbb{E}_{s,\, a \sim \pi_{\text{old}}}\left[ \frac{\pi_{\theta}(a \mid s)}{\pi_{\text{old}}(a \mid s)}\, A^{\text{old}}(s, a) \right]$

At $\theta = \theta_{\text{old}}$ the surrogate matches the true return exactly, with matching gradients. The guarantee behind the whole family: true return stays above the surrogate minus a penalty on the worst-case KL divergence between the policies. So any update that raises the penalized surrogate provably cannot make things worse. The penalty constant is too conservative to use directly, which is where the two methods diverge.

TRPO: a hard circle in distribution space

TRPO maximizes the surrogate subject to a mean-KL constraint with radius $\delta$ :

$\max_{\theta}\; L(\theta) \quad \text{subject to} \quad \bar{D}_{\text{KL}}(\theta_{\text{old}}, \theta) \le \delta$

It linearizes the objective, approximates the KL surface quadratically with the Fisher information matrix, and steps along the natural gradient direction scaled to land exactly on the circle. Inverting that matrix for millions of weights is impossible, so implementations solve for the direction with conjugate gradients (matrix-vector products only, never the matrix) plus a line search that rejects steps violating the bound. Exact, principled, and expensive.

PPO: clip the ratio instead

PPO keeps the first-order simplicity of ordinary gradients. With ratio $r_t(\theta) = \pi_{\theta}(a_t \mid s_t) / \pi_{\text{old}}(a_t \mid s_t)$ and clipping range $\epsilon$ (usually $0.2$ ):

$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t\left[ \min\left( r_t(\theta) \hat{A}_t,\; \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon)\, \hat{A}_t \right) \right]$

The $\min$ takes the pessimistic view: when the advantage is positive and the ratio runs past $1 + \epsilon$ , the objective stops rewarding further movement, so the gradient goes quiet at the boundary. No second-order machinery, just Adam on a clipped loss.

Worked example: one safe step, one rejected step

One state, two actions. Old policy $\pi_{\text{old}} = [0.80, 0.20]$ , advantages $A = [+0.50, -2.00]$ . Sanity check first: $0.80(0.50) + 0.20(-2.00) = 0.40 - 0.40 = 0$ , as it must, since the average action scores zero. Trust radius $\delta = 0.010$ .

Candidate 1, the careful shift $\pi_1 = [0.85, 0.15]$ . Surrogate gain: $0.85(0.50) + 0.15(-2.00) = 0.425 - 0.300 = +0.125$ . KL divergence: $0.80 \ln(0.80/0.85) + 0.20 \ln(0.20/0.15) = -0.048500 + 0.057536 = 0.009037$ . Since $0.009037 \le 0.010$ , the step passes and banks $+0.125$ .

Candidate 2, the greedy grab $\pi_2 = [0.98, 0.02]$ . Surrogate gain: $0.98(0.50) + 0.02(-2.00) = 0.490 - 0.040 = +0.450$ , nearly four times larger. KL: $0.80 \ln(0.80/0.98) + 0.20 \ln(0.20/0.02) = -0.162353 + 0.460517 = 0.298164$ , about thirty times the radius. Rejected. The local map promised $+0.450$ exactly where it could no longer be trusted.

The PPO view of candidate 2, with $\epsilon = 0.2$ : the ratio on the good action is $0.98/0.80 = 1.225$ , past the $1.2$ ceiling, so its contribution clips from $1.225 \times 0.50 = 0.6125$ down to $1.20 \times 0.50 = 0.60$ . The extra greed buys nothing, which is the clipping doing its job. All three numbers appear in the diagram above and the code below.

Code

The snippet evaluates both candidates against the KL bound and shows the PPO clip biting on the greedy one.

1import math2from typing import List3
4
5def surrogate_gain(pi_new: List[float], advantages: List[float]) -> float:6    """Expected advantage under the candidate policy."""7    return sum(p * a for p, a in zip(pi_new, advantages))8
9
10def kl_divergence(pi_old: List[float], pi_new: List[float]) -> float:11    """D_KL(pi_old || pi_new), the trust-region distance."""12    return sum(p * math.log(p / q) for p, q in zip(pi_old, pi_new))13
14
15if __name__ == "__main__":16    pi_old = [0.80, 0.20]17    advantages = [0.50, -2.00]18    delta = 0.01019    for cand in ([0.85, 0.15], [0.98, 0.02]):20        gain = surrogate_gain(cand, advantages)21        kl = kl_divergence(pi_old, cand)22        print(round(gain, 4), round(kl, 6), kl <= delta)23    # -> 0.125 0.009037 True24    # -> 0.45 0.298164 False25    ratio = 0.98 / 0.8026    print(round(ratio, 4))  # -> 1.22527    print(round(min(ratio, 1.2) * 0.5, 4))  # -> 0.628    print(round(ratio * 0.5, 4))  # -> 0.6125

Watch Out For

Paying the full second-order price when PPO would do

The symptom: a TRPO implementation that trains slowly, fights the conjugate-gradient hyperparameters, and breaks on recurrent policies, all for stability you could have had cheaper. The Fisher machinery (10 to 20 conjugate-gradient iterations plus a line search per update) exists to respect the circle exactly.

The fix: default to PPO's clipped objective unless you need the hard guarantee for analysis. It runs on plain Adam, composes with any architecture, and matches TRPO empirically on most benchmarks. Keep TRPO for the settings where a provable bound matters more than throughput.

A passing mean KL that hides a spiking state

The symptom: the mean KL sits comfortably under $\delta$ while behavior on a few critical states swings wildly (a robot that is steady everywhere except the one doorway that matters). The constraint averages over the state distribution, so rare states can diverge almost unchecked while frequent ones comply.

The fix: monitor the maximum per-state KL alongside the mean, and investigate when they disagree by an order of magnitude. If a handful of states dominate the risk, raise their sampling weight or tighten $\delta$ until the max behaves, not just the average.

The Quick Version

Plain policy gradients step in weight space, where a small step can mean a huge behavior change; since the policy makes its own training data, one bad step can collapse training permanently.
The surrogate objective estimates the new policy's return from old-policy data via the probability ratio, and the improvement bound says maximizing it inside a KL radius cannot hurt.
TRPO enforces the radius $\bar{D}_{\text{KL}} \le \delta$ exactly with natural gradients and conjugate gradients; PPO approximates it by clipping the ratio to $[1 - \epsilon, 1 + \epsilon]$ .
On the worked state (bound 0.01), the careful shift gains 0.125 at KL 0.009037 and passes, while the greedy grab promises 0.45 at KL 0.298164 and is rejected.
PPO clips the greedy ratio 1.225 to 1.2 so the extra greed earns nothing; watch the max per-state KL, not just the mean, since averages hide spikes.