Phasic Policy Gradient (PPG)

Phasic Policy Gradient splits training into a PPO policy phase and a value distilling auxiliary phase, so shared features help without wrecking the policy.

PPG alternates a PPO policy phase with a KL guarded auxiliary phase where value loss 0.5000 plus KL 0.0038 gives joint loss 0.5038

Why Does This Exist?

Actor critic methods want shared features. Value prediction gives a dense signal on every state, so a shared trunk learns visual structure far faster than sparse policy gradients alone. That is why joint PPO networks beat separate networks on pixel tasks like Procgen when they work.

The catch is interference. Value targets swing hard each rollout, and those large regression gradients yank the shared trunk around. The policy head sits on the same trunk, so its action probabilities wobble even when the PPO clip did its job. You get fast features and an unstable policy in one package.

Separate networks fix the wobble and lose the features. The actor must rediscover from scratch what the critic already knows, which costs far more environment steps.

This page covers the PPG fix from Cobbe and co: alternate a pure policy phase with an offline auxiliary phase that distills value features into the policy trunk while a KL leash holds the policy still. For the PPO update used inside the policy phase, see trust region and proximal methods. For the actor critic setup it builds on, see actor critic methods.

You need PPO clipping and KL divergence before this page. PPO caps each policy step with a clipped ratio, and KL measures how far one action distribution has drifted from another. PPG uses the second to protect the first.

Think of It Like This

Field trips, then lab weeks

You lead a small survey team in rough country. During field weeks you walk, dodge trouble, and bag soil samples. You do not run chemistry in the field. You just record where you went and what you picked up.

Every month you return to base for lab week. You assay every bag, update the shared map, and check one rule before going out again: your trail reflexes must still match the log. If the map work changed how you walk, you redo it.

That check is the KL leash. Field skill stays intact while map knowledge grows.

Where the analogy stops: your team has separate legs and lab gear. PPG has one shared trunk under both heads, so lab gradients touch field weights directly. Without the cloning loss, the map rewrite would overwrite the walking skill.

How It Actually Works

PPG keeps two networks. The policy network holds a shared trunk with weights theta, a policy head pi, and an auxiliary value head. The value network holds separate weights phi and the true value head used for advantages. Symbols: s is state, a is action, V hat is the empirical return target, pi old is the frozen policy from rollout time, N pi counts policy iterations per cycle, E aux counts auxiliary epochs, beta clone weights the leash.

The policy phase

Run N pi iterations, commonly 32. Collect rollouts under the current policy, compute advantages from the separate value network, update the policy head with the PPO clipped loss, and update the separate value head with squared error. Freeze the auxiliary head. No value gradient touches theta here.

Cache every transition into an auxiliary buffer B aux as tuples of state, stored action probabilities, and return target. Then pause the environment.

The auxiliary phase

Run E aux epochs, commonly 6, over the full buffer. Train the auxiliary head to predict the stored returns:

$L^{aux} = \\frac{1}{2} \\hat{E}[(V^{aux}(s) - \\hat{V})^2]$

Protect the policy with a cloning loss against the stored snapshot:

$L^{clone} = \\hat{E}[D_{KL}(\\pi_{old}(\\cdot \\mid s) \\parallel \\pi(\\cdot \\mid s))]$

Optimize the joint loss on theta:

$L^{joint} = L^{aux} + \\beta_{clone} L^{clone}$

with beta clone usually 1.0. Train the separate value head a few more epochs on the same buffer, then flush the buffer and return to the field.

Worked example

One buffered state has two actions, target return 3.50, and stored logits [1.0, 0.0]. Softmax gives pi old as [0.7311, 0.2689] because exp(1.0) is 2.7183 and the normalizer is 3.7183.

The auxiliary head predicts 2.50. The distillation loss is 0.5 times (2.50 minus 3.50) squared, which is 0.5000.

Auxiliary training nudges the policy logits to [1.1, minus 0.1]. Softmax gives pi new as [0.7685, 0.2315] because exp(1.1) is 3.0042, exp(minus 0.1) is 0.9048, and the normalizer is 3.9090. The KL term is 0.7311 times log(0.7311 divided by 0.7685) plus 0.2689 times log(0.2689 divided by 0.2315), which is minus 0.0365 plus 0.0403, or 0.0038 nats.

Joint loss is 0.5000 plus 1.0 times 0.0038, which is 0.5038. Value pressure dominates, the leash adds a small pull, and the diagram shows exactly these three numbers side by side.

Code

This snippet recomputes the worked numbers. It is the same check you would run before trusting a PPG auxiliary step.

1from math import exp, log2
3
4def softmax(logits: list[float]) -> list[float]:5    """Stable softmax over a short logit list."""6    peak = max(logits)7    exps = [exp(z - peak) for z in logits]8    total = sum(exps)9    return [e / total for e in exps]10
11
12def kl(q: list[float], p: list[float]) -> float:13    """KL divergence D(q || p) in nats."""14    return sum(qi * log(qi / pi) for qi, pi in zip(q, p))15
16
17old = softmax([1.0, 0.0])18new = softmax([1.1, -0.1])19l_aux = 0.5 * (2.50 - 3.50) ** 220l_kl = kl(old, new)21print([f"{v:.4f}" for v in old])22# -> ['0.7311', '0.2689']23print([f"{v:.4f}" for v in new])24# -> ['0.7685', '0.2315']25print(f"{l_aux:.4f} {l_kl:.4f} {l_aux + l_kl:.4f}")26# -> 0.5000 0.0038 0.5038

Watch Out For

Auxiliary epochs that erase the policy

The symptom is a policy that looks great leaving the policy phase, then collapses after lab week. Action entropy falls off a cliff, rollout returns swing, and the next policy phase starts from a worse place than the last one ended.

The cause is too much distillation with too little leash. Six epochs with beta clone at 1.0 is the tested balance. Pushing to 12 epochs or dropping beta clone to 0.1 lets value gradients drag the trunk wherever they want, and the KL term cannot hold the policy head still.

Fix it by capping E aux near 6, keeping beta clone at 1.0, and watching mean KL during the auxiliary phase. If it passes about 0.02, stop early and flush. The value head can wait for the next cycle.

Sharing gradients by accident

The symptom is PPG that behaves exactly like joint PPO: fast early gains, then jittery updates and sudden drops. People add the buffer and the KL term but leave the auxiliary head unfrozen during the policy phase, so value gradients leak into theta on every rollout.

Fix it with one rule. During the N pi policy iterations, only PPO touches theta. During the E aux epochs, only the joint loss touches theta. If your framework shares an optimizer, use separate optimizer instances or freeze the auxiliary head parameters explicitly and check that their gradients stay at zero during rollouts.

The Quick Version

PPG alternates a PPO policy phase with an offline auxiliary phase to get shared features without gradient interference.
The policy phase runs about 32 iterations with the auxiliary head frozen, and every transition lands in an auxiliary buffer.
The auxiliary phase distills returns into the policy trunk for about 6 epochs under a KL cloning leash with beta clone near 1.0.
The worked state gives value loss 0.5000 plus KL 0.0038 for a joint loss of 0.5038.
Too many auxiliary epochs or a weak leash erases the policy, and leaking value gradients into the policy phase turns PPG back into joint PPO.