Paper breakdown

Proximal Policy Optimization (PPO)

Introduced PPO, an incredibly stable and sample-efficient Reinforcement Learning algorithm that became the default standard, eventually powering RLHF in ChatGPT.

Paper: Proximal Policy Optimization Algorithms

Authors: John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov · 2017

Read the paper

PPO uses a clipped surrogate objective function to ensure that policy updates are never too large, preventing the agent from accidentally destroying its own performance.

The Problem

In Reinforcement Learning (RL), policy gradient methods update the agent's behavior based on rewards. However, they were famously unstable. A single bad update step could cause the agent's performance to fall off a cliff, and it might never recover. Earlier attempts to fix this, like TRPO (Trust Region Policy Optimization), were mathematically complex, computationally expensive, and difficult to implement.

The Idea

Schulman and the OpenAI team wanted the stability of TRPO but with the simplicity of standard gradient descent. They needed a way to guarantee that an update step wouldn't change the agent's policy too much in one go.

How It Works

PPO introduces a brilliantly simple modification to the objective function: clipping.

When the agent calculates how much better (or worse) a new action is compared to its old policy, it calculates a ratio. PPO clips this ratio between $[1 - \epsilon, 1 + \epsilon]$ (where $\epsilon$ is usually 0.2).

If the new policy finds an action that is vastly better than the old policy, PPO limits how much the agent is allowed to 'cash in' on that discovery in a single step. By clipping the incentive to change drastically, PPO mathematically guarantees that the policy only takes small, safe steps in the right direction, preventing the catastrophic drops in performance that plagued earlier algorithms.

Why It Mattered

PPO achieved the Holy Grail of RL: it was stable, sample-efficient, computationally cheap, and astonishingly easy to implement. It became OpenAI's default RL algorithm, mastering complex tasks from robotics to playing Dota 2.

What Came After

PPO's ultimate crowning achievement came five years later. When researchers needed a stable RL algorithm to fine-tune Large Language Models based on human feedback (RLHF) to create InstructGPT and ChatGPT, PPO was the algorithm they used. It remains the core engine for aligning modern LLMs.