Training LMs to Follow Instructions with Human Feedback

The Problem

Large language models are inherently trained to predict the next word on a massive corpus of internet text. While this objective produces models that are extremely knowledgeable and capable of generating fluent text, it does not inherently produce models that are helpful, harmless, or honest. If prompted with a question, a base language model might answer the question, but it might just as easily generate a list of related questions, complete a hypothetical dialogue, or output toxic content, because all of those patterns exist on the internet.

The core challenge was the "alignment problem": how do we bridge the gap between a model trained to predict the next token and a model that acts as a helpful assistant that understands and follows user intent?

The Idea

The authors proposed using Reinforcement Learning from Human Feedback (RLHF) to align language models with user intent. While RLHF had been explored in previous, smaller-scale work, this paper proved it could be applied successfully to massive language models (the GPT-3 family), resulting in InstructGPT.

The central idea is to mathematically capture human preferences—what a human considers a "good" or "helpful" response—in a separate neural network called a Reward Model. Once this Reward Model can reliably score responses the way a human would, it can be used to automatically train the main language model using reinforcement learning, steering its outputs toward high-reward territory.

How It Works

The RLHF pipeline defined in this paper is executed in three distinct steps:

Step 1: Supervised Fine-Tuning (SFT). First, human labelers write high-quality prompts and the ideal responses to those prompts. The base language model is fine-tuned on this dataset using standard supervised learning. This teaches the model the basic format of dialogue and instruction-following, producing an SFT model. However, this step is expensive because writing full, high-quality responses is slow.

Step 2: Training the Reward Model (RM). The SFT model is given new prompts and generates multiple different responses for each one. Human labelers are shown pairs of these responses and simply rank which one is better (e.g., A is better than B, or C is better than A). It is much faster and cheaper for humans to rank responses than to write them from scratch. A new model—the Reward Model—is trained on this ranking data. Its job is to take a prompt and a response, and output a single scalar value predicting how highly a human would rate that response.

Step 3: Optimization with PPO. The SFT model is now treated as a reinforcement learning agent. It is given prompts and generates responses. The Reward Model scores these responses. Using the Proximal Policy Optimization (PPO) algorithm, the language model updates its internal weights to maximize the reward it gets from the Reward Model. To prevent the model from "gaming" the reward model by generating degenerate text that happens to score highly, a KL-divergence penalty is added to ensure the PPO model doesn't drift too far from the original SFT model.

Why It Mattered

The results were transformative. The authors showed that a 1.3 billion parameter InstructGPT model (trained with RLHF) was preferred by human evaluators over a 175 billion parameter base GPT-3 model. This meant alignment wasn't just a safety feature; it was a massive capability multiplier that made the model vastly more usable and efficient.

This paper essentially provided the blueprint for how modern AI assistants are created. It proved that aligning models with human intent requires going beyond simple next-token prediction and fundamentally changed the industry's approach to post-training.

What Came After

This exact three-step pipeline became the industry standard and directly enabled the launch of ChatGPT later that same year. For over a year, every major AI lab copied this exact RLHF formula.

However, RLHF has proven complex and unstable to train due to the PPO step. As a result, researchers sought simpler alignment methods. It was eventually largely superseded by Direct Preference Optimization (DPO), which mathematically collapses Step 2 and Step 3 into a single step, aligning the model directly on human preference data without needing a separate Reward Model or reinforcement learning loop.