Let's Verify Step by Step

The Problem

As models got better at generating Chains of Thought to solve math problems, the standard way to train them via Reinforcement Learning was to use Outcome Supervision.

In Outcome Supervision, a Reward Model (ORM) only looks at the final answer of a generated chain. If the final answer is correct, the entire chain of thought gets a positive reward. If it's wrong, the chain gets a negative reward.

This creates a massive "alignment tax" and a major credit assignment problem. What if the model makes a brilliant logical deduction in step 2, but makes a dumb arithmetic error in step 5, leading to a wrong answer? Under outcome supervision, that brilliant step 2 gets penalized. Conversely, what if the model uses totally flawed, hallucinatory logic, but coincidentally arrives at the correct final number? The model gets rewarded for bad logic (a phenomenon known as "reward hacking" or a "lucky guess").

The Idea

OpenAI proposed shifting from Outcome Supervision to Process Supervision.

Instead of training a Reward Model to look only at the final answer, they trained a Process Reward Model (PRM) to evaluate every single intermediate step in the model's chain of thought. Human labelers were hired to meticulously grade individual steps of math solutions as "positive" (correct and logical), "negative" (incorrect logic or math error), or "neutral" (ambiguous but not strictly wrong).

By providing dense, step-by-step feedback, the PRM can precisely reward good logic and penalize bad logic exactly where it happens, solving the credit assignment problem and heavily discouraging reward hacking.

How It Works

1. Data Collection: Human mathematicians were shown model-generated solutions to challenging math problems (from the MATH dataset). They graded each step of the solution individually.
2. Training the PRM: A Process Reward Model was trained on this step-by-step data. Given a partial chain of thought (up to step $k$), the PRM predicts whether step $k$ is logically sound.
3. Inference / Search: To solve a new problem, a generator model produces many different chains of thought (similar to Self-Consistency or Tree of Thoughts). The PRM scores every step of every chain. The overall score for a chain is usually calculated as the minimum score of any of its steps (if any step is mathematically invalid, the whole proof is invalid). The chain with the highest PRM score is selected as the final answer.

Why It Mattered

Process supervision proved to be vastly superior to outcome supervision. Using a PRM to search through generated solutions resulted in significantly higher accuracy on the notoriously difficult MATH dataset compared to using an ORM.

Crucially, Process Supervision aligns the model's internal reasoning with human logic. An ORM only cares that the model got the right answer, even if the reasoning is an uninterpretable mess. A PRM demands that the model gets the right answer for the right reasons, making the model's output far more interpretable, safer, and less prone to hallucination.

OpenAI open-sourced the dataset used in the paper (PRM800K), which contained 800,000 step-level human labels, providing a massive resource for the community to train their own process reward models.

What Came After

Process Reward Models (PRMs) became the secret sauce for the next generation of reasoning models. While Tree of Thoughts proved that tree search was a good idea, PRMs provided the ultra-accurate heuristic evaluator needed to guide that search effectively.

When OpenAI later released the o1 family of models, the community consensus was that they were trained using massive-scale reinforcement learning guided by advanced Process Reward Models. The concept of "verifying step by step" is now considered a mandatory component of any system attempting to solve complex, multi-step logical problems.