DeepSeekMath (GRPO)

The Problem

Reinforcement Learning from Human Feedback (RLHF) via PPO (Proximal Policy Optimization) was incredibly effective for training reasoning models, but it had a massive hardware cost.

Standard PPO requires maintaining a "Value Model" in memory during training. The Value Model's job is to look at a prompt and estimate how much reward the policy model should expect to get. This acts as a "baseline." If the policy model generates an answer that scores higher than the baseline, the action is encouraged; if it scores lower, it's penalized.

The problem is that the Value Model is typically the exact same size as the Policy Model. If you are training a 70-billion parameter model, you need to hold another 70-billion parameter Value Model in memory, plus the gradients, plus the optimizer states. This extreme memory overhead meant that large-scale PPO was restricted to labs with massive GPU clusters.

The Idea

The DeepSeek team proposed Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm designed specifically to eliminate the memory overhead of the Value Model.

The core insight of GRPO is that you don't need a massive neural network to estimate a baseline reward. Instead, you can calculate the baseline dynamically by generating a group of responses to the same prompt and comparing them against each other.

If the model generates 5 different answers to a math problem, you simply calculate the reward for each of the 5 answers (e.g., using a rule-based checker or a reward model). You then calculate the average reward of that group. Responses that scored above the group average are given a positive advantage (encouraged), and responses that scored below the group average are given a negative advantage (penalized).

How It Works

GRPO drastically simplifies the memory footprint of the RL loop:

1. Group Generation: The Policy Model receives a prompt (e.g., a math question) and samples a group of $G$ different outputs (e.g., 4 or 8 different reasoning paths and answers).
2. Reward Scoring: Each of the $G$ outputs is assigned a reward score. In domains like mathematics or coding, this doesn't even require a neural Reward Model; you can use a deterministic rule-based checker (e.g., did the output extract the correct final number? Did the code pass the unit tests?).
3. Relative Advantage: The rewards for the group are normalized. The average reward is subtracted from each individual reward to calculate the "advantage."
4. Policy Update: The Policy Model weights are updated. Outputs with a positive advantage have their probability increased; outputs with a negative advantage have their probability decreased. A KL-divergence penalty is still applied to prevent the model from drifting too far from its original state.

Because the baseline is calculated purely via the statistics of the group's rewards, the massive Value Model is entirely deleted from the architecture.

Why It Mattered

GRPO slashed the memory requirements for reinforcement learning by nearly half. This allowed researchers to train significantly larger reasoning models on the same hardware budget.

Using GRPO, the DeepSeek team trained DeepSeekMath 7B, a small model that matched the mathematical reasoning capabilities of massive frontier models like Gemini Pro and GPT-3.5. It proved that the standard PPO architecture was unnecessarily bloated for tasks where rewards could be evaluated objectively (like math and code).

What Came After

GRPO remained a somewhat niche technique until 2025, when it became the foundational training algorithm for DeepSeek-R1.

By scaling up GRPO and removing the neural reward model entirely in favor of rule-based logical verifiers, DeepSeek was able to train a world-class "thinking" model (R1) at a fraction of the cost of OpenAI's o1. GRPO proved that massive-scale reinforcement learning could be done efficiently, triggering a massive shift in how the open-source community approached post-training for reasoning models.