Gen AI

Reasoning Distillation

Instead of forcing a small model to figure out complex math on its own, you take a massive, brilliant model, ask it to write out its step-by-step logic, and train the small model to mimic that exact chain of thought.

A massive teacher model generates the reasoning traces (the 'think' tags), which are then used as supervised training data for a smaller student model.

Why Does This Exist?

Training a model to inherently "reason" (break a problem down, double-check itself, backtrack on errors) requires Reinforcement Learning (like RLVR). This is incredibly expensive, unstable, and typically only works well on models larger than 30B parameters. Small models (like an 8B parameter model) struggle to discover reasoning pathways on their own during RL.

However, once a massive model (like o1 or DeepSeek-R1) has learned how to reason, it can write down its thoughts. Reasoning distillation bypasses the expensive RL phase for smaller models. It takes the step-by-step thinking traces of the massive model and uses them as standard Supervised Fine-Tuning (SFT) data for a small model. The small model learns to mimic the thought process of the genius.

Think of It Like This

The Nobel Laureate and the High School Student

You want a high school student to solve a college-level physics problem.

Standard RL: You lock the student in a room with a textbook, tell them if their final answer is right or wrong, and force them to guess and check for weeks until they invent a methodology.

Reasoning Distillation: You hire a Nobel Laureate to solve the problem on a whiteboard, writing down every single step, mistake, and correction they make. You hand that whiteboard to the student and say, "Memorize this process." The student learns the methodology in an afternoon.

How It Actually Works

The Data Generation Pipeline

The Prompts: Gather 100,000 complex prompts (math, coding, logic puzzles).
The Teacher: Feed them to a massive, frontier reasoning model (e.g., DeepSeek-R1 671B).
The Traces: The Teacher outputs a highly structured response, usually enclosed in <think>...</think> tags, detailing its internal monologue, followed by the <answer>...</answer>.
The Filter: Verify the final answer. If it's correct, keep the entire trace. If it's wrong, throw it away (this is a form of Rejection Sampling).

The Distillation (SFT)

Take the curated dataset of 100,000 perfect thought traces and perform standard Supervised Fine-Tuning on a small, untrained base model (e.g., Llama 3 8B).

The student model doesn't need RL. It simply optimizes its weights to predict the next token in the teacher's thought process. Surprisingly, this is enough. The small model absorbs the "habit" of breaking problems down and correcting itself, leading to a massive spike in its logic benchmarks without ever undergoing reinforcement learning.

The Ceiling

Distillation is incredibly efficient, but a student cannot surpass its teacher. The student model will only ever be as good as the traces it was trained on. To push the absolute boundary of AI capabilities, you still have to run the expensive RLVR process on a massive model to generate better traces.

Watch Out For

Distilling hallucinated confidence

If the teacher model uses a flawed logic step but coincidentally arrives at the right answer, the verifier will keep the trace. The student model will then learn that flawed logic step. At scale, this can teach the small model to confidently assert nonsensical middle-steps to reach a memorized conclusion.

The Quick Version

Training a small model to reason from scratch using RL is difficult and expensive.
Reasoning distillation uses a massive, already-trained reasoning model (the Teacher) to generate step-by-step solutions (traces).
The small model (the Student) is trained via SFT to mimic these traces.
This allows small, cheap models to exhibit complex reasoning behaviors that normally require massive parameter counts.