Policy Distillation: Compressing Policies

Policy distillation compresses big teacher policies into one small fast student by copying soft action chances instead of learning from rewards alone.

Two frozen teachers at 0.70, 0.20, and 0.10 train one student at 0.50, 0.30, and 0.20 with KL 0.0851 nats

Why Does This Exist?

Big policies are painful to ship. You train one expert per game or robot task, and now you own ten large nets that each need memory, compute, and upkeep. Training one small net on all tasks from rewards sounds better, but joint reward training is jumpy: tasks fight over shared weights and the small net never reaches expert quality.

Distillation splits the job. First train big teachers normally, one per task, with no size pressure. Then freeze them and teach one compact student to copy how they act. The student never touches reward in this step. It watches states, asks each teacher what it would do, and matches that answer.

This page covers that second step: the teacher to student KL loss, the temperature trick that keeps ranking detail, and multi task fusion. For the policies being copied, start with actor critic architecture. For one strong teacher family, read Soft Actor Critic. For the bigger picture of deep policies, see deep reinforcement learning.

You need KL divergence before this page. KL measures how far a student mix p sits from a teacher mix q, and it punishes misses hardest where the teacher puts real weight.

Think of It Like This

The apprentice with three tasting spoons

Picture a kitchen with three master cooks. One owns pastry, one owns grill, one owns salads. Each carries years of habit in their hands.

Now send in one apprentice with a small notebook. You do not ask the apprentice to redo ten years of burnt pans and lucky saves. You ask each master to narrate while they work: not just serve this dish but this one first, that one close second, never the third. The apprentice writes the full ranking for every situation and practices until their own hands agree.

Temperature is how chatty the masters are. A curt master only names the winner, so the apprentice learns little. A chatty master ranks every option, so the apprentice learns which runner up is almost as good and which choice is poison.

Where the analogy stops: real teachers are frozen nets, not people, and they never adjust to the student. The student also copies mistakes faithfully, which is why the last section of this page matters so much.

How It Actually Works

Write the teacher as pi T(a given s) and the student as pi S(a given s) with far fewer weights. Both output chances over the same moves. Sample a state s, often by rolling the teacher or reading a replay store, read teacher mix q and student mix p, and push p toward q with:

$L(s) = D_{KL}(q \\parallel p) = \\sum_a q(a) \\log \\frac{q(a)}{p(a)}$

Order matters. This direction punishes the student hardest where q is large, so the top teacher move gets matched first. Cross entropy on the same soft targets gives the same gradients up to a constant, so you will see both names for one idea.

Why soft targets beat hard labels

An argmax label says action 1 wins and deletes the rest. A soft target says 0.70, 0.20, 0.10, which tells the student that action 2 beats action 3 by two to one. That ranking is the dark knowledge. It guides the student through states it has barely seen.

Temperature tau controls how much ranking survives. Soften logits z before softmax as:

$q_i = \\frac{\\exp(z_i / \\tau)}{\\sum_j \\exp(z_j / \\tau)}}$

With logits [2.0, 1.0, 0.0], tau 1 gives [0.6652, 0.2447, 0.0900]. At tau 2 you get [0.5065, 0.3072, 0.1863]. The winner still wins, but the third move doubles from 0.09 to 0.19, so the student keeps learning the full order. Use the same tau on both sides while training, then ship at tau 1.

Multi task fusion by cycling tasks

One student can absorb many teachers. Keep frozen experts, pick a task, sample its states, distill one batch, switch tasks, repeat. The shared trunk learns common structure while small task heads keep action spaces apart. Actor Mimic follows this pattern and adds a feature match so the student copies middle layer activity too, not just final chances.

Worked example

One state, three moves. Teacher q is [0.70, 0.20, 0.10]. Student p is [0.50, 0.30, 0.20]. Use natural logs term by term:

Top: 0.70 times log(0.70 divided by 0.50) equals 0.70 times log(1.40) equals 0.70 times 0.3365, which is 0.2355.
Runner up: 0.20 times log(0.20 divided by 0.30) equals 0.20 times log(0.6667) equals 0.20 times minus 0.4055, which is minus 0.0811.
Last: 0.10 times log(0.10 divided by 0.20) equals 0.10 times log(0.50) equals 0.10 times minus 0.6931, which is minus 0.0693.

Sum is 0.2355 minus 0.0811 minus 0.0693, which is 0.0851 nats, about 0.1228 bits. Almost all the pain sits in the first term because the student gives only 0.50 to a move the teacher loves at 0.70. Fix that gap first. The diagram draws these two bar sets with the KL total beneath them.

Code

This snippet recomputes the KL and the temperature lift.

1from math import exp, log2
3
4def softmax(logits: list[float], tau: float = 1.0) -> list[float]:5    """Softmax with temperature tau."""6    scaled = [z / tau for z in logits]7    peak = max(scaled)8    exps = [exp(v - peak) for v in scaled]9    total = sum(exps)10    return [e / total for e in exps]11
12
13def kl(q: list[float], p: list[float]) -> float:14    """KL divergence D(q || p) in nats."""15    return sum(qi * log(qi / pi) for qi, pi in zip(q, p))16
17
18teacher = [0.70, 0.20, 0.10]19student = [0.50, 0.30, 0.20]20print(f"{kl(teacher, student):.4f}")21# -> 0.085122print([f"{v:.4f}" for v in softmax([2.0, 1.0, 0.0], tau=2.0)])23# -> ['0.5065', '0.3072', '0.1863']

Watch Out For

Student capacity collapse

A tiny student cannot hold five experts. The symptom looks confusing: KL keeps falling on training batches, yet reward on the hardest task falls off a cliff while easy tasks look fine. You optimized the average and starved the hard task.

Fix it by checking per task reward, not just pooled loss. Grow the trunk, give each task its own output head, or sample hard tasks more often. If one task still lags after that, it needs its own student.

Teacher suboptimality lock-in

The student copies flaws as faithfully as skills. If the teacher brakes too late in one corner, the student brakes too late there too, and extra distillation batches never fix it because the target itself is wrong. The symptom is KL near zero while teacher and student fail the same new variation together.

Fix it with a short reward fine tune after distillation. Freeze nothing, let the student touch the environment, and mix a little teacher KL with real reward so the student can outgrow its master where the master was weak.

The Quick Version

Policy distillation trains a small student to copy frozen teachers by minimizing KL between soft action mixes.
Soft targets carry ranking detail that hard argmax labels delete, and temperature controls how much survives.
Multi task fusion cycles tasks through one shared student while teachers stay frozen.
The worked KL of 0.0851 nats comes almost entirely from the student underweighting the teacher top move.
A small student can collapse on hard tasks, and a student never beats a flawed teacher without reward fine tuning.