DeiT Trains Vision Transformers Fast

DeiT adds a distillation token that copies a CNN teacher, so vision transformers learn ImageNet in days instead of months.

A distillation token learns the teacher soft targets beside the class token over 196 patches, and the pair lifts accuracy past either head alone

Why Does This Exist?

The plain vision transformer needs hundreds of millions of photos to beat convolutional networks. Most labs have $1.2$ M ImageNet photos and a few GPU-days, and on that budget the transformer underfits while the CNN cruises. The architecture was fine; the training was the problem.

DeiT keeps the transformer shape and changes how it learns. It pairs image classification with a convolutional teacher, borrows that teacher's built-in bias through distillation, and ships pretrained models others fine-tune. The convolutional layers page explains the bias being borrowed; the hybrid sequel lives in hybrid CNN transformer models.

Think of It Like This

A student with answer key plus tutor hints

You study past exams alone and score $79$ . Then a tutor shows the answers plus how confident each option is. You learn which distractors are close calls and which are absurd, and you jump to $81$ .

Patches are the questions, the class token is your answer, and the distillation token is the tutor hints. Where it stops: a wrong tutor teaches wrong confidence, just as a weak teacher caps the student.

How It Actually Works

Split a $224 \times 224$ image into $16 \times 16$ patches for $N = 196$ tokens. Prepend two extra tokens, the class token $c$ and the distillation token $d$ , plus position codes, and run the standard transformer. Both tokens train their own classifier heads. The loss mixes true labels and teacher soft targets:

L = 0.5\, CE(s_c, y) + 0.5\, KL(s_d / T, t / T)

Here $s_c$ and $s_d$ are the two student heads, $y$ is the true label, $t$ holds the teacher soft targets, $T$ is the temperature that softens both distributions, $CE$ is cross entropy and $KL$ is the divergence pulling the distillation head toward the teacher. Heavy augmentation (RandAugment plus Mixup plus CutMix) supplies the data variety that a private $300$ M-image dataset would otherwise provide.

Worked example

Take student logits $[2.0, 1.0, 0.1]$ . Exponentiate to $[7.389, 2.718, 1.105]$ , sum to $11.21$ , divide to get $[0.659, 0.242, 0.099]$ . The teacher gives $[0.70, 0.25, 0.05]$ . The cross term against the teacher is $-(0.70 \log 0.659 + 0.25 \log 0.242 + 0.05 \log 0.099)$ , which splits into $0.292 + 0.354 + 0.116 = 0.762$ . The distillation token chases $0.762$ while the class token chases the hard label, and the pair beats either head alone.

Code

The snippet below recomputes the worked example: softmax of the student logits, then the teacher-matching term. It prints what the prose claims.

1import math2
3
4def softmax(logits: list[float]) -> list[float]:5    exps = [math.exp(x) for x in logits]6    total = sum(exps)7    return [e / total for e in exps]8
9
10student = softmax([2.0, 1.0, 0.1])11print([round(p, 3) for p in student])  # -> [0.659, 0.242, 0.099]12
13teacher = [0.70, 0.25, 0.05]14ce = -sum(t * math.log(p) for t, p in zip(teacher, student))15print(round(ce, 3))  # -> 0.762

The $0.762$ is the distillation signal for this one sample: small when the student already shares the teacher's doubts, large when it is confidently different.

Watch Out For

Distilling from a weaker teacher than the student

A small convolutional teacher below the student capacity drags accuracy down instead of lifting it. Symptom: the distilled run underperforms plain hard-label training. Fix it by using a strong teacher at or above your target accuracy, or by turning the distillation weight down when the teacher top-1 trails the student by points.

Skipping the augmentation recipe

DeiT without RandAugment, Mixup and CutMix overfits ImageNet-1k fast, because the transformer has little built-in bias to restrain it. Symptom: training accuracy climbs while validation stalls tens of points below the reported numbers. Fix it by running the full recipe, or by starting from the released pretrained weights instead of training from scratch.

The Quick Version

DeiT adds a distillation token beside the class token over $196$ patch tokens, with one head per token.
The loss mixes hard labels and teacher soft targets with temperature scaling.
A sample with teacher $[0.70, 0.25, 0.05]$ contributes a distillation term of $0.762$ , teaching which distractors are close.
Strong augmentation replaces web-scale data, making $1.2$ M ImageNet photos enough.
Teacher quality caps the gains, so match the teacher to or above your target accuracy.