DINOv2 Teacher Student at Scale

Train a student to copy a slow teacher on big and small crops, at both image and patch level. Scale plus curation makes frozen features excel.

A student vision transformer copies a slow momentum teacher on whole-image and patch tokens across 142 million curated photos, so frozen features serve dense tasks

Why Does This Exist?

Small self-supervised runs give good image tags but weak dense maps: the network learns what the photo shows yet cannot say where anything is. DINOv2 (Oquab et al., 2023) scales data to $142$ million curated photos and adds a patch-level match, so frozen features serve depth, segmentation, and retrieval with no tuning. A pretrained backbone becomes a general tool, and semantic segmentation probes show the dense gain.

This page covers student-teacher training plus the patch loss only. Masked rebuilding variants such as MAE are separate pages in this wave.

The one idea you need first.

Self-distillation means the teacher is a slow copy of the student, updated by momentum rather than gradients. The class token $c$ names the whole image; the patch tokens $p_1$ through $p_N$ name small regions of it. DINOv2 matches both.

Think of It Like This

An apprentice copying a master on big and small tiles

An apprentice copies a master mosaic on the full wall and on single tiles. The master works slowly and never explains, yet the apprentice learns both scene sense and tile craft by matching at two scales.

Student and teacher are the pair, class loss is the wall, and patch loss is the tile. Where it stops: mosaics stay still, while training keeps changing the views with crops and blur, so the apprentice must match a moving target.

How It Actually Works

From each photo cut two global crops and several local crops. The student predicts the teacher's sharpened distributions: cross-entropy on the class token for the whole-image match, and a second head matching masked patch tokens for the dense match. The teacher's weights follow the student by exponential moving average, kept from collapse by centering and sharpening its outputs. Data curation into LVD-142M removes near-duplicates and balances scenes across the $142$ million images.

In plain words: global crops teach layout, local crops teach detail, the patch loss forces every tile to mean something, and curation keeps the $142$ million useful instead of repetitive.

Worked example

Say class logits student $[2.0, 1.0, 0.1]$ versus teacher soft target $[0.70, 0.25, 0.05]$ . Exponentiate the student: $[7.389, 2.718, 1.105]$ , summing to $11.212$ , so the student softmax is $[0.659, 0.242, 0.099]$ . Cross-entropy against the teacher is $-(0.70 \log 0.659 + 0.25 \log 0.242 + 0.05 \log 0.099) = 0.292 + 0.354 + 0.116 = 0.762$ . Patch terms add a similar $0.7$ per masked patch, so dense detail carries real weight in the total.

Code

The point of this snippet is to reproduce the worked example's $0.762$ exactly, so the loss arithmetic above can be checked rather than trusted.

1import math2
3student = [2.0, 1.0, 0.1]4teacher = [0.70, 0.25, 0.05]5
6exp = [math.exp(x) for x in student]7soft = [x / sum(exp) for x in exp]8print([round(x, 3) for x in soft])9# -> [0.659, 0.242, 0.099]10print(round(-sum(t * math.log(s) for t, s in zip(teacher, soft)), 3))11# -> 0.762

Watch Out For

Using class features for dense tasks

Class tokens pool the scene and smear boundaries. Depth maps then look blobby. The symptom is fine retrieval with weak segmentation. Read patch tokens for dense heads and keep class tokens for global search in image retrieval.

Fully fine-tuning a giant backbone on tiny data

Unfreezing a ViT-g on a few hundred labelled images rewrites the pretrained features before the head learns anything. The symptom is worse accuracy than a frozen baseline that took minutes. Fix: start from a linear probe on frozen features, and only unfreeze the last blocks with a small learning rate once that baseline holds.

The Quick Version

DINOv2 distills class plus patch tokens from a momentum teacher.
Training spans 142 million curated photos with global and local crops.
Patch loss is what lifts dense tasks like depth and segmentation.
Frozen backbones work without fine-tuning.
Curation matters as much as scale.