Self-Training Uses Confident Labels

Self-training teaches a new model from the old model's most confident predictions, growing the labeled set without growing the labeling bill.

A teacher trained on 1000 labels predicts 5000 unlabeled frames, the 1800 predictions above 0.9 confidence join the retraining set, and accuracy climbs from 0.74 to 0.79.

Why Does This Exist?

Labeled images run out fast while unlabeled ones pile up: a thousand hand-tagged street frames against a hard drive of fifty thousand raw ones. Training only on the labeled slice wastes the pile, but trusting every model prediction poisons training with confident mistakes. Image classification with scarce labels is exactly this bind.

Self-training takes the middle path. Train a teacher on labeled data, predict the unlabeled pool, keep only the confident slice as pseudo-labels, and retrain a student on the union. This page covers that outer selection-and-retraining loop specifically; the umbrella of methods that learn from unlabeled data is semi-supervised learning, and the same loop under distribution shift needs domain adaptation.

Think of It Like This

A senior who grades practice papers

A senior solves a thousand problems with answer keys, then attempts five thousand more without keys and marks only the solutions they feel sure about for a junior to study. The junior trains on the solved thousand plus the confident attempts together, and next term the junior becomes the senior.

Confidence is the marking pen and retraining rounds are school terms. The analogy stops at calibration: seniors know the limits of their certainty far better than neural nets, whose raw softmax scores run overconfident and need their own checking against a labeled probe set.

How It Actually Works

The three-step loop

Round one trains the teacher on labeled images only. Round two runs the teacher over the unlabeled pool and keeps predictions above a threshold $\tau$ (tau, the confidence gate), commonly $\tau = 0.9$ , or the top- $k$ most confident predictions per class to protect rare classes. Round three retrains a student from scratch, or fine-tunes, on labeled plus pseudo-labeled data, sometimes with stronger augmentation on the pseudo slice so the student cannot just memorize the teacher. Repeat until validation gains flatten, refreshing pseudo-labels every round so early mistakes do not fossilize into the dataset.

Worked example

Start with $1{,}000$ labeled street images and a model at $0.74$ accuracy. The teacher predicts $5{,}000$ unlabeled frames; $1{,}800$ clear the $0.9$ gate, a keep share of $1{,}800 / 5{,}000 = 0.36$ . A labeled probe shows the kept slice is $0.88$ precise, so retraining adds about $1{,}800 \times 0.88 = 1{,}584$ correct labels and $216$ wrong ones to the original thousand. The student trains on $2{,}800$ mixed images and reaches $0.79$ . The $216$ wrong labels are the drift tax, paid willingly for $1{,}584$ free teachers.

Code

1def select_pseudo(max_probs: list[float], tau: float = 0.9) -> tuple[int, float]:2    """Count and share of predictions clearing the confidence gate."""3    kept = sum(1 for p in max_probs if p >= tau)  # hard gate per image4    return kept, kept / len(max_probs)5
6
7print(select_pseudo([0.95] * 1800 + [0.6] * 3200))8# -> (1800, 0.36)

Watch Out For

Retraining on the same frozen mistakes each round

Freezing pseudo-labels once and retraining for many rounds bakes early errors deeper each time. The symptom is accuracy rising for one round then sliding as wrong labels compound into the teacher. Refresh predictions every round, cap the pseudo slice weight below the labeled weight, and track probe precision of the kept slice, not just its size.

One global gate starving rare classes

A single $\tau = 0.9$ threshold keeps mostly easy majority-class frames, since the teacher is confident where data is dense. The symptom is a bigger dataset that still underperforms on the tail. Select top- $k$ per predicted class or set per-class thresholds from probe precision, so every class contributes pseudo-labels.

The Quick Version

Self-training loops teacher prediction, confident selection, and student retraining.
The gate sits at $\tau = 0.9$ or top- $k$ per class so the tail stays represented.
Worked numbers: $1{,}800$ of $5{,}000$ kept at $0.36$ share, accuracy $0.74$ to $0.79$ .
About $216$ wrong labels ride along as the drift tax on $1{,}584$ free correct ones.
Refresh pseudo-labels each round and weigh them below true labels.