Zero-Shot Image Labels

Write each class as a sentence, embed them once, and label photos by nearest text. No training photos needed for new names.

Each photo is labeled by the class sentence it embeds closest to, so adding a class means writing sentences, not labeling photos.

Why Does This Exist?

New classes arrive faster than labels. A pet app adds a breed a week, and collecting thousands of training photos per breed each time is hopeless. Zero-shot classification reuses a joint image-text space instead: encode each class name as sentences once, embed each photo once, and label by nearest text. Naming plus matching replaces training plus labeling.

It assumes one prerequisite: an image-text embedding where cosine similarity means semantic closeness, the setup behind modern classifiers covered in image classification. This page covers prompt classification of whole images only. Open-vocabulary detectors that add boxes are a separate topic.

Think of It Like This

Sorting mail by reading name cards aloud

A sorter never met the tenants but holds name cards, each read aloud in eighty tones of voice. Every letter goes to the card it sounds closest to, and a new tenant needs only a new card, never a retraining of the sorter.

Photos are letters, prompt readings are the eighty tones, and cosine similarity is closeness of sound. Where it stops: tenant names are exact, while photo subjects vary by phrasing, so wording moves accuracy by points.

How It Actually Works

Encode the photo to a unit-norm vector $v$ . For each class name $c$ , encode $80$ prompt templates such as a photo of a cat, then average them into one text prototype $t_c$ . Averaging over templates damps wording noise. Score cosine similarities scaled by $\gamma$ , apply softmax, and take the argmax: the label is $\arg\max_c \mathrm{sim}(v, t_c)$ . Prompt tuning per domain with a few labeled shots is allowed, but then the setup is few-shot rather than zero-shot.

Worked example

A photo scores cat $0.31$ , dog $0.26$ , fox $0.24$ . With scale $\gamma = 100$ the logits are $31$ , $26$ , $24$ . Exponentials are $e^{31} \approx 2.90 \times 10^{13}$ , $e^{26} \approx 1.96 \times 10^{11}$ , $e^{24} \approx 2.65 \times 10^{10}$ , summing to about $2.93 \times 10^{13}$ . The cat probability is $2.90 / 2.93 \approx 0.992$ . The $0.05$ similarity gap becomes near-certainty after scaling, which is why $\gamma$ matters as much as the embeddings. At dataset scale, CLIP reaches $76.2$ percent top-1 on ImageNet with no training labels, about matching a supervised ResNet-50.

Watch Out For

One template deciding everything

A single a photo of a X prompt swings accuracy by points with article and plural changes, and classes flip on wording alone. The symptom is accuracy that collapses when a user rephrases a class name. Always ensemble the template set and test paraphrases before quoting a number.

Reading 76 percent as task-ready

ImageNet top-1 measures single centered objects, so the headline number overstates performance on cluttered, multi-object, or fine-grained photos. The symptom is a demo that shines on curated images and stumbles on phone photos. Re-measure on a sample of the deployment distribution before promising the headline figure.

The Quick Version

Zero-shot classification labels photos by nearest class sentence in a joint embedding.
Eighty templates per class average into one prototype that damps wording noise.
Worked numbers: similarities $0.31$ , $0.26$ , $0.24$ scale to cat probability $0.992$ .
CLIP reaches $76.2$ percent top-1 on ImageNet with no training photos.
Wording shifts accuracy by points, so ensemble prompts and re-measure on deployment photos.