Open Vocabulary Detection

Open vocabulary detection finds objects named in plain words instead of a fixed label list, scoring each image region against each phrase so new names work with no retraining.

One text phrase scores every proposed box by cosine similarity, so a new name finds its object with no retraining.

Why Does This Exist?

A standard detector only names the classes it trained on. A warehouse model taught 80 fixed labels walks past the one new machine part nobody labeled, because its final layer has no row for it. Adding each new object means collecting boxes and retraining, which never catches up with the long tail of rare things.

Open vocabulary detection breaks the fixed list. It trains region features and language features in one shared space using detection boxes plus phrase-to-box grounding pairs, so at test time any written phrase becomes a classifier on the spot. The object detection page covers boxes, scores and suppression; the concrete two-stage system lives in Grounding DINO. This page covers the scoring rule, the training mix and how the tail is graded.

The one idea you need first. A detector proposes boxes, then each box gets a feature vector. Closed models dot that vector with learned per-class rows. Open models dot it with the embedding of your phrase instead, so the vocabulary is decided at query time, not training time.

Think of It Like This

A lost-and-found desk that works from spoken descriptions

A lost-and-found desk holds hundreds of unclaimed items. Instead of memorizing a catalog, the clerk listens to your spoken description, walks the shelves, and holds up whatever matches best. A description heard for the first time still works, because matching happens at query time.

Phrases are the spoken descriptions, proposed boxes are the shelf items, and cosine similarity is the clerk judging fit. Where it stops: a clerk asks follow-up questions when two items match, while the detector scores each phrase once and keeps the top boxes.

How It Actually Works

Score every box against every phrase

Each proposed box $i$ carries a region embedding $b_i$ , a $d$ -dimensional vector ( $d$ is the feature length, a few hundred numbers). Each query phrase $c$ carries a text embedding $t_c$ from a language encoder, in the same space. The match score is cosine similarity:

$s_{ic} = \frac{b_i \cdot t_c}{\lVert b_i \rVert \lVert t_c \rVert}$

where $\cdot$ is the dot product and $\lVert \cdot \rVert$ is vector length, so $s_{ic}$ runs from $-1$ (opposite) to $1$ (identical direction). Boxes scoring above a threshold $\tau$ (a cutoff like $0.70$ ) survive per phrase, then non-maximum suppression (NMS, the standard step that keeps the best box and deletes its heavy overlaps) removes duplicates. Proposals stay class-agnostic, meaning no base-class filter runs before language sees them, so novel shapes reach the scoring step.

Train regions and phrases together

Training mixes two supervisions. Detection data supplies plain boxes with fixed labels, which teaches where things are. Grounding data supplies phrase-to-box pairs in full sentences, which teaches what words attach to which regions. The text encoder usually starts from a pretrained vision-language model, so rare words already have sane embeddings before detection training begins. Evaluation holds out whole classes: LVIS splits its 1,200-plus labels into frequent, common and rare, and rare-class average precision grades exactly the names the detector never trained on.

Worked toy scores

Take 3-dimensional embeddings and the phrase "striped scooter" with $t = [0.9, 0.4359, 0.0]$ , whose length is $\sqrt{0.81 + 0.1900} \approx 1.0000$ . Box $A$ embeds as $a = [0.8, 0.6, 0.0]$ with length $1.0$ :

$a \cdot t = 0.8 \times 0.9 + 0.6 \times 0.4359 = 0.72 + 0.2615 = 0.9815$

so $s_A = 0.9815 / (1.0 \times 1.0000) = 0.9815$ . Box $B$ embeds as $b = [0.2, 0.98, 0.0]$ with length $\sqrt{0.04 + 0.9604} = \sqrt{1.0004} \approx 1.0002$ :

$b \cdot t = 0.2 \times 0.9 + 0.98 \times 0.4359 = 0.18 + 0.4272 = 0.6072$

so $s_B = 0.6072 / 1.0002 = 0.6071$ . With $\tau = 0.70$ , box $A$ is kept and box $B$ is dropped: the phrase found its scooter in one scoring pass.

Code

1import math2
3def cosine(a, b):4    dot = sum(x * y for x, y in zip(a, b))5    na = math.sqrt(sum(x * x for x in a))6    nb = math.sqrt(sum(x * x for x in b))7    return dot / (na * nb)8
9box_a = [0.8, 0.6, 0.0]10box_b = [0.2, 0.98, 0.0]11phrase = [0.9, 0.4359, 0.0]12scores = {"A": round(cosine(box_a, phrase), 4), "B": round(cosine(box_b, phrase), 4)}13kept = [k for k, s in scores.items() if s >= 0.70]14print(scores)15# -> {'A': 0.9815, 'B': 0.6071}16print(kept)17# -> ['A']

Watch Out For

Proposals trained only on base classes

When the proposer learns from base-class boxes alone, it suppresses unfamiliar shapes before language ever scores them. The symptom is rare-class recall near zero while base-class numbers look healthy. Fix it by training class-agnostic proposers on broad box data and measuring proposal recall on held-out classes, not just final precision on base ones.

Thresholds tuned on base names misfiring on new ones

A cutoff $\tau$ picked for familiar phrases can flood or starve new ones, because similarity scales shift across words. The symptom is a new query returning either the whole image or nothing. Fix it by calibrating $\tau$ on a validation set of held-out phrases and running suppression separately per phrase rather than once globally.

The Quick Version

Open vocabulary detection replaces fixed class rows with phrase embeddings scored by cosine similarity.
Any written phrase becomes a classifier at query time, so new names need no retraining.
Training mixes detection boxes (where things are) with grounding pairs (which words attach where).
Proposals must stay class-agnostic or novel shapes die before scoring.
LVIS rare-class precision grades exactly the names missing from training.