DETR Detects Sets Without Anchors

DETR treats detection as dealing a fixed hand of 100 guesses and pairing each truth with exactly one of them, so hand-tuned anchors and cleanup by NMS disappear.

One hundred learned queries decode into boxes and Hungarian matching pairs each truth with exactly one of them, so no anchors are proposed and no duplicates need removal

Why Does This Exist?

Classic detectors decide by hand where to look. Engineers tile the image with anchor boxes at several scales and aspect ratios, label each one by an IoU threshold, then prune the pile of overlapping survivors with non-maximum suppression. Every step carries a knob, and crowded scenes break the heuristics first: two overlapping people produce a thicket of near-duplicate boxes that no single NMS threshold separates cleanly.

DETR (Carion et al., 2020) deletes the whole pipeline. It predicts a fixed set of $N = 100$ boxes in one shot and pairs each ground truth with exactly one prediction, so duplicates are impossible by construction and there is nothing left for NMS to prune. The cost is training time: the matching must settle before the boxes sharpen, which takes roughly 500 epochs on COCO against the tens that a Faster R-CNN needs.

This page covers the set framing and the matching. The deformable attention that speeds it up lives in transformer-based detectors, and the general detection setup lives in object detection.

Think of It Like This

Assigned parking instead of an open lot

An open lot needs circling drivers plus a tow truck for double-parked cars: anyone may take any space, so conflicts are resolved after the fact. Assigned parking gives every resident exactly one bay, and an attendant matches each arriving car to its bay at the gate. Nothing is ever towed because nothing is ever double-parked.

The queries are the bays, the Hungarian algorithm is the attendant, and each ground-truth object is a car that gets exactly one spot. Where it stops: a building with too few bays overflows onto the street, just as too few queries miss objects in dense crowds — the fixed set size caps how much can be found.

How It Actually Works

A CNN backbone turns the image into feature maps of width $d = 256$ , a transformer encoder adds fixed sine positional encodings and mixes global context into them, and then the decoder starts from $N = 100$ learned object queries — slots with no image content yet, whose embeddings are trained. Each query attends to the encoder memory and outputs two things: a class distribution $\hat{p}$ over the classes plus a no-object class, and a box $\hat{b} = (cx, cy, w, h)$ with normalized center, width and height.

Training pairs truths to predictions before any gradient flows. For truth $i$ and prediction $j$ the match cost combines classification and box terms, $C_{ij} = -\hat{p}_j(c_i) + \lVert b_i - \hat{b}_j \rVert_1 + \text{GIoU}(b_i, \hat{b}_j)$ , where $c_i$ is the true class and GIoU is the generalized overlap penalty. The Hungarian algorithm finds the one-to-one assignment with the lowest total cost in $O(N^3)$ time, and the actual loss is computed only on those matched pairs; every unmatched query is supervised as no-object. Because each truth owns exactly one prediction, the network learns to never emit two boxes for one object.

Worked example

Take $N = 3$ queries and two truths: a cat at $b_1 = [0.2, 0.2, 0.4, 0.4]$ and a dog at $b_2 = [0.7, 0.7, 0.3, 0.3]$ . Suppose the three predictions produce this cost matrix, where each entry already sums its class and box terms:

C = \begin{bmatrix} -0.79 & 1.83 & 0.50 \\ 1.83 & -0.75 & 2.20 \\ 0 & 0 & 0 \end{bmatrix}

Here $-0.79 = -0.85 + 0.06$ (query 1 is confident it sees the cat and close on the box) and $-0.75 = -0.80 + 0.05$ (query 2 on the dog). The third row is all zeros: an unmatched query costs nothing because it answers no-object. Trying the assignments: pairs $(1, 1), (2, 2)$ total $-0.79 + -0.75 + 0 = -1.54$ , while $(1, 3), (2, 2)$ totals $0.50 + -0.75 + 0 = -0.25$ and every other pairing is positive. The winner is query 1 takes the cat, query 2 takes the dog, query 3 takes nothing — each truth paired once, no duplicates left for NMS to remove.

Code

The point of this snippet is the matching, not the network: brute-forcing all $3! = 6$ assignments on the worked example's matrix confirms the pairing the prose derived by hand.

1import itertools2
3cost = [4    [-0.79, 1.83, 0.50],5    [1.83, -0.75, 2.20],6    [0.0, 0.0, 0.0],7]8
9best = min(10    itertools.permutations(range(3)),11    key=lambda a: sum(cost[i][a[i]] for i in range(3)),12)13print(best)14# -> (0, 1, 2)15print(round(sum(cost[i][best[i]] for i in range(3)), 2))16# -> -1.54

Query 0 takes truth 0, query 1 takes truth 1, query 2 takes nothing, at total cost $-1.54$ . Real training uses the Hungarian algorithm instead of brute force because $N = 100$ has far too many permutations to enumerate.

Watch Out For

Training 50 epochs and calling DETR broken

DETR needs about 500 epochs to settle its matching before the boxes sharpen. The symptom is low mean average precision at 50 epochs blamed on the code or the data. Fix: train long, or move to deformable variants that converge several times faster, before judging the run.

Expecting small objects to work out of the box

Single-scale DETR underperforms on small objects because its encoder reads one low-resolution map, where a 12-pixel object is barely a smudge. The symptom is respectable overall AP with a collapsed AP on small objects. Fix: use multi-scale deformable attention, which samples several feature levels, rather than raising the input resolution and paying full self-attention cost on it.

The Quick Version

DETR predicts a fixed set of $N = 100$ boxes in one shot, with no anchors proposed anywhere.
The Hungarian algorithm pairs each ground truth with exactly one prediction in $O(N^3)$ time.
Unmatched queries are supervised as no-object, so duplicates never exist and NMS has nothing to prune.
The fixture pairing costs $-1.54$ : query 1 takes the cat, query 2 takes the dog, query 3 takes nothing.
Budget around 500 epochs or use deformable variants, and expect weak small objects from single-scale DETR.