Anchor-Free Detection

Anchor-free detection predicts boxes from points or keypoints instead of preset templates, removing thousands of hand-tuned anchors per image.

One interior point plus four edge distances decodes directly into a box, with no anchor scales, aspect ratios, or IoU matching anywhere in the path

Why Does This Exist?

Anchor boxes tile every image with thousands of preset templates, each needing scales, aspect ratios, IoU thresholds, and dataset clustering. That machinery is brittle on new domains and pure overhead at deployment: move from street scenes to aerial or medical imagery and the whole template table needs re-tuning.

Anchor-free detection, rising from CornerNet in 2018 through FCOS and CenterNet in 2019, predicts geometry straight from feature locations: corners, centers, or per-pixel edge distances. Fewer knobs, matching accuracy, faster adaptation to odd datasets. The full pipeline around either choice is object detection.

Think of It Like This

Sketching freehand versus tracing stencils

Anchor methods trace: pick the closest of nine stencils per spot, then erase and adjust. Anchor-free methods sketch freehand: mark the corners or the middle, then draw the sides outward.

Stencils speed up beginners on familiar shapes and slow down everyone on strange ones. Freehand demands better drawing but handles any shape it meets, which is exactly the trade on unfamiliar datasets.

Where it stops: a sketch has no wrong answers, but a detector still needs supervision. Freehand here means freedom from templates, not freedom from labels: every corner, center, and distance is still learned from annotated boxes.

How It Actually Works

Three encodings cover nearly every anchor-free detector. CornerNet predicts top-left and bottom-right corner heatmaps plus embeddings that pair corners of the same object. CenterNet predicts one center heatmap per class and regresses width and height at each peak. FCOS skips keypoints entirely: every interior feature point regresses four distances to the left, top, right, and bottom edges, filtered by a centerness score that down-weights points near the rim.

Worked example: one point becomes a box

Take a feature point at $(160, 140)$ with predicted distances $l = 60$ , $t = 40$ , $r = 80$ , $b = 60$ . Subtract for the top-left corner, add for the bottom-right:

x_1 = 160 - 60 = 100, \quad y_1 = 140 - 40 = 100

x_2 = 160 + 80 = 240, \quad y_2 = 140 + 60 = 200

The box is $(100, 100, 240, 200)$ : 140 wide, 100 tall, decoded with four subtractions and additions. No template was consulted, no IoU threshold applied.

What disappears: anchor clustering, IoU matching bands, per-level scale tables. What stays: multi-scale pyramids, classification losses, and duplicate suppression, because neighboring points still fire on the same object. Anchor-free removes the template bureaucracy, not the detection fundamentals, which is why modern hybrids mix both ideas and why DETR pushes further by replacing suppression with learned queries.

Code

The point of this snippet is the decoding step: four edge distances from one interior point become a box with plain arithmetic.

1import numpy as np2
3point: np.ndarray = np.array([160.0, 140.0])  # feature point inside the object4ltrb: np.ndarray = np.array([60.0, 40.0, 80.0, 60.0])  # left, top, right, bottom5
6box: np.ndarray = np.array(7    [8        point[0] - ltrb[0],  # x1 = 160 - 60 = 1009        point[1] - ltrb[1],  # y1 = 140 - 40 = 10010        point[0] + ltrb[2],  # x2 = 160 + 80 = 24011        point[1] + ltrb[3],  # y2 = 140 + 60 = 20012    ]13)14print(box)  # -> [100. 100. 240. 200.]

The same four numbers the worked example derived by hand, now as the array the loss function would compare against the annotated box.

Watch Out For

Declaring anchors dead on every dataset

On extreme aspect ratios, like long thin defects or poles, tuned anchors still beat point methods because the template prior carries real information. The symptom is ragged boxes on skinny objects right after migrating to anchor-free, while ordinary objects look fine.

Fix it by benchmarking both families on your own boxes before committing. Let the data decide: anchor-free wins on adaptability, tuned anchors can win where shapes are strange but consistent.

Forgetting duplicates still need suppression

Anchor-free removes templates, not duplicate predictions: neighboring points still fire on the same object. The symptom is tight clusters of overlapping boxes around each detection instead of one clean box.

Fix it by keeping non-maximum suppression, Soft-NMS, or center-peak extraction in the pipeline. The detection is anchor-free; the cleanup is not optional.

The Quick Version

Anchor-free detection predicts boxes from corners, centers, or per-pixel edge distances instead of preset templates.
CornerNet pairs corners, CenterNet regresses size from centers, and FCOS regresses four edge distances per interior point.
The worked point (160, 140) with distances 60, 40, 80, 60 decodes to the box (100, 100, 240, 200).
Anchor scales, aspect ratios, and IoU matching thresholds disappear; pyramids, losses, and suppression stay.
Tuned anchors can still win on extreme aspect ratios, so benchmark both on each dataset.
Neighboring points fire duplicates, so NMS or peak extraction remains mandatory.