BEVFormer Camera 3D Detection

BEVFormer turns six camera views into one bird's-eye map by letting each map cell ask the right image pixels for evidence.

BEVFormer lifts six surround-camera views into one 200 by 200 bird's-eye grid, where each cell samples a few image points with deformable attention and carries the past grid forward

Why Does This Exist?

Cameras are cheap and dense, but each view is flat. Lifting six surround views into 3D by first guessing depth smears distant objects, because a small depth error becomes a large position error far away. Stereo vision avoids guessing, but it needs overlapping views, which surround cameras mostly lack.

BEVFormer avoids explicit depth entirely. It keeps a learnable bird's-eye grid and lets each grid cell pull evidence from whichever image views see its location, using attention instead of projection-then-depth. The previous grid is carried forward too, so motion and occluded objects persist across frames.

The one idea you need first. Object detection finds boxes in a single image; BEVFormer asks where those boxes sit in a shared top-down map. If boxes, scores, and non-maximum suppression are unfamiliar, read that page first, then come back.

This page covers the BEV query mechanism: the grid, the two attentions, and the worked numbers. Pure LiDAR voxel methods are the alternative when centimeter depth accuracy matters more than camera cost.

Think of It Like This

Six security monitors and one floor plan

Six guards watch six monitors while you hold a floor plan of the shop. You point at one square of the plan and ask each guard only about the pixels that show that square, ignoring the rest of their screens. Then you update the square, remember last minute's plan as well, and move to the next square.

BEV cells are floor squares, cameras are monitors, and attention is the pointing. The analogy stops at geometry: the pointing is computed from calibrated camera projection, not by a person waving at screens.

How It Actually Works

The BEV grid holds $H = 200$ by $W = 200$ learnable queries covering about $100$ m, so each cell spans roughly $0.5$ m. Here $H$ and $W$ are the grid height and width in cells. Each query is updated in two attention steps per layer. First, temporal self-attention lets the query read the previous frame's BEV grid, aligned by ego motion, which carries velocity and keeps occluded objects alive. Second, spatial cross-attention samples the camera images: a 3D reference point $(x, y, z)$ above the cell is projected by each camera's calibration $P$ to pixel $(u, v)$ , and the query reads only $K = 4$ learned offsets around each landing point, weighted by predicted attention weights. Points landing outside a view are skipped, so each query touches a handful of pixels instead of six whole images.

Worked example

Take the query at cell $(100, 120)$ , which sits at $(0.0, 10.0)$ m in ego coordinates. Reference heights $z \in \{-1, 0, 2\}$ m project through the front camera to about $(u, v) = (812, 430)$ , $(810, 470)$ , and $(808, 540)$ in a $1600 \times 900$ image. Around one landing point the query samples $4$ offsets with weights $[0.5, 0.3, 0.15, 0.05]$ , and the sampled two-channel features blend to $[0.65, 0.35]$ . Combined across heights and views, the cell's feature moves from $[0.2, 0.1]$ to $[0.62, 0.44]$ , and the detection head emits car score $0.83$ with box center $(0.1, 10.2, 0.8)$ m.

Code

The point of this snippet is the sampling step: a few weighted reads replace a whole-image operation. The weights sum to $1$ , so the blend is a convex combination of the sampled features.

1import numpy as np2
3weights: np.ndarray = np.array([0.5, 0.3, 0.15, 0.05])4sampled: np.ndarray = np.array(5    [[0.8, 0.2], [0.6, 0.4], [0.4, 0.6], [0.2, 0.8]]6)7
8updated: np.ndarray = weights @ sampled9print(updated)  # -> [0.65 0.35]10print(weights.sum())  # -> 1.0

The query reads four pixels, not six images, and the update $[0.65, 0.35]$ is the number the worked example feeds into the cell feature.

Watch Out For

Stale history smearing fast motion

Carrying the full past BEV grid with ego-motion error above about $0.3$ m drags boxes behind moving cars. The symptom is velocity underestimated by roughly $2$ m/s on highway scenes. Re-align history with the exact ego pose before temporal attention instead of trusting raw odometry.

Reference heights missing low curbs

Sampling only $z \in \{0, 2\}$ m lets a $0.15$ m curb fall between reference planes, so no query ever looks at it. Include a below-ground reference such as $z = -1$ m so ground-level evidence reaches the queries that need it.

The Quick Version

BEVFormer keeps one bird's-eye query grid fed by all cameras through attention, with no explicit depth map.
Temporal self-attention carries the past grid forward, which gives velocity and survives occlusion.
Each query samples only a few pixels per view through deformable cross-attention, chosen by calibrated projection.
A $200$ by $200$ grid of $0.5$ m cells covers highway range without any per-pixel depth estimate.
Ego-pose errors above about $0.3$ m smear moving objects, so alignment matters more than grid size.
Where centimeter depth matters, LiDAR voxel methods still win over camera-only BEV.