3D Object Detection

3D detection draws boxes with length, width, height and heading, not flat rectangles, lifting 2D finds into metric cuboids a planner can avoid.

A 4 m car shifted 0.6 m still scores 1D IoU 0.74 and passes the 0.7 bar, while a 30 degree heading error sinks it.

Why Does This Exist?

Driving needs distance plus size plus direction: a 2D box cannot tell a truck beside you from a billboard ahead. 3D detection exists to output $(x, y, z, l, w, h, \theta)$ cuboids in meters that a planner can brake around or steer past. Getting any of the seven numbers wrong puts the box in the wrong lane, at the wrong distance, or facing the wrong way.

This page covers how those metric boxes are predicted from lidar or cameras, and how they are scored. If flat boxes are new, start with object detection, which builds the proposal-plus-regression picture this page lifts into three dimensions. The unordered 3D points the lidar path consumes are covered in point clouds. Combining both sensors is the sequel in sensor fusion.

Think of It Like This

Parking with shoebox outlines

A parking camera draws shoeboxes around cars, including which way each nose points. Two drawings of the same car that overlap loosely fail the test; only snug, same-direction boxes pass.

The boxes are cuboids and the overlap test is 3D IoU. The analogy stops at height: cameras guess height while lidar measures it, which is why the two sensors differ so much in accuracy and cost.

How It Actually Works

From sensor to cuboid

Single-stage heads voxelize or pillar-encode the cloud, run a 3D or 2D backbone, then regress box residuals against anchors plus heading bins that handle angle wrap. Two-stage variants propose coarsely and refine with point-level features. Here $(x, y, z)$ is the box center in meters in the sensor frame, $(l, w, h)$ are length, width and height in meters, and $\theta$ is the heading angle around the vertical axis. Matching predictions to truth uses 3D IoU, with KITTI thresholds $0.7$ for cars and $0.5$ for pedestrians; bird's-eye-view AP drops height for a fast planning score. Duplicate removal runs non-maximum suppression in 3D or BEV, as drawn in the non-maximum suppression visual.

Worked example: 0.6 m of slack, 30 degrees of failure

Take a $4.0$ m car with a perfect box except a $0.6$ m shift along its length. Along that axis truth spans $[0.0, 4.0]$ and prediction spans $[0.6, 4.6]$ : the overlap is $4.0 - 0.6 = 3.4$ m and the union is $4.0 + 4.0 - 3.4 = 4.6$ m, so the 1D IoU is $3.4 / 4.6 \approx 0.74$ . That clears the $0.7$ car bar, which is exactly what the description means by a 4 m car allowing 0.6 m of error.

Now keep the center perfect but rotate the prediction $30$ degrees. On a $4.0 \times 1.8$ m footprint the corners swing out of the true rectangle and the overlap collapses below $0.5$ despite a perfect center, so the same detection fails. Heading errors sink IoU faster than center errors. For scale, moderate car AP sits near $80$ percent for lidar but near $20$ percent for monocular cameras, which is the accuracy gap the sensor choice buys.

Code

This snippet computes the length-axis IoU from the worked example so you can check the 0.74 by hand:

1def iou_1d(a0: float, a1: float, b0: float, b1: float) -> float:2    inter = max(0.0, min(a1, b1) - max(a0, b0))  # overlap length3    union = (a1 - a0) + (b1 - b0) - inter  # sum of lengths minus overlap4    return inter / union5
6
7print(round(iou_1d(0.0, 4.0, 0.6, 4.6), 2))8# -> 0.74

Watch Out For

Judging 3D by 2D AP alone

High 2D AP with poor depth passes image tests while cars sit meters off. The symptom is great demos with harsh braking in the vehicle. Always report 3D IoU and BEV AP with heading included, never 2D AP alone, for anything that drives.

Heading wrap punished as a huge error

Angles wrap: $179$ degrees and $-179$ degrees are 2 degrees apart, but a naive angle loss sees a 358-degree gap. The symptom is training that oscillates on nearly symmetric viewpoints and boxes that flicker direction frame to frame. Predict heading bins plus a residual, or use a circular loss, instead of regressing raw degrees.

The Quick Version

3D detection outputs metric cuboids $(x, y, z, l, w, h, \theta)$ with a heading angle planners can act on.
Lidar leads on accuracy near 80 percent moderate car AP; monocular cameras lead on cost near 20 percent.
KITTI passes cars at 3D IoU 0.7 and pedestrians at 0.5.
A 4.0 m car shifted 0.6 m scores IoU 0.74 and passes; a 30-degree heading error fails the same box.
BEV AP drops height for a fast planning score.
Report 3D metrics with heading, never 2D AP alone, for driving.