Sensor Fusion

Sensor fusion combines camera detail with lidar depth and radar velocity, so the car keeps working when any single sensor degrades.

Camera, lidar, and radar report one car with different noise, and inverse-variance fusion lands the track at 40.36 m near the lidar reading.

Why Does This Exist?

Cameras read texture but guess depth. Lidar measures depth directly but its returns thin with range and scatter in heavy spray. Radar reports velocity straight from the Doppler shift but localizes poorly sideways. Any single sensor has weather or a range where it quits, so a stack built on one sensor fails exactly when driving gets hard.

Sensor fusion exists to arrange complementary quits. Glare blinds the camera while lidar still holds shape; spray scatters lidar while radar still holds speed. The fused track outlives each sensor alone because at least one modality stays informative in every condition.

This page covers the fusion choice itself: early, late, and middle fusion plus the noise-weighted math that blends them. The two-camera depth prerequisite is stereo vision, and the bird's-eye-view architecture that consumes fused features is BEVFormer.

Think of It Like This

Three witnesses with different strengths

One witness remembers faces, one paces distances exactly, and one hears how fast things move. Alone each misleads sometimes, but together they convict, because the detective weights each claim by how reliable that witness is in these conditions. Fog makes the face-witness unreliable, so the distance-witness carries the case that night.

The car treats cameras, lidar, and radar the same way, weighting each by its noise. The analogy stops at testimony: witnesses never need microsecond timing and millimeter calibration to agree with each other, while sensors do.

How It Actually Works

Every observation $x_i$ arrives with a variance $\sigma_i^2$ stating its noise. The fused estimate is the inverse-variance weighted mean

\hat{x} = \frac{\sum_i x_i / \sigma_i^2}{\sum_i 1 / \sigma_i^2}

with fused variance $1 / \sum_i 1 / \sigma_i^2$ , which sits below every single input variance. Here $x_i$ is one sensor's reading, $\sigma_i^2$ is how noisy that sensor is right now, and the weights $1 / \sigma_i^2$ hand leadership to whichever sensor is cleanest. Three placements exist: early fusion concatenates raw or voxel features before detection, late fusion runs per-sensor detectors then associates their tracks, and middle fusion shares bird's-eye-view features, the currently favored point. Time sync aligns timestamps and extrinsic calibration aligns coordinate frames before any blending happens.

Worked example

A car sits near 40 m. The camera reports lateral position 40.8 m with variance 0.64, and lidar reports 40.3 m with variance 0.09. Weights are $1 / 0.64 = 1.5625$ and $1 / 0.09 = 11.1111$ , summing to 12.6736. The fused mean is $(1.5625 \times 40.8 + 11.1111 \times 40.3) / 12.6736 = 511.5278 / 12.6736 = 40.362$ m, pulled close to the lidar reading, with fused variance $1 / 12.6736 = 0.0789$ , tighter than either input. Radar contributes almost nothing to lateral position here yet dominates the velocity channel, which is exactly the point: each sensor leads where its noise is lowest, and the weights move leadership automatically as conditions change.

Code

1def fuse(xs, variances):2    weights = [1.0 / v for v in variances]3    total = sum(weights)4    estimate = sum(x * w for x, w in zip(xs, weights)) / total5    return round(estimate, 3), round(1.0 / total, 4)6
7
8print(fuse([40.8, 40.3], [0.64, 0.09]))9# -> (40.362, 0.0789)

The lidar variance of 0.09 outweighs the camera variance of 0.64 about seven to one, so the estimate lands near the lidar reading with uncertainty below both inputs.

Watch Out For

Silent mistiming that bends velocity

A 50 ms camera-lidar offset at 15 m per s shifts associations by 0.75 m, so the filter invents lateral velocity to explain the gap. The symptom is oscillating headings on straight roads. Timestamp at the sensor, compensate for exposure and sweep time, and monitor delay residuals instead of assuming zero.

Cookbook covariances that ignore weather

A fixed noise matrix tuned on sunny data keeps trusting lidar inside spray and the camera inside glare. The symptom is confident wrong tracks that appear only in bad weather. Make each variance a function of measured conditions, such as return density and exposure, so the weights demote the degraded sensor on their own.

The Quick Version

Cameras give texture, lidar gives depth, and radar gives velocity directly.
Fusion places every reading in one frame at one time before deciding anything.
Inverse-variance weights hand leadership to the cleanest sensor per condition.
Calibration and sync failures look exactly like perception failures downstream.
Design for complementary degradation, not for average accuracy.