Depth Maps from Single Images

A depth map stores one distance per pixel, which turns a photo into a 2.5D point cloud but leaves occlusions unknown.

Each pixel unprojects to one 3D point through the focal length, and the same one-pixel error grows from 0.0125 m nearby to 0.0625 m far away

Why Does This Exist?

Full 3D is heavy, but many tasks need only how far each pixel sits. Stereo vision triangulates depth from two views; monocular depth estimation regresses it from one. A depth map $D(u, v)$ in meters plus camera intrinsics unprojects into a point cloud, which is why detectors lift image features into bird's-eye view on top of depth. The limit is one layer: occluded backs have no pixel. The camera math underneath is image formation and camera model.

Think of It Like This

A theater seating chart with row numbers

Each seat has a row number but no note about who hides behind tall hats. You know how far every visible head sits, yet the hidden heads have no entry at all.

Pixels are seats and depth is the row. The analogy stops at scale: row numbers are exact while monocular depth needs a scale fix.

How It Actually Works

Pinhole unprojection is $X = (u - c_x) \cdot Z / f$ and $Y = (v - c_y) \cdot Z / f$ , with depth $Z = D(u, v)$ , focal length $f$ and principal point $(c_x, c_y)$ . Here $u$ and $v$ are pixel coordinates, and $X$ , $Y$ , $Z$ are meters in camera space. Monocular networks predict inverse depth $1/Z$ rather than $Z$ directly, which balances near precision against far range during training. Error grows near-quadratically with distance because one pixel spans more meters far away.

Worked example

Take $f = 800$ , $(c_x, c_y) = (640, 360)$ , pixel $(840, 360)$ and depth $Z = 10$ m. Then $X = (840 - 640) \times 10 / 800 = 2.5$ m and $Y = 0$ m. A one-pixel error to $u = 841$ gives $X = 2.5125$ m, off by $0.0125$ m. At $Z = 50$ m the same one-pixel slip gives $0.0625$ m, five times worse. That growth is why far bird's-eye-view cells smear.

Code

Unprojection plus the error-growth check, printing exactly what the worked example claims.

1def unproject(u: float, v: float, z: float, f: float, cx: float, cy: float) -> tuple[float, float, float]:2    x = (u - cx) * z / f3    y = (v - cy) * z / f4    return (x, y, z)5
6
7print(unproject(840, 360, 10.0, 800.0, 640.0, 360.0))  # -> (2.5, 0.0, 10.0)8print(unproject(841, 360, 10.0, 800.0, 640.0, 360.0))  # -> (2.5125, 0.0, 10.0)9print(unproject(841, 360, 50.0, 800.0, 640.0, 360.0))  # -> (12.5625, 0.0, 50.0)

The last two lines differ from their exact-pixel versions by $0.0125$ m and $0.0625$ m: the same pixel error costs five times more at $50$ m than at $10$ m.

Watch Out For

Trusting monocular scale across cameras

One camera's $10$ m is another's $11$ m without calibration, so fused clouds split. Symptom: doubled walls in bird's-eye-view grids. Fix the scale per camera on known baselines first.

Regressing depth instead of inverse depth

Predicting $Z$ directly spends most of the loss on far pixels and starves near precision. Symptom: crisp distant background with mushy nearby obstacles, the opposite of what driving needs. Fix it by predicting $1/Z$ and converting back only at the end.

The Quick Version

Depth maps store meters per pixel and unproject with intrinsics through $X = (u - c_x) \cdot Z / f$ .
One pixel covers more meters far away, so the same slip costs $0.0125$ m at $10$ m but $0.0625$ m at $50$ m.
Use depth when one visible layer suffices for driving or lifting to bird's-eye view.
Do not use it for backs of objects; occlusions have no pixels.
Predict inverse depth to balance near precision against far range.
Monocular scale stays ambiguous per camera until calibration or stereo fixes it.