Depth Estimation from Images Guide

Turn flat photos into meters per pixel. Depth estimation is the umbrella that splits into stereo, single-image, and filled-sparse methods.

One depth estimation umbrella splits into stereo, monocular, and completion branches, and only the branches with a known baseline or lidar input answer in true meters

Why Does This Exist?

Robots need meters. Cameras record brightness. Depth estimation exists to bridge that gap in all its forms. Stereo vision triangulates from two views, monocular depth estimation regresses from one, and completion densifies sparse lidar. This overview maps the family so you pick the right branch; each branch then gets its own page.

The representation underneath all three is depth maps: one distance per pixel, plus intrinsics when you want 3D points.

Think of It Like This

Judging distance three ways

Cover one eye and judge by size memory; open both and feel the stereo snap; then tap a cane to fill the gaps. All three give distance, with different guarantees.

One eye is monocular, two eyes are stereo, and the cane is sparse lidar. The analogy stops at learning: people carry size memory while networks need training data for every new domain.

How It Actually Works

Stereo fixes scale through $Z = f B / d$ , with focal length $f$ , known baseline $B$ and disparity $d$ . Monocular models predict either metric depth $Z$ or affine-invariant disparity $1/Z$ up to an unknown scale $s$ and shift $t$ , resolved later by median alignment $s \times \hat{Z} + t$ against sparse truth. Completion takes sparse truth plus the image and regresses dense depth with a sparsity-aware encoder. Evaluation reports absolute relative error $\mathrm{AbsRel} = |Z - \hat{Z}| / Z$ and root mean square error, both over valid pixels only.

Worked example

A pixel with truth $Z = 4.0$ m and prediction $\hat{Z} = 4.4$ m gives $\mathrm{AbsRel} = 0.4 / 4.0 = 0.10$ . Over four pixels with errors $0.4$ , $0.2$ , $0.0$ and $0.6$ m, RMSE is $\sqrt{(0.16 + 0.04 + 0 + 0.36)/4} = \sqrt{0.14} \approx 0.374$ m. Stereo with $f = 800$ and $B = 0.12$ m at disparity $24$ px gives $Z = 800 \times 0.12 / 24 = 4.0$ m exactly, with no alignment step needed.

Code

Absolute relative error is the first score to compute, per pixel, before any averaging.

1def abs_rel(pred: float, gt: float) -> float:2    return abs(pred - gt) / gt  # scale-aware error per pixel3
4
5print(abs_rel(4.4, 4.0))  # -> 0.10000000000000009

The float dust past $0.1$ is ordinary binary rounding, not a bug. Average this quantity over valid pixels for the dataset score.

Watch Out For

Comparing metric and affine outputs raw

Affine models output disparity up to scale and shift, so their raw numbers look broken against meters. Symptom: near-perfect shapes with absurd AbsRel. Fix it by median-aligning predictions to ground truth before scoring, and never deploy affine depth without a scale source.

Scoring invalid pixels like sky

Sky, out-of-range and missing-lidar pixels carry no truth, yet they dominate the error if left in the average. Symptom: RMSE that swings wildly between scenes with different sky fractions. Fix it by masking to valid pixels before every metric, and report the valid-pixel fraction alongside the score.

The Quick Version

Depth estimation predicts meters per pixel from images plus optional sparse range.
Stereo gives metric depth by baseline, since $Z = f B / d$ ; monocular gives shape up to scale.
Completion densifies sparse lidar guided by the image.
AbsRel and RMSE over valid pixels only are the standard scores.
A $0.4$ m error at $4$ m is AbsRel $0.10$ ; four errors of $0.4$ , $0.2$ , $0.0$ and $0.6$ m give RMSE $0.374$ m.
Align affine outputs before comparing or deploying, and mask invalid pixels before scoring.