Triangulation from Two Image Rays

Each camera turns a pixel into a ray in space. Triangulation is where those two rays come closest, and that closest point is the 3D answer.

Two back-projected rays from pixels 480 and 400 meet at depth 2.0 m, and a shifted pixel leaves a 1 pixel reprojection residual.

Why Does This Exist?

Matches alone are flat. Given a left pixel, a right pixel, and known cameras, something must lift the pair back into 3D, and that lifter is triangulation. Every structure from motion pipeline, every stereo vision depth map, and every SLAM map point passes through it. Without it there are correspondences but no shape.

This page covers the two-view lift and its residual. The match constraint lives in the fundamental matrix and the wider two-view setup in multi-view geometry; the joint refinement over many points lives in bundle adjustment.

Think of It Like This

Two laser pointers crossing in fog

Two friends shine laser pointers at the same balloon in fog. Each beam alone says direction only; where the beams cross says distance. When the fog jitters, the beams miss by a hair and the crossing becomes the shortest bridge between them.

Pixels are laser directions and triangulation is that bridge midpoint. The analogy stops at calibration: friends aim freely while cameras need known poses before any beam means anything.

How It Actually Works

The projection equation

A calibrated camera writes $x = P X$ with $P = K [R | t]$ , where $x$ is the homogeneous pixel, $X$ is the homogeneous 3D point, $K$ holds the focal length $f$ and center $cx$ , and $[R | t]$ is the camera pose. Back-projection inverts the direction half: pixel $u$ becomes the ray $X(Z) = (cx + (u - cx) \cdot Z / f, 0, Z)$ in the camera frame, a line of candidate depths $Z$ through the optical center. One ray says direction, never distance; distance needs the second ray.

The linear solve and the midpoint

The direct linear transform builds two rows per view from the cross product $x \times (P X) = 0$ and stacks them into a $4 \times 4$ system solved by SVD, taking the smallest singular vector as $X$ . In exact arithmetic the rays cross; in noise they skew past each other, so the answer becomes the midpoint of their common perpendicular, the shortest bridge between the lines. Quality is reported as reprojection residual: project $X$ back into each view and measure pixel distance $\lVert x - \hat{x} \rVert$ . Then comes the cheirality check: keep $X$ only when its camera-frame depth $Z$ is positive in both views.

Worked numbers

Use $f = 800$ , center $320$ , left camera at the origin and right camera at $x = 0.2$ m. Left pixel $u_1 = 480$ gives ray slope $(480 - 320) / 800 = 0.2$ ; right pixel $u_2 = 400$ from $x = 0.2$ gives $(400 - 320) / 800 = 0.1$ relative to its own center. Solving $Z \cdot 0.2 = 0.2 + Z \cdot 0.1$ gives $Z \cdot 0.1 = 0.2$ , so $Z = 2.0$ m and $X = 0.4$ m. Reprojecting $(0.4, 0, 2.0)$ returns exactly $(480, 400)$ : left $320 + 800 \cdot 0.4 / 2.0 = 480$ , right $320 + 800 \cdot (0.4 - 0.2) / 2.0 = 400$ , residual $0.0$ px. Shifting $u_2$ to $401$ moves the slope to $81 / 800 = 0.10125$ and the depth to $0.2 / (0.2 - 0.10125) \approx 2.03$ m, a 3 cm move from a 1 px nudge, which is why far points wobble.

Code

The point of this snippet is the closed-form depth for the rectified pair in the worked example: slope difference in the denominator, baseline on top.

1def triangulate_depth(u1: float, u2: float, f: float, cx: float, baseline: float) -> float:2    slope1 = (u1 - cx) / f  # left ray direction X / Z3    slope2 = (u2 - cx) / f  # right ray direction relative to its center4    return baseline / (slope1 - slope2)  # depths where the rays meet5
6
7print(triangulate_depth(480.0, 400.0, 800.0, 320.0, 0.2))8# -> 2.09print(round(triangulate_depth(480.0, 401.0, 800.0, 320.0, 0.2), 4))10# -> 2.0253

The first print replays the $2.0$ m answer. The second shows one extra pixel pushing depth to $2.0253$ m: tiny parallax converts pixel noise into meters.

Watch Out For

Triangulating points at tiny parallax

When the two rays are nearly parallel, a one-pixel shift moves depth by meters. The symptom is far points with huge variance that still report sub-pixel residuals. Reject points whose parallax angle sits under about one degree, or down-weight them heavily in the next optimization.

Skipping the cheirality test

Algebra returns an $X$ even when it sits behind a camera, and its reprojection error can look excellent. Downstream code then fuses ghost geometry. Always transform into each camera frame and require $Z > 0$ twice.

The Quick Version

Triangulation lifts one match plus two known cameras into one 3D point.
The linear method stacks $x \times (P X) = 0$ rows and solves by SVD.
Real rays skew, so the answer is the midpoint of their common perpendicular.
Reprojection residual in pixels is the quality number to watch.
Pixels $480$ and $400$ with $f$ $800$ and baseline $0.2$ m meet at $2.0$ m.
Tiny parallax and negative depth are the two traps to filter.