Bundle Adjustment for Maps and Poses

Initial cameras and points are roughly right but disagree by pixels. Bundle adjustment nudges all of them together until every 3D point lands on its photos.

Bundle adjustment shifts two cameras and three points together so that every projected ray lands closer to its observed pixel, cutting mean error from 2.4 to 0.35 pixels

Why Does This Exist?

Triangulation builds each 3D point from fixed cameras, but those cameras came from noisy matches too. Fixing one side while freezing the other leaves pixels on the table: better points want better cameras, and better cameras want better points. Bundle adjustment exists to solve both at once. It is the final polish inside every structure from motion pipeline and every visual SLAM backend, minimizing one shared reprojection error instead of many local ones.

The one idea you need first. Reprojection error is the pixel distance between where a 3D point is observed in a photo and where the current cameras and point predict it should appear. If that residual is unfamiliar, triangulation works one through before this page uses it everywhere.

This page covers the joint cost and the sparsity trick that makes it solvable. Single-point lifting lives in triangulation; drift repair across revisits lives in loop closure and pose graph optimization.

Think of It Like This

Tightening a tent with all ropes at once

A tent sags because no single rope is right. Pulling one rope tight just moves the sag elsewhere. You walk around loosening and tightening every rope a little until the canvas sits flat and every peg agrees.

Cameras are pegs, 3D points are canvas corners, and reprojection error is the slack. The analogy stops at convexity: tents settle anywhere flat, while the reprojection cost has narrow valleys that need damping to enter without overshooting.

How It Actually Works

The cost sums squared reprojection residuals over every observation: $C = \sum_{i,j} \|x_{ij} - \pi(P_i, X_j)\|^2$ , where $x_{ij}$ is the observed pixel of point $j$ in camera $i$ , $P_i$ is camera $i$ with its rotation, translation, and intrinsics, $X_j$ is 3D point $j$ , and $\pi$ is projection. Each residual touches exactly one camera and one point, so the Gauss-Newton Hessian has a block-arrow shape: the dense camera-camera blocks stay small while the huge point-point block is block-diagonal and cheap to invert. The Schur complement eliminates the points first, solves the small camera system, then back-substitutes for the points. Levenberg-Marquardt damping handles the valley floors, and a robust loss such as Huber down-weights mismatches so one bad match cannot drag the map.

Worked example

Ten points start with mean residual $2.4$ px, so $C = 10 \times 2.4^2 = 57.6$ . One damped step moves each camera rotation by $0.3$ degrees and each point by $0.02$ m, cutting the mean residual to $0.9$ px ( $C = 8.1$ ). A second step reaches $0.35$ px ( $C \approx 1.2$ ), a roughly $47$ -fold cost drop. Moving points alone stalls near $1.1$ px because the shared camera bias remains, which is exactly why the joint solve earns its keep.

Code

The point of this snippet is the cost itself: the mean residual is the diagnostic practitioners watch, and the sum of squares is what the solver minimizes.

1def reprojection_cost(residuals: list[float]) -> float:2    """Sum of squared pixel residuals over all observations."""3    return sum(r * r for r in residuals)4
5
6start: list[float] = [2.4] * 107final: list[float] = [0.35] * 108print(round(reprojection_cost(start), 3))  # -> 57.69print(round(reprojection_cost(final), 3))  # -> 1.22510print(round(reprojection_cost(start) / reprojection_cost(final), 2))  # -> 47.02

The ratio prints about $47$ : the joint solve cuts the cost forty-seven-fold where points-only refinement stalls.

Watch Out For

Running full bundle adjustment on every frame

A global solve over thousands of frames grows with the cube of the camera count without the Schur trick, and even with it, per-frame global solves miss real time by orders of magnitude. The symptom is a tracker that starts smooth then lags seconds behind. Use local windows per frame and save the global solve for loop closures.

Leaving outliers in the squared cost

One $20$ px mismatch contributes $400$ to $C$ and drags good cameras toward it, so the map bends to please the worst match. Wrap residuals in a Huber loss and re-check residuals after solving, or the adjustment optimizes the mistakes.

The Quick Version

Bundle adjustment refines all cameras and all points against one shared reprojection cost.
The cost is the sum of squared pixel residuals across every observation.
Sparsity plus the Schur complement makes the large system solvable by eliminating points first.
Damping and a robust loss keep the solver inside narrow valleys and away from outlier traps.
Mean error typically falls from whole pixels to sub-pixel in a few joint steps.
Solve locally per frame and globally only at loop closures.