Fundamental Matrix of Two Camera Views

The fundamental matrix is the 3 by 3 table that encodes how two uncalibrated cameras sit relative to each other. It turns a pixel in one image into a search line in the other, and the true match always scores zero.

F turns a left pixel into a right epipolar line, and the true match makes the constraint score exactly zero

Why Does This Exist?

Matching a pixel across two views by searching the whole second image is slow and ambiguous. Epipolar geometry cuts the search to one line, and the fundamental matrix $F$ is that line in algebraic form, needing no calibration data at all. It is the workhorse behind uncalibrated reconstruction: estimate $F$ from matches, then recover cameras and structure. Its calibrated sibling is the essential matrix, and the full pipeline continues in structure from motion.

Think of It Like This

Two witnesses, one lineup

Witness A says the suspect stood third from the left. Witness B, standing elsewhere, cannot reuse third from the left, but A can draw B a sight-line across the lineup room: the suspect must stand somewhere along that line. B still has to look, yet the search collapses from a room to a line.

$F$ is the written sight-line rule between two cameras. Where it stops: the line says where to look, while recognizing the right face along it remains its own matching problem.

How It Actually Works

For corresponding homogeneous pixels $x$ and $x'$ , the epipolar constraint reads $x'^T F x = 0$ . Here $F$ is $3 \times 3$ with rank $2$ : its left and right null spaces are the two epipoles, which is why every epipolar line passes through its epipole. Counting freedoms gives $9 - 1$ (global scale) $- 1$ (rank constraint) $= 7$ degrees of freedom, so seven point pairs determine $F$ in principle.

In practice the $8$ -point algorithm stacks one linear row per match and solves by SVD, then projects the answer onto rank $2$ by zeroing its smallest singular value. RANSAC wraps the solver to survive mismatches. With intrinsics $K$ and $K'$ known, $F$ upgrades to $E = K'^T F K$ .

Worked example

Pure translation along the $x$ axis gives $F$ with rows $(0,0,0)$ , $(0,0,-1)$ , $(0,1,0)$ . Take $x = (2, 1, 1)$ :

l' = F x = (0,\ -1,\ 1)

so the epipolar line is $0 \cdot u - 1 \cdot v + 1 = 0$ , the horizontal line $v = 1$ . The true match $x' = (5, 1, 1)$ lies on it, and the constraint confirms:

x'^T F x = 5 \times 0 + 1 \times (-1) + 1 \times 1 = 0

A wrong match such as $(5, 2, 1)$ scores $5 \times 0 + 2 \times (-1) + 1 \times 1 = -1$ , nonzero and therefore rejected.

Code

The point of this snippet is the constraint check in standard-library-only code: it replays the worked example residual of 0.0 for the true match and -1.0 for the wrong one.

1def epipolar_residual(xp, x):2    f = ((0.0, 0.0, 0.0), (0.0, 0.0, -1.0), (0.0, 1.0, 0.0))3    line = [sum(f[r][c] * x[c] for c in range(3)) for r in range(3)]4    return sum(xp[r] * line[r] for r in range(3))5
6print(epipolar_residual((5.0, 1.0, 1.0), (2.0, 1.0, 1.0)))7# -> 0.08print(epipolar_residual((5.0, 2.0, 1.0), (2.0, 1.0, 1.0)))9# -> -1.0

The first call lands on exactly 0.0 because $(5, 1)$ sits on the line $v = 1$ , while the second returns $-1.0$ because $(5, 2)$ misses it by one row.

Watch Out For

Estimating F from a flat scene

If every matched point lies on one plane, a homography explains all matches and $F$ becomes degenerate: the solver returns confidently with tiny residuals while the epipolar lines are fiction. The symptom is plausible RANSAC scores with absurd geometry. Test a homography first on suspiciously flat scenes, and estimate the plane map instead, as homography estimation does.

Skipping the rank-2 projection

The raw $8$ -point answer is full rank because of noise, but a true fundamental matrix has a zero singular value. Without zeroing the smallest one, epipolar lines systematically miss their matches by a few pixels and downstream triangulation inherits the bias. One SVD truncation fixes it: solve, decompose, zero, rebuild.

The Quick Version

The fundamental matrix encodes uncalibrated two-view geometry in one $3 \times 3$ rank-2 matrix.
The epipolar constraint $x'^T F x = 0$ tests any candidate match algebraically.
The worked check proves it: $(5, 1)$ scores 0 on the line $v = 1$ , while $(5, 2)$ scores $-1$ .
$F$ has $7$ degrees of freedom: nine entries minus scale minus the rank constraint.
The $8$ -point algorithm plus SVD rank projection is the standard estimator.
Flat scenes make $F$ degenerate; a homography owns that case instead.