Floating Point Arithmetic

What happens when you actually add and multiply rounded numbers. Tiny errors cancel, accumulate, or explode depending on the order of operations.

Adding small numbers first keeps the 0.49 alive to reach 2000.49, while adding big first rounds it away and leaves 2000

Why Does This Exist?

Formats decide which numbers exist. Arithmetic decides what happens when you compute with them. Every operation rounds its true result to the nearest representable neighbour, and those roundings interact: sometimes they cancel harmlessly, sometimes they accumulate into a wrong answer with a confident face.

Two facts organize everything. Relative error per operation stays near machine epsilon, $\epsilon = 2^{-52} \approx 2.2 \times 10^{-16}$ in double precision, but absolute error scales with operand size. So adding a tiny number to a huge one rounds the tiny one away entirely, while subtracting nearly equal numbers promotes old rounding dust into the full answer. The formats live in floating point formats; the failure catalogue they cause lives in numerical stability.

The one idea you need first

A double carries about 16 significant decimal digits. Machine epsilon $\epsilon \approx 2.2 \times 10^{-16}$ is the gap between 1 and the next double above it. Near 1, operations round to multiples of $\epsilon$ . Near $10^{16}$ , consecutive doubles sit 2 apart, so fractional parts stop existing and $10^{16} + 1$ equals $10^{16}$ .

Think of It Like This

A cash register that rounds every line

A register rounds each line item to the nearest dollar as it scans, then totals the rounded lines. A 49-cent pack of gum scanned beside a 2000-dollar television contributes nothing: swallowed by that line's rounding.

Scan gum with gum first and the cents accumulate into dollars before meeting the television, surviving the rounding. Same items, different order, different total. Catastrophic cancellation is the mirror image: two nearly equal totals subtracted, where the big matching dollars cancel and the answer is built entirely from accumulated rounding dust. Where the analogy stops: binary rounding scales with magnitude rather than rounding every line by the same dollar.

How It Actually Works

Three mechanisms cover most incidents. Here $a$ is a large value, $b$ is a small one, and $\delta$ marks rounding already carried:

Absorption. $a + b$ rounds to $a$ when $|b| < \epsilon |a| / 2$ . In doubles, $1 + 10^{-16} = 1$ because $10^{-16}$ sits below $\epsilon / 2 \approx 1.1 \times 10^{-16}$ . Small change dies beside big totals.
Catastrophic cancellation. $(a + \delta_{1}) - (a + \delta_{2})$ cancels the $a$ values and leaves $\delta_{1} - \delta_{2}$ promoted to the full answer. Every operation rounded correctly, yet relative error explodes because the true result is tiny.
Summation order. Adding smallest first, pairwise, or with Kahan compensation keeps small contributions alive. Well-ordered error grows like $O(n \epsilon)$ , random rounding like $O(\sqrt{n} \epsilon)$ , and adversarial ordering arbitrarily badly.

Worked example

The classic one-liner in double precision:

(0.1 + 0.2) - 0.3 = 5.55 \times 10^{-17} \neq 0

Neither 0.1 nor 0.2 exists in binary, so each rounds to its nearest double first. Those roundings survive the addition, then survive the subtraction because 0.3 rounds slightly differently, leaving residue $5.55 \times 10^{-17}$ . It is pure rounding archaeology, not a language bug.

Meanwhile $(1 + 10^{-16}) - 1 = 0.0$ . Absorption kills the $10^{-16}$ before subtraction even sees it, since $10^{-16} < \epsilon / 2$ . One example leaves dust behind; the other erases the signal first. Both follow from the same epsilon.

Code

The point is the check: confirm both phenomena in three lines before designing any float comparison, using standard-library arithmetic only.

1print((0.1 + 0.2) - 0.3)2# -> 5.551115123125783e-173print((0.1 + 0.2) == 0.3)4# -> False5print((1 + 1e-16) - 1)6# -> 0.0

Equality on floats is almost always wrong. Compare with a tolerance such as abs(a - b) <= 1e-9, and add small numbers first when a long sum must keep them.

Watch Out For

Testing float equality with ==

Loss comparisons, convergence checks, and unit tests written as == on floats fail on dust like the $5.55 \times 10^{-17}$ above. They pass on one machine and fail in CI under a different reduction order, because the dust depends on operation order.

Symptom: exact-equality asserts flip without any logic change. Fix: use tolerances everywhere, relative for scale-free checks and absolute near zero, and never exact equality past arithmetic.

Shrinking finite-difference steps until the check breaks

Gradient checks subtract nearly equal function values, which is cancellation by design. Below $h \approx 10^{-5}$ in doubles, rounding noise dominates the difference and the check fails on correct gradients.

Symptom: a verified gradient fails its numeric check, and smaller $h$ makes agreement worse. Fix: park near the bottom of the U-shaped error curve at $h \approx 10^{-5}$ , compare with relative tolerance, and never gradient-check in single precision.

The Quick Version

Every operation rounds; relative error stays near epsilon while absolute error scales with size.
$(0.1 + 0.2) - 0.3$ leaves $5.55 \times 10^{-17}$ : rounding residue, not a bug.
$1 + 10^{-16}$ absorbs to 1 in doubles, so small change dies beside big totals.
Add smallest first or compensate to keep small contributions alive in long sums.
Never compare floats with equality; gradient-check near $h = 10^{-5}$ , never far below.