Math

Floating Point Formats

Every number format splits its bits between how large a value it can hold and how finely it can distinguish nearby ones — and training cares far more about the first than the second.

The bits a format spends on its exponent buy range and the bits spent on its mantissa buy precision, and bfloat16 keeps all of float32's range by giving up precision instead

Why Does This Exist?

Two 16-bit formats. Same memory, same bandwidth, roughly the same hardware throughput — and one of them trains a large model reliably while the other needs an extra mechanism to avoid producing NaN.

The difference is how they split their bits. float16 spends 5 on the exponent and 10 on the mantissa; bfloat16 spends 8 and 7. That single choice decides whether your gradients survive, and it explains why loss scaling exists, why bfloat16 became the default on hardware that supports it, and why the answer to "just use fewer bits" is more involved than it sounds.

This page is the one tier-S page in the band: the formats themselves are stable and standardised, but which hardware accelerates which is not. So it names no accelerator generation and carries an updatedAt — check the current hardware documentation for support, and use this for the arithmetic that does not change.

Think of It Like This

Scientific notation with a fixed number of boxes

You must write every number as $d.ddd \times 10^{ee}$ , and you get a fixed number of boxes: some for the digits, some for the exponent.

Spend more boxes on the exponent and you can write enormous and minuscule numbers — $10^{38}$ and $10^{-38}$ — but you have fewer digits, so nearby values collapse together.

Spend more on the digits and you distinguish 1.00001 from 1.00002 beautifully, but you cannot write $10^{38}$ at all. Try and you get "too big", which in floating point is infinity.

Now the ML question. Gradients in a deep network span an enormous range — a product of many chain-rule factors can land anywhere from $10^{-10}$ to $10^{5}$ — and you mostly need to know their rough size and direction, not their sixth digit. You are asking about magnitude, not fine detail.

So spend the boxes on the exponent. That is precisely what bfloat16 does, and why it works where float16 struggles.

How It Actually Works

A floating-point number is a sign bit, an exponent field, and a mantissa (significand) field:

\text{value} = (-1)^{s} \times 1.m \times 2^{e - \text{bias}}

The exponent width sets the range — the largest and smallest representable magnitudes. The mantissa width sets the precision — how many significant bits, and therefore the machine epsilon.

The formats that matter, and the two numbers that matter about each:

float32 — 8 exponent, 23 mantissa. Max $\approx 3.4 \times 10^{38}$ , epsilon $\approx 1.2 \times 10^{-7}$ (about 7 decimal digits). The baseline.
float16 — 5 exponent, 10 mantissa. Max 65,504, epsilon $\approx 9.8 \times 10^{-4}$ (about 3 digits). Smallest normal value $\approx 6.1 \times 10^{-5}$ .
bfloat16 — 8 exponent, 7 mantissa. Max $\approx 3.4 \times 10^{38}$ — identical range to float32 — epsilon $\approx 7.8 \times 10^{-3}$ (about 2 digits).
tf32 — 8 exponent, 10 mantissa. float32 range with reduced precision; used internally by matmul units while the tensors stay float32.
fp8 — two variants. E4M3 (4 exponent, 3 mantissa, max 448) favours precision and suits forward activations; E5M2 (5 exponent, 2 mantissa, max 57,344) favours range and suits gradients.

Note the pattern: float16 and bfloat16 are the same size and differ only in the split. That is the whole design decision.

Why range beats precision for gradients

Two failures matter, and they are not symmetric.

Overflow is loud. A value past the maximum becomes inf, then nan, and the model is destroyed. float16's ceiling of 65,504 is genuinely low — a chain-rule product over 30 layers with an average factor of 1.5 reaches about $1.9 \times 10^{5}$ , straight past it.

Underflow is quiet and worse. A gradient of $10^{-8}$ is below float16's smallest normal value, so it becomes zero (or a lossy subnormal). No error, no warning — those parameters simply stop learning, and you go looking for architectural reasons for a plateau that is arithmetic.

bfloat16 has float32's exponent, so neither happens. It pays with precision — about 2 decimal digits — and that turns out to be acceptable because a gradient's job is to indicate a direction.

What mixed precision actually does

"Mixed precision" is not "use 16 bits". It is a set of specific defences, and knowing them tells you which one failed when a run diverges:

A float32 master copy of the weights. Updates are often far smaller than the weights they modify, and adding $10^{-7}$ to 1.0 in bfloat16 — whose epsilon is $7.8 \times 10^{-3}$ — changes nothing at all. Forward and backward run narrow; the optimiser step happens wide.
Loss scaling. Multiply the loss by a large constant before the backward pass so small gradients land inside the representable range, then divide it out before the update. float16 needs this; bfloat16 usually does not, which is the single clearest practical consequence of the range difference. Dynamic loss scaling backs off when it detects overflow — so a run that looks healthy may be silently skipping batches.
Wide accumulation. Kernels accumulate in float32 even when inputs are narrower, because summing thousands of low-precision values loses bits at every step.
Selectively wider layers. Normalisation statistics, softmax, and the final projection are often kept wider, because that is where range runs out first.

Worked example

Take a gradient of $g = 10^{-8}$ and ask whether each format can hold it.

float16. Smallest normal value is $2^{-14} \approx 6.10 \times 10^{-5}$ . Since $10^{-8}$ is far below that, it lands in the subnormal range (down to about $6 \times 10^{-8}$ ) or rounds to exactly zero. That parameter receives no update.

bfloat16. Same exponent width as float32, so the smallest normal value is about $1.18 \times 10^{-38}$ . $10^{-8}$ is comfortably representable.

Now the other end, $g = 10^{5}$ :

float16. Maximum is 65,504, and $10^5 = 100{,}000 > 65{,}504$ , so this overflows to inf. One nan later, training is over.

bfloat16. Maximum $3.4 \times 10^{38}$ . Fine.

So across a range gradients genuinely occupy, float16 fails at both ends and bfloat16 at neither. Loss scaling exists to shift the whole distribution into float16's narrow window — multiply by $2^{15} = 32{,}768$ and $10^{-8}$ becomes $3.3 \times 10^{-4}$ , now representable.

The precision cost, for contrast. bfloat16 epsilon is $7.8 \times 10^{-3}$ , so adding a weight update of $10^{-4}$ to a weight of 1.0 gives... 1.0. Nothing changes. That is why the float32 master copy is not optional — without it, small updates are silently discarded and training stalls at a worse loss for reasons no learning-rate sweep will fix.

Show Me the Code

1import numpy as np2
3for dt in (np.float32, np.float16):4    info = np.finfo(dt)5    print(dt.__name__, f"{info.max:.4g}", f"{info.eps:.3g}", f"{info.tiny:.3g}")6# -> float32 3.403e+38 1.19e-07 1.18e-387# -> float16 6.55e+04  0.000977 6.104e-058
9g_small, g_large = 1e-8, 1e510print(np.float16(g_small), np.float16(g_large))     # -> 1e-08 (subnormal) inf11print(np.float32(g_small), np.float32(g_large))     # -> 1e-08 100000.012
13# Loss scaling lifts a tiny gradient into float16's window.14print(np.float16(g_small * 2**15))                   # -> 0.000327815
16# Why the float32 master copy is mandatory: the update vanishes in low precision.17w = np.float32(1.0)18print(np.float32(w + 1e-4))                          # -> 1.0001   float32 keeps it19print(np.float16(np.float16(1.0) + np.float16(1e-4)))  # -> 1.0     float16 discards it

The float16 maximum of 65,504 and the overflow of $10^{5}$ to inf match the hand calculation, and the last two lines are the master-weight argument as an executable check. NumPy has no native bfloat16, but its exponent field is identical to float32, so float32's range row describes it.

Watch Out For

Switching to float16 without loss scaling and blaming the model

model.half() is one call, appears to work, and then training diverges or plateaus for reasons that look architectural.

Both failure directions are silent in their own way. Gradients that underflow to zero mean some parameters stop updating — the loss still falls, driven by the rest, so you get a model that trains to a worse optimum with no error. Gradients that overflow produce inf, then nan, and the run dies at an arbitrary step with no obvious trigger.

The fix is to use the framework's mixed-precision path rather than casting by hand. torch.amp.autocast plus GradScaler handles the master weights, the loss scaling, and the per-operation precision choices. Casting the model yourself skips all three.

Two diagnostics worth having. Log the loss-scale value if your framework exposes it: a scale that keeps backing off is telling you overflow is happening and batches are being skipped, which is information you otherwise never see. And watch the gradient norm, since it drifts toward the ceiling before anything breaks, giving you warning that a loss curve does not.

If your hardware supports bfloat16, prefer it. Same memory saving, no loss scaling, far fewer ways to fail — and the lost precision genuinely does not matter for gradients.

Comparing narrow-format results without controlling for seed variance

Reduced precision often shows up as increased variance across seeds rather than a worse mean, and a single-run comparison cannot see that.

The mechanism is straightforward: lower precision adds rounding noise to every operation, which perturbs the optimisation trajectory. On a non-convex surface that means different runs land in different places. Average quality may be unchanged while the spread widens noticeably — and the run-to-run gap can exceed the difference you were trying to measure.

So "we switched to float16 and accuracy dropped 0.3 points" is not a measurement from one run of each. It might be precision, and it might be the seed.

Compare distributions: at least three seeds per configuration, reported as mean and standard deviation. If the change is smaller than the seed spread, you have not measured it — the same argument as anywhere else in non-convex optimisation.

Two more traps in the same area. Numerical reproducibility is weaker in narrow formats, because reduction order affects the result more, so bitwise-identical reruns are harder even with fixed seeds. And evaluation should usually run in the wider format — casting the model for inference introduces a precision difference between your measurement and your training, which is a confound you do not need.

The Quick Version

A float is sign, exponent and mantissa. Exponent width sets range; mantissa width sets precision.
float32: max $3.4\times10^{38}$ , epsilon $1.2\times10^{-7}$ , ~7 digits.
float16: max 65,504, epsilon $9.8\times10^{-4}$ , smallest normal $6.1\times10^{-5}$ .
bfloat16: same range as float32, epsilon $7.8\times10^{-3}$ , ~2 digits.
float16 and bfloat16 are both 16 bits and differ only in the 5/10 versus 8/7 split.
Overflow is loud (inf → nan); underflow is quiet (parameters silently stop learning). float16 risks both.
Gradients need range far more than precision, so bfloat16 wins for training.
float16 needs loss scaling; bfloat16 usually does not. That is the clearest practical consequence.
Mixed precision means a float32 master copy, loss scaling, wide accumulation, and selectively wider layers — not "cast the model".
Without master weights, a $10^{-4}$ update to a weight of 1.0 vanishes in bfloat16.
fp8 E4M3 favours precision for activations; E5M2 favours range for gradients.
Narrow formats often raise seed variance rather than lowering the mean. Compare across at least three seeds.