FP8 Training Is Not Free

Narrow floating-point formats get announced the same way every time: a throughput multiple, a memory saving, and a chart. The numbers are usually honest. The framing is not, because it presents a hardware upgrade where what you are actually being offered is a trade of numerical range for speed — and range is the thing training runs out of.

Here is the part worth internalising before you migrate anything. float32 reaches about $3.4 \times 10^{38}$. float16 reaches 65,504. That is not a small reduction, it is a collapse of twenty-eight orders of magnitude, and it puts real quantities inside the danger zone rather than comfortably outside it.

The failure is structural, not unlucky

Training multiplies things. That is what the chain rule does — a gradient reaching an early layer is a product of every local derivative along the path, and products of many numbers are unforgiving in a way sums are not.

Take thirty layers with an average local factor of 1.5. The product is about $1.9 \times 10^{5}$. In float32 that is a large number and nothing more. In float16 it is inf, and inf becomes NaN on the next operation, and one NaN converts every parameter it touches. The model was fine at step 400 and unrecoverable at step 401.

The mirror case is quieter and worse. Average factor 0.5 over thirty layers gives roughly $10^{-9}$, which underflows to exactly zero in float16. Nothing crashes. Your early layers simply stop learning, and you go looking for architectural explanations for a plateau that is arithmetic.

Attention has its own version. A dot product over 4,096 dimensions sums 4,096 products into one accumulator, and scores can exceed 65,504 before any softmax gets the chance to normalise them. This is why cross-entropy and softmax are fused into one kernel and why attention divides by $\sqrt{d_k}$ — those are not micro-optimisations, they are the reason the layer works at all in reduced precision.

What the tooling is actually doing for you

Mixed precision is not "use fewer bits". It is a set of specific defences, and it is worth knowing them because when a run diverges you need to know which one failed:

• A float32 master copy of the weights. Updates are often far smaller than the weights they modify, and adding a tiny number to a large one in low precision rounds to no change at all. The optimiser step happens in wide precision; only the forward and backward passes are narrow.
• Loss scaling. Multiply the loss by a large constant before the backward pass so small gradients land inside the representable range, then divide it back out before the update. Dynamic loss scaling adjusts that constant when it detects an overflow — which means a run that appears healthy may be silently discarding batches.
• Wide accumulation. Kernels accumulate in float32 even when inputs are float16 or FP8, because summing thousands of narrow values loses low-order bits at every step.
• A retained wider format for the sensitive layers. Normalisation statistics, the softmax, and the final projection are frequently kept wider, because that is where range runs out first.

BF16 deserves specific mention, because it is the format that made this manageable. It has the same exponent width as float32 — so the same range — and pays for it with fewer mantissa bits. For training that is exactly the right trade: gradients need range far more than they need precision. If you are choosing today and your hardware supports it, BF16 is the boring correct answer and FP8 is the one that needs a numerics budget.

The honest recommendation

Narrow formats are genuinely worth adopting. Memory halves, and memory is what caps your batch size and your context length. But budget for the migration as numerics work rather than a config change:

Watch the gradient norm, not just the loss. A loss curve looks fine right up until it does not; the gradient norm shows the drift toward the ceiling in advance. Log the loss-scale value if your framework exposes it, because a scale that keeps backing off is telling you overflow is happening. And when you compare against a float32 baseline, compare the variance across seeds rather than a single run — the effect of reduced precision often shows up as instability rather than a worse mean, which is a distinction a single number cannot express.

Then take the speedup. Just do not accept it as free.