Core ML

Gradient Clipping

Cap the gradient's size before the optimizer step, either by shrinking each entry to a fixed range or by rescaling the whole vector, so one bad batch can't throw the weights somewhere training never recovers from.

A gradient vector with norm 41 gets rescaled down to norm 5 while keeping its direction exactly, so the optimizer still moves the right way, just a shorter distance

Why Does This Exist?

A character-level language model trains smoothly for 40,000 steps, loss falling steadily, then step 40,001 hits a rare, unusually long run of nested quotation marks the model has barely seen. The loss on that one batch spikes hard, the gradient computed from it is enormous, and the next parameter update throws every weight somewhere it's never been — the loss reported for step 40,002 is nan, and stays nan for the rest of training. Nothing about the architecture or optimizer was wrong for 40,000 steps straight. One batch, one abnormally large gradient, was enough to end the run.

That's exploding gradients: a normal, correctly-computed gradient that happens to be far larger than the ones training has been taking steps with, and a large step from a bad starting point can land the parameters somewhere the loss function returns inf or worse. The fix doesn't require finding and removing the unusual batch, which is often impractical — you want the model to eventually handle nested quotes too. It requires putting a ceiling on how large any single gradient is allowed to be before it reaches the optimizer, so an unusual batch produces an unusually large but still bounded step, rather than an unbounded one.

Think of It Like This

A steering wheel with a mechanical limit on how far it turns

A go-kart's steering wheel can, in principle, be yanked hard enough to send the front wheels to a full 90 degrees — but most karts build in a mechanical stop limiting how far the wheel physically turns, regardless of how hard the driver pulls. A sudden jolt on a bumpy patch of track still turns the wheel toward the correct side; it just can't turn it past the stop, so the kart corrects by a bounded amount instead of spinning out entirely.

The direction of that correction is exactly right either way — steer left when the track curves left. What the stop prevents is the magnitude of a single sudden yank translating one-for-one into the wheels' angle. Gradient clipping is that mechanical stop, applied to a parameter update instead of a steering wheel: the direction the gradient says to move in is kept, but how far any single step is allowed to go gets capped.

How It Actually Works

Two ways to impose the cap

Clip by value caps every individual entry of the gradient vector independently: any entry above some threshold $c$ gets set to $c$ , any entry below $-c$ gets set to $-c$ , everything in between is untouched. It's cheap and simple, but it changes the gradient's direction — shrinking one huge entry while leaving a moderate entry untouched rotates the vector, not just its length.

Clip by norm — the version almost every modern training recipe means when it says "gradient clipping" without qualification — treats the whole gradient vector $g$ as one object:

g' = \begin{cases} g & \text{if } \lVert g \rVert \le c \\ g \cdot \dfrac{c}{\lVert g \rVert} & \text{if } \lVert g \rVert > c \end{cases}

$\lVert g \rVert$ is the gradient's Euclidean norm — its overall length — and $c$ is the chosen maximum. When the norm exceeds $c$ , every entry gets multiplied by the same scalar $c / \lVert g \rVert$ , which rescales the vector's length down to exactly $c$ while leaving its direction completely unchanged. Gradient descent still moves in the direction the gradient actually pointed — just a shorter distance than the raw, unclipped gradient would have taken.

Where it's load-bearing rather than optional

Vanilla RNNs are the textbook case: the same weight matrix multiplies in at every time step, so a long sequence's backward pass through time is exactly the kind of repeated multiplication that can compound a gradient upward across many steps, the mirror image of how the same repeated multiplication compounds a gradient downward into vanishing gradients. Clip-by-norm at a threshold like 1.0 or 5.0, applied to the entire flattened gradient across every parameter at once, is close to a default setting in recurrent training recipes rather than an occasional safety net.

What it doesn't fix

A model that needs clipping on nearly every single step — not just the rare outlier batch — still has an underlying instability that clipping is masking, not curing. That's usually the learning rate set too high for the loss surface's curvature, or an initialization that leaves gradients running hot from the very first step. Clipping bounds the damage from a bad step; it does nothing to prevent the step from being computed badly in the first place, which is a job for weight initialization and the learning rate schedule.

Show Me the Code

The same gradient vector, clipped by value and by norm, checking exactly what each does to direction.

1import numpy as np2
3
4def clip_by_norm(g: np.ndarray, max_norm: float) -> np.ndarray:5    norm = np.linalg.norm(g)6    return g if norm <= max_norm else g * (max_norm / norm)7
8
9g = np.array([0.5, -8.0, 3.0, 40.0])  # one entry is way out of line with the rest10by_value = np.clip(g, -5.0, 5.0)11by_norm = clip_by_norm(g, 5.0)12
13print(f"original norm: {np.linalg.norm(g):.2f}")           # -> 40.9014print(f"clip by value: {np.round(by_value, 2)}")           # -> [ 0.5 -5.   3.   5. ]15print(f"clip by norm:  {np.round(by_norm, 2)}")            # -> [ 0.06 -0.98  0.37  4.89]16orig_dir, clipped_dir = g / np.linalg.norm(g), by_norm / np.linalg.norm(by_norm)17print(f"direction preserved by norm: {np.allclose(orig_dir, clipped_dir)}")  # -> True

Clip by value leaves the small entries untouched and only trims the two large ones, which rotates the vector. Clip by norm shrinks every entry by the same factor, so the direction — what the optimizer actually steps along — comes out identical to the original, just shorter.

Watch Out For

Clipping every single step and calling the problem solved

Loss stops producing nan, so clipping looks like it fixed things — but if the gradient norm exceeds the threshold on nearly every step rather than occasionally, the model is running at a lower effective learning rate than the configured one implies, and training that looks stable can still be slower or worse than it should be. Log the fraction of steps that actually get clipped; a healthy run clips rarely, on genuine outliers, not as a matter of routine.

Clipping each parameter tensor separately instead of the whole gradient at once

Clip-by-norm is only equivalent to a single global rescale when the norm is computed across every parameter in the model together. Clipping each layer's gradient to the same threshold independently changes the relative size of updates between layers — a layer whose gradient was already small gets clipped identically to one whose gradient was enormous, which distorts training in a way a single global clip doesn't. Most framework implementations default to the global version; verify rather than assume when writing one by hand.

The Quick Version

Gradient clipping caps a gradient's size before the optimizer step, protecting against a single unusually large gradient derailing training.
Clip by value caps each entry independently and changes the gradient's direction; clip by norm rescales the whole vector and preserves it.
Clip by global norm, computed across every parameter at once, is the version most training recipes default to.
Recurrent networks need it structurally, because backpropagation through time can compound a gradient upward across many steps.
Clipping on nearly every step is a symptom, not a fix — the underlying cause is usually the learning rate or the initialization.