Core ML

Gradient Descent

How a model learns by feeling which way is downhill on an error landscape and taking small steps in that direction until it can't improve.

Starting high on a bowl-shaped loss curve, each step moves downhill opposite the slope until it reaches the minimum

Why Does This Exist?

Training a model means finding the weights that make its predictions good. "Good" is measured by a loss function: a single number that's large when the model is wrong and small when it's right. So training boils down to one goal: find the weights that make the loss as small as possible.

The trouble is there's no formula to jump straight to the answer for anything but the simplest models. A modern network has millions of weights, and the loss depends on all of them in a tangled, nonlinear way. Gradient descent is the workhorse that solves this anyway. Rather than solving for the minimum directly, it starts somewhere, figures out which direction is downhill, and walks that way one small step at a time. Almost every neural network you've heard of was trained this way.

Think of It Like This

Descending a hill in thick fog

You're on a foggy hillside and want to reach the valley, but you can only see the ground right at your feet. What do you do? Feel which way the slope tilts downward and take a step that way. Check again, step again. You never see the whole landscape, yet by always moving downhill you steadily descend. Gradient descent is exactly this: the slope under your feet is the gradient, and the valley is the lowest loss.

How It Actually Works

The gradient is the multi-dimensional slope of the loss: for each weight, it says how much the loss would rise if you nudged that weight up. It points in the direction of steepest increase, so to reduce the loss you step in the opposite direction.

Measure the loss for the current weights on some training data.
Compute the gradient with respect to every weight. In a neural network this is exactly what backpropagation delivers.
Step downhill. Subtract a small fraction of the gradient from each weight. That fraction is the learning rate.
Repeat until the loss stops meaningfully dropping.

Computing the gradient over the entire dataset every step is expensive, so in practice we use stochastic gradient descent (SGD): estimate the gradient from a small random batch each step. It's noisier but far faster, and the noise can even help escape shallow bad spots. One full pass over the data is an epoch.

One gradient descent step

Step 1 of 4

Evaluate loss

Run a batch through the model and measure how wrong it is.

Show all steps

Evaluate loss: Run a batch through the model and measure how wrong it is.
Find the slope: Compute the gradient — the direction that would increase the loss fastest.
Step opposite: Move each weight a little in the downhill direction, scaled by the learning rate.
Repeat: Loop over batches and epochs until the loss flattens out.

Show Me the Code

1from typing import Callable2
3
4def gradient_descent(grad: Callable[[float], float], x: float, lr: float = 0.1) -> float:5    for _ in range(100):6        x = x - lr * grad(x)  # step opposite the slope, scaled by lr7    return x8
9
10# Minimize f(x) = (x - 3)^2, whose slope is f'(x) = 2*(x - 3).11minimum = gradient_descent(lambda x: 2 * (x - 3), x=0.0)12print(round(minimum, 3))  # -> 3.0, the bottom of the bowl

The same loop scales to millions of weights: replace the single number with a weight vector and grad with backpropagation.

Watch Out For

The learning rate is the make-or-break knob

Too large and each step overshoots the valley, bouncing around or diverging to infinity. Too small and training crawls, taking forever to converge. A bad learning rate is the most common reason a model "won't train", so it's the first thing to tune.

Local minima and flat plateaus

A bumpy loss landscape can trap the descent in a dip that isn't the true lowest point, or on a flat plateau where the gradient is nearly zero and progress stalls. Momentum-based optimizers and good initialization help the ball roll past these traps.

The Quick Version

Gradient descent minimizes the loss by repeatedly stepping downhill.
The gradient points uphill, so each step moves in the opposite direction.
The learning rate controls step size, and it's the most sensitive hyperparameter.
Stochastic gradient descent estimates the gradient from small batches for speed.
Watch for overshooting, local minima, and flat plateaus.