Core ML

Backpropagation

The algorithm that tells each weight in a network how to nudge itself so the next prediction comes out a little less wrong.

A forward pass produces a loss, then gradients flow backward through each layer to update the weights

Why Does This Exist?

A neural network can have thousands or millions of weights. Training means finding values for all of them that make the predictions good. You could imagine nudging one weight, checking whether the error dropped, then moving to the next, but with millions of weights that would take effectively forever.

Backpropagation is the shortcut. In a single backward sweep it computes exactly how much each weight contributed to the error. Armed with that, the network can adjust every weight at once, each in the direction that lowers the error. It's the reason training deep networks is practical at all.

Think of It Like This

Tracing blame back through a chain of coworkers

A project ships late. The manager asks the last person in the chain, "how much of this delay was yours?" That person points upstream: "some of it, but you handed me things late too." The blame flows backward, each person taking a share proportional to how much they actually affected the outcome. Backprop assigns blame for the error to each weight the exact same way, from the output back toward the input.

How It Actually Works

Training runs two passes over each example.

Forward pass. Push the input through the network to get a prediction, then compare it to the correct answer with a loss function, a single number measuring how wrong the network was.
Backward pass. Starting from that loss, apply the chain rule from calculus to work out the gradient for every weight: how much the loss would change if you tweaked that weight a hair. This is computed layer by layer from the output backward, and each layer reuses the results of the one after it, which is what makes it fast.

Then comes a gradient descent step: move each weight a small amount, set by the learning rate, in the opposite direction of its gradient. Opposite, because the gradient points uphill and you want to go down. Repeat across many examples and the error keeps shrinking.

One training step

Step 1 of 4

Forward pass

Run the inputs through the network to produce a prediction and measure the loss.

Show all steps

Forward pass: Run the inputs through the network to produce a prediction and measure the loss.
Measure the slope: Backpropagation applies the chain rule to compute the gradient of the loss for every weight.
Step downhill: Nudge each weight a little in the opposite direction of its gradient, scaled by the learning rate.
Repeat: Loop over many batches until the loss stops improving.

Show Me the Code

1import numpy as np2
3
4def sgd_step(w: np.ndarray, x: np.ndarray, y: float, lr: float = 0.1) -> np.ndarray:5    y_hat = float(x @ w)  # forward: the current prediction6    grad = 2.0 * (y_hat - y) * x  # d(squared error)/dw, via the chain rule7    return w - lr * grad  # step downhill, opposite the gradient8
9
10w = np.array([0.0, 0.0])  # start the weights at zero11x, y = np.array([1.0, 2.0]), 5.0  # a single training example12for _ in range(50):  # each step shrinks the error a little13    w = sgd_step(w, x, y)14print(w, float(x @ w))  # weights settle so the prediction lands near 5.0

This is one weight vector and one example, but the idea scales directly: every weight in a deep network gets its own gradient and its own downhill nudge.

Watch Out For

Vanishing and exploding gradients

In deep networks, gradients are products of many numbers. Multiply enough small ones and the signal vanishes toward zero; multiply enough large ones and it blows up. Both stall learning. Careful initialization, normalization, and activations like ReLU are the common defenses.

The learning rate is finicky

Too large and the steps overshoot and diverge; too small and training crawls. It's usually the first knob worth tuning when a model won't learn.

The Quick Version

Backprop computes how much each weight contributed to the error, in one efficient backward pass.
Under the hood it's just the chain rule from calculus, applied layer by layer.
Gradient descent then nudges each weight downhill to shrink the error.
Repeat over many examples and the network learns.
Watch for vanishing or exploding gradients and a badly chosen learning rate.