Backpropagation

When a neural network produces an incorrect prediction, the objective function outputs a single scalar loss value, such as $L = 4.0$. That single number is the culmination of thousands or millions of interconnected weights across multiple layers. The central challenge of deep learning is credit assignment: determining precisely how much each individual weight contributed to the final error without perturbing parameters one by one.

Multiply Local Slopes

Backpropagation resolves credit assignment through the calculus chain rule. For any composite function $L(y(w))$, the total rate of change decomposes into a product of local slopes: $\frac{\partial L}{\partial w} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial w}$. When the loss gradient with respect to output is $\frac{\partial L}{\partial y} = -4$ and the activation's local slope with respect to weight is $\frac{\partial y}{\partial w} = 2$, multiplying these local derivatives yields an exact total gradient of $-8$.

Walk Forward First

Before any gradient can flow backward, the forward pass must evaluate the network graph from input to output. Given input $x = 2.0$ and initial weight $w = 0.5$, linear multiplication yields intermediate activation $y = 1.0$. Measuring against target $t = 3.0$ under squared error loss $L = (y - t)^2 = (1.0 - 3.0)^2$ produces $L = 4.0$. Crucially, the forward pass caches these intermediate states ($x = 2.0, y = 1.0, y - t = -2.0$), storing the receipts required to compute local derivatives later.

Sweep Back Once

In the backward pass, signals flow in reverse. The upstream loss derivative $\frac{\partial L}{\partial y} = 2(y - t) = -4.0$ arrives at the node and multiplies the cached input $x = 2.0$, producing weight gradient $\frac{\partial L}{\partial w} = -8.0$. Applying gradient descent with learning rate $\eta = 0.1$ adjusts the parameter: $w_{new} = w - \eta \frac{\partial L}{\partial w} = 0.5 - 0.1 \times (-8.0) = 1.3$. Re-evaluating forward ($y = 1.3 \times 2.0 = 2.6$) cuts the loss from $4.0$ down to $(2.6 - 3.0)^2 = 0.16$ in a single round trip.

Two Beats a Million

Numerical differentiation using finite differences requires wiggling each parameter independently, demanding $N + 1$ forward evaluations for $N$ weights—over one million passes for a standard model. Backpropagation (reverse-mode automatic differentiation) shares intermediate computations across the entire graph, computing exact analytic derivatives for every parameter simultaneously in exactly two passes: one forward sweep to cache activations, and one backward sweep to distribute gradients.

Leash Wild Steps

Deep networks with recurring products can experience exploding gradients, causing sudden numerical overflow and NaN loss spikes. Gradient clipping enforces a maximum norm threshold $\tau = 1.0$. A calm gradient vector ($g = 0.4$) passes untouched. When a gradient surges to $g = 8.0$, clipping scales its magnitude down to $1.0$ while preserving its directional orientation, ensuring stable parameter updates.

Where It Breaks

Chaining derivatives across deep networks exposes a severe limitation when using saturating activation functions like Sigmoid, where the derivative peaks at $\sigma'(z) \le 0.25$. Chaining ten consecutive layers multiplies these fractional terms: $(0.25)^{10} \approx 9.5 \times 10^{-7}$. The error signal reaching early layers decays to a mere millionth of its origin, causing early weights to freeze and stalling deep network training.

The Quick Version

• Scalar loss must be apportioned among thousands of network weights.
• The chain rule computes total gradients by multiplying local slopes.
• The forward pass evaluates outputs and caches intermediate values.
• The backward pass propagates errors and applies gradient descent.
• Reverse-mode autodiff computes all gradients in exactly two passes.
• Gradient clipping bounds exploding updates while preserving direction.
• Chained fractional derivatives decay exponentially, freezing early layers.