Neural Networks

Backpropagation

How the chain rule assigns "blame" for errors to individual weights across multiple layers.

Input value
Forward weight
Negative weight
Backward gradient

Input x₁

The first input. It is also the last factor in the chain for w₁₁.

Weight w₁₁

The first-layer weight whose gradient this lab traces back to the loss.

Prediction (y)2.1900

Data flows left to right. x₁ is weighted by w₁₁ and squashed to h₁, then weighted by w₃₁ to give the prediction y. Move x₁ or w₁₁ and watch every downstream number follow.

Check your understanding

Watch the chain rule assign blame to individual weights — as a product, factor by factor, not as an answer to take on trust.

Backpropagation is the algorithm that lets a neural network learn. It answers one question: how much is each weight to blame for the error?

When the network makes a prediction, data flows forward from the inputs through the hidden layer to the output. That is the forward pass. The prediction is then compared to the target, and half the squared difference is the loss.

Blame is a product, not a number

To reduce the loss, we need ∂L/∂w for every weight w. The chain rule gives it as a product of local derivatives along the path from that weight to the loss — and this lab shows you the product rather than only its value.

For the output weight, three factors:

1∂L/∂w₃₁ = (y − t) · f′(z₃) · h₁

For the first-layer weight, five:

1∂L/∂w₁₁ = (y − t) · f′(z₃) · w₃₁ · f′(z₁) · x₁

Each factor answers a small question. (y − t) is how wrong the prediction was. f′(z₃) is how responsive the output unit was at the value it happened to be sitting at. w₃₁ is the weight the signal travelled back along. h₁ and x₁ are the inputs the weights received — which is why a weight that received a zero input gets a zero gradient, no matter how wrong the answer was.

Every extra layer adds exactly two factors to that product: one weight, and one activation derivative. Watch the chain under the scene grow from three terms to five as you step through the stages.

The Vanishing Gradient Problem

Of those five factors, only two are structurally capped. Weights can be any size and grow during training; the input and the error are given by the data. But an activation derivative has a ceiling, and one of them enters the product per layer.

The sigmoid's derivative never exceeds 0.25 anywhere, so two layers of it cannot beat 0.25 × 0.25 = 0.0625 however well tuned the network is — and once a unit saturates it is far worse than that. The Vanishing Factor readout is exactly this product, f′(z₃) · f′(z₁):

ReLU, the default: exactly 1. Above zero ReLU is the identity, so depth costs nothing.
Sigmoid at the same settings: 0.0305, and 0.0083 once you push Input x₁ past 3 and the hidden unit crosses its saturation onset at a sum of 2.887.
At the sliders' extremes it reaches 8.7 × 10⁻⁹, where the first-layer gradient reads zero to four decimal places while the prediction is still visibly wrong.

Tanh saturates sooner than sigmoid despite its derivative peaking at 1 rather than 0.25, which is why swapping one squashing function for another only ever bought a little time. ReLU's derivative being exactly 1 is what made deep networks trainable — and its being exactly 0 below zero is the price, a unit that can die outright rather than merely learn slowly.

Gradients are not updates

This lab computes gradients and never changes a weight. There is no learning rate here and no step: nothing applies w ← w − η·∂L/∂w.

That separation is deliberate and it is how real frameworks are built. Backpropagation answers "how much is each weight to blame"; an optimiser decides what to do about it. The same backward pass feeds plain SGD, momentum and Adam without modification — see Gradient Descent for the half this lab leaves out.

Reference

Chain Rule: dL/dw = the product of the local derivatives along the path from w to L
Forward pass: z₁ = x₁w₁₁ + x₂w₁₂ + b₁, h₁ = f(z₁), z₃ = h₁w₃₁ + h₂w₃₂ + b₃, y = f(z₃)
Loss: L = ½(y − t)², so ∂L/∂y = y − t
Output delta: δ₃ = (y − t)·f′(z₃) — the blame carried by the output unit
Weight gradient: ∂L/∂w = δ × the input that weight received. Zero input, zero gradient
Hidden delta: δ₁ = δ₃·w₃₁·f′(z₁) — blame routed backward through the weight it came along
Vanishing factor: f′(z₃)·f′(z₁): at most 0.0625 for sigmoid, 1 for tanh, exactly 1 for ReLU above zero
No update here: this lab computes gradients only — applying them is gradient descent

Break it on purpose

Switch the activation to Sigmoid and push Input x₁ up. Two sigmoid derivatives now sit in the product for ∂L/∂w₁₁, and neither can exceed 0.25 — so the vanishing factor starts at 0.0305, falls to 0.0083 once the hidden unit saturates past x₁ = 3, and reaches 8.7e-9 at the extremes, where the first-layer gradient is zero to four decimal places. Choose ReLU and the same two factors are exactly 1.