Gradient Descent

What you are looking at

A scatter of observations and a straight line trying to describe them. The line has two knobs — its slope and where it crosses zero — and for any setting of those two knobs there is a single number saying how badly it fits.

That number is the whole game. Everything else on this page is machinery for making it smaller.

Why a line, when you already know the answer

Straight-line regression has an exact solution. You can compute the best possible slope and intercept in closed form, in one pass, with no iteration at all — and the lab shows you that value, labelled "best possible loss", so you always know what you are aiming at.

So gradient descent is overkill here, and that is exactly why it is worth watching here. Almost nothing else has a closed form. Neural networks do not. Once you move past a straight line, iterating toward an answer is the only option you have — and this is the one place where you can watch the iteration happen and check it against the answer.

Fog on a hillside

You are standing on a hillside in thick fog and want to reach the valley floor. You cannot see where it is. What you can do is feel which way the ground slopes under your feet, and step that way.

Then feel again, and step again.

That is gradient descent. The slope under your feet is the gradient. How big a step you take is the learning rate. And the reason fog matters to the analogy is that the optimiser genuinely cannot see the minimum — it only ever knows the slope at the exact point it is standing.

The loss, and why it is squared

For each observation, the residual is the gap between what the line predicted and what actually happened. Square each one, average them, and you have the mean squared error:

$$L(w, b) = \frac{1}{n} \sum_{i=1}^{n} \left( (w x_i + b) - y_i \right)^2$$

Squaring does two things. It makes the sign irrelevant — being 3 too high is as bad as being 3 too low. And it makes large errors dominate: a single point 4 away contributes 16, while four points 1 away contribute 4 between them.

That second effect is worth feeling directly. Drag one point far from the line and watch the loss lurch while the mean absolute error barely moves. The squared version cares enormously about outliers, and that is a choice with consequences, not a neutral default.

What the gradient actually is

Differentiate the loss with respect to each parameter and you get the slope of the loss surface in that direction:

$$\frac{\partial L}{\partial w} = \frac{2}{n} \sum_{i=1}^{n} \left( \hat{y}_i - y_i \right) x_i \qquad \frac{\partial L}{\partial b} = \frac{2}{n} \sum_{i=1}^{n} \left( \hat{y}_i - y_i \right)$$

These are not estimates. They are the exact derivatives, and the arrow you see in the second panel is drawn from them. Its length is the step the optimiser is about to take, which is why raising the learning rate makes the arrow longer before anything moves.

Then the update, which is the entire algorithm:

$$w \leftarrow w - \eta \frac{\partial L}{\partial w}$$

Move against the gradient, because the gradient points uphill and you want to go down. Scale by $\eta$, the learning rate, because the raw gradient is a direction rather than a distance.

Use Step, not Play

Press Step and watch the order of events: the arrow appears, then the parameters move, then the loss updates. Press it again and the same three things happen from the new position.

Play is the same thing faster, and it is worse for learning, because at speed the three events blur into one motion. The pause between them is where the causality lives.

Break it

There is a real threshold on the learning-rate slider, computed from the data currently on screen rather than picked as a round number. Below it, gradient descent on this loss surface always converges. Above it, it always diverges.

$$\eta < \frac{2}{\lambda_{\max}(H)}$$

Push past the marker and watch what happens: the step lands further from the minimum than it started, so the next gradient is larger, so the next step is bigger still. The loss climbs, then climbs faster, then leaves the plot.

A higher learning rate is not a faster learning rate

The intuition that a bigger step gets you there sooner holds right up until it catastrophically does not. Below the threshold, raising the rate genuinely does converge faster. Above it, the same change makes the model worse every single step, and no amount of extra iterations recovers.

This is why learning-rate tuning is not a matter of turning it up until training is quick. The useful range has a hard ceiling, and the failure past that ceiling is immediate rather than gradual.

Then drag a point far away and try again. You will find the threshold has moved: it depends on the data, not just on the optimiser. That is why a learning rate that worked on one dataset can diverge on the next.

What to take away

The loss is a number you can compute for any parameter setting. The gradient is the exact slope of that number's surface. Descent is the loop of measuring the slope and stepping against it. And the learning rate is the one knob that can turn the whole thing from convergent into divergent without any warning in between.

Everything a modern network does when it trains is this loop, with millions of parameters instead of two and no closed-form answer to check against.