Visual explainer

Gradient Descent, Visually

Five pictures: stand on the loss surface, read the uphill arrow, flip it, take small steps, and see where the flat stretches fool you.

Standing somewhere on the loss surface with no map, the only question that matters is which direction is downhill

You're on a surface with one number at every point: how wrong the model is right now. There's no map, only the ground under your feet.

The gradient points the wrong way on purpose

The gradient at your feet is an arrow pointing uphill; training walks the exact opposite way

The one measurement you can take at any point is an arrow pointing steeply uphill. Training flips it and walks the other way. That flip is the entire algorithm.

Small steps, shrinking as you go

Measure the gradient, take a small step opposite it, and repeat, each hop shorter than the last as the slope flattens

Measure, step, measure again. Near the minimum the slope itself goes flat, so the steps shrink on their own — nobody has to tell it to slow down.

The step size is the only knob that matters

The step size decides everything: too small crawls, too large climbs away, and the right size lands on the minimum

Same start, three sizes. Too small never arrives. Too large overshoots every single step and the loss climbs. One size in between actually lands.

Where It Breaks

On a flat stretch the gradient is nearly zero in every direction, so the steps shrink to nothing long before the real minimum

A long flat stretch looks exactly like arrival: the gradient is nearly zero, the steps shrink to nothing, and progress stalls well short of the real minimum sitting further along.

The Quick Version

The gradient points uphill; training steps the opposite way.
Each step is small — the learning rate decides how small.
Steps naturally shrink as the surface flattens near a minimum.
Too large a step overshoots and diverges; too small crawls forever.
A flat stretch can stall progress long before the true minimum.