Optimizers
Advanced Optimizers
Modern optimizers adapt by remembering past gradients (Momentum) or scaling step sizes (RMSProp), rather than just reacting to the local slope like SGD.
Stage 1 of 4: 1. SGD in a Ravine
Step 0. SGD loss is 16.2100. No optimizer has diverged.
- SGD
0: Ravine, 1: Saddle, 2: Plateau
Basic Stochastic Gradient Descent zig-zags across the steep walls while making slow progress along the flat bottom.
Challenge
The ravine is 20 times steeper across than it is along. Plain SGD wastes almost all of its motion bouncing between the walls, so getting to the bottom quickly is less about a bigger step than about a step that stops arguing with itself.
On the ravine, get any optimizer within 0.05 of the minimum in 20 steps or fewer.
Challenge not yet solved.
Check your understanding
Advanced Optimizers: SGD, Momentum, RMSProp, and Adam
Advanced Optimizers: SGD, Momentum, RMSProp, and Adam
Stochastic Gradient Descent (SGD) is the foundation of neural network training, but navigating the loss surface can be notoriously tricky. If the surface is a steep ravine, SGD will bounce back and forth across the walls while making very slow progress along the bottom.
To fix this, modern algorithms add history. By remembering past steps, they can damp out oscillations and accelerate in consistent directions.
Momentum works like a heavy ball rolling down a hill. It builds up speed in directions where the gradient consistently points the same way, and cancels out zig-zags where the gradient keeps flipping sign.
RMSProp adapts the learning rate per parameter. It divides the gradient by a moving average of its recent magnitude. This normalises the steps: steep slopes get smaller steps (preventing overshoot), and gentle slopes get larger steps (speeding up progress).
Adam combines both. It keeps a moving average of the gradient (like Momentum) and a moving average of the squared gradient (like RMSProp). This makes it incredibly robust to different loss landscapes, which is why it is the default optimizer for most deep learning today.
The saddle surface is worth a deliberate visit, because it is the one place where "downhill" is a trap. At the centre the gradient vanishes in one direction and points away in the other, so plain SGD crawls while anything carrying momentum coasts straight through. Nothing converges there — a saddle has no minimum, only a flat spot on the way past — and watching four optimizers disagree about how quickly to leave it is the clearest picture of what "adaptive" actually buys you.
Break it on purpose
Push the learning rate toward 1.0 on the ravine. The steep walls return gradients so large that a single step overshoots to the far wall, harder, and every optimizer flies apart within a few steps — the adaptive ones last a little longer, which is exactly what dividing by recent gradient magnitude buys you.