Learning Rate Scheduling, Visually
Five pictures: How to adjust the learning rate during training to escape plateaus, avoid wild oscillations, and smoothly settle into the optimal minimum.
A single, fixed learning rate is a difficult compromise. If it is kept small, the model crawls across initial flat plateaus and takes an eternity to learn. If it is large, it bounces chaotically around the steep walls of the minimum at the end, completely unable to settle.
Step Decay
Step decay simply reduces the learning rate by a fixed multiplier at predefined training milestones. This tiered approach allows aggressive, fast progress early on, followed by smaller, fine-grained steps that precisely lock into the deepest part of the minimum.
Cosine Decay
Instead of relying on harsh discrete jumps, cosine decay smoothly and continuously anneals the learning rate down to zero. This continuous mathematical descent completely avoids sudden shocks to the network weights, resulting in a significantly more stable training trajectory.
Linear Warmup
At the very beginning of training, entirely random weights generate massive, noisy gradients. A linear warmup phase starts the learning rate near zero and gradually increases it to peak capacity, preventing these early wild updates from destroying the network.
Where It Breaks
Decaying the learning rate is fundamentally a one-way trip. If the learning rate drops too drastically and too prematurely, the optimizer entirely loses the momentum and energy needed to escape small local dips, permanently leaving it stranded far from the global minimum.
The Quick Version
- A static step size is too slow early on and too chaotic later.
- Step decay drops the rate at fixed epoch milestones like literal stairs.
- Cosine decay continuously and smoothly anneals the learning rate down to zero.
- Linear warmup protects fragile initial weights by ramping up the rate slowly.
- The honest failure mode: Dropping the rate prematurely traps the model permanently.