Momentum
An optimization technique that accelerates gradient descent by adding a fraction of the previous update vector to the current one, dampening oscillations.
Think of It Like This
Like a heavy bowling ball rolling down a hill, easily smashing through small bumps that would stop a lightweight marble.
Momentum helps the optimizer escape shallow local minima and navigate flat plateaus much faster than standard gradient descent. It accumulates velocity in directions with persistent gradients while canceling out noise in fluctuating directions. It is a foundational component of modern optimizers like Adam and RMSprop.