Adam / AdamW
An incredibly popular adaptive optimization algorithm that combines momentum and scaling to dynamically adjust the learning rate of individual parameters.
Think of It Like This
Like a driver who remembers their past speed (momentum) while adapting to the current road conditions (scaling) for each individual tire.
Standard Adam is the default optimizer for most deep learning workloads because it converges rapidly and requires minimal hyperparameter tuning. However, its original implementation of weight decay was flawed, leading to the creation of AdamW, which decouples weight decay from the gradient update. AdamW is now the standard choice for training transformers and other large models because it generalizes significantly better.