Second-Order Optimization
Optimization that uses the Hessian, or an approximation of it, to account for curvature and take a more direct step than the gradient alone allows.
Newton's method updates using the inverse Hessian times the gradient, converging on a purely quadratic function in a single step because the Hessian fully describes that surface's curvature. The Hessian of an n-parameter model is an n×n matrix, though, which is computationally out of reach at any modern model's scale — for 7 billion parameters that's on the order of 10^19 entries.
Quasi-Newton methods like L-BFGS approximate curvature from recent gradient history instead of computing the Hessian directly, and Adam's second moment is a cheap, diagonal-only stand-in for the same idea — a small fraction of what a true second-order method sees, but affordable at scale.