Gradient Descent
A foundational optimization algorithm used in machine learning to minimize loss functions by iteratively adjusting model parameters in the steepest direction.
Think of It Like This
Like a blindfolded hiker trying to find the bottom of a deep valley by repeatedly taking small steps in whichever direction feels like the steepest downhill.
Gradient descent computes the derivative of the loss function with respect to the network's weights. By updating the weights in the opposite direction of the gradient, the algorithm slowly navigates the high-dimensional loss landscape toward a minimum. Variants like Stochastic Gradient Descent (SGD) and Adam are heavily utilized to speed up this process.