SGD
Stochastic Gradient Descent is a foundational optimization algorithm that updates model weights iteratively using the gradient of a single random data batch.
Think of It Like This
Like a blindfolded hiker taking a quick step downhill based only on the slope right under their foot, rather than surveying the whole mountain.
Unlike standard Gradient Descent which calculates the error across the entire dataset before moving, SGD moves much faster but with a noisy, erratic trajectory. This noise actually helps the model escape shallow local minima in complex loss landscapes. It is typically enhanced with Momentum or adaptive learning rates (like Adam).