Batch vs Mini-Batch vs Stochastic
How much data should the model look at before taking a step down the gradient?
Verdict: Mini-Batch is the industry standard. It balances the stable gradients of Batch with the fast updates and hardware efficiency of Stochastic.
The Short Answer
When training a neural network using gradient descent, you have to decide how many data points to look at before updating the model's weights.
- Batch Gradient Descent: Look at the entire dataset. Calculate the exact average gradient. Take one perfect step. Repeat.
- Stochastic Gradient Descent (SGD): Look at exactly one random data point. Calculate a very noisy gradient based on just that point. Take a step. Repeat.
- Mini-Batch Gradient Descent: Look at a small chunk of data (e.g., 32, 64, or 256 points). Calculate a fairly accurate gradient. Take a step. Repeat.
Where They Differ
| Feature | Batch | Mini-Batch | Stochastic (SGD) |
|---|---|---|---|
| Batch Size | N (Whole dataset) | 16 to 1024 | 1 |
| Path to Minimum | Smooth and direct. | Slightly noisy. | Highly erratic. |
| Update Frequency | Once per epoch. | Many times per epoch. | N times per epoch. |
| Memory Required | Massive (must hold entire dataset in RAM). | Moderate (fits in GPU VRAM). | Minimal. |
| Vectorization | High (fully utilizes GPU). | High (fully utilizes GPU). | Poor (can't parallelize 1 sample). |
Gradient Noise as a Feature, Not a Bug
It might seem like the smooth, perfect path of Batch Gradient Descent is the ideal scenario, and the noisy path of SGD is a necessary evil. In modern deep learning, the opposite is true.
Noise is actually a knob you want to tune.
If your loss landscape has "local minima" (shallow divots that aren't the true bottom), a perfectly smooth Batch path might walk straight into one and get stuck.
The erratic jumping of SGD and Mini-Batch acts as a form of exploration. The "noise" injected by only looking at a subset of data literally bounces the model out of bad local minima, helping it find a better global solution.
Choose Batch When
- Your dataset is very small: If your entire dataset fits easily into memory (e.g., 1,000 rows of tabular data), Batch is perfectly fine and mathematically stable.
- You are using strict convex optimization: Problems where there is mathematically only one global minimum (like standard Linear Regression).
Choose Mini-Batch When
- You are training a Neural Network: This is the default 99.9% of the time.
- You want to maximize GPU utilization: GPUs are designed to perform matrix math in parallel. Passing 1 sample (SGD) wastes 99% of the GPU's cores. Passing 32 or 64 samples at once perfectly fills the GPU's memory and parallel processing lanes.
Choose Stochastic (SGD) When
- You are in a strict online learning setting: Data streams in one by one and you must update the model immediately in real-time before throwing the data away.
- You have massive redundancy in your data: If every sample looks almost identical, calculating the gradient on 100 of them is a waste of time. One sample gives you the exact same directional signal.
What People Get Wrong
Confusing the algorithm 'SGD' with the batch size
In popular deep learning frameworks like PyTorch, the optimizer is called optim.SGD(). People often think this means they are using pure Stochastic Gradient Descent (batch size of 1). However, if you pass a data loader with batch_size=32 to the SGD optimizer, you are actually performing Mini-Batch Gradient Descent.