Batch Size Effects
How the number of examples averaged into one gradient estimate changes training noise, the largest stable learning rate, and final generalization.
A minibatch gradient's noise falls as one over the square root of the batch size, while the number of steps per epoch falls in direct proportion to it — a much steeper drop. Doubling the batch only cuts noise by about 1.4x, but it also halves the number of steps a fixed dataset produces.
Past a certain size, very large batches are repeatedly observed to reach worse test performance than smaller ones trained to the same training loss — the noise a small batch introduces acts as a mild regularizer that a large, low-noise batch loses.