Double Descent
A phenomenon where a model's test error initially decreases, then spikes as it interpolates the data, and finally decreases again as model capacity expands.
Think of It Like This
Like learning to ride a bike: you improve, then suddenly crash a lot as you try advanced tricks, before finally mastering it perfectly.
The double descent curve challenges the classical bias-variance tradeoff. It suggests that heavily overparameterized models eventually reach a regime where adding more capacity acts as an implicit regularizer. This explains why extremely massive deep neural networks often generalize well even when they have enough parameters to perfectly memorize the training set.