Superposition
A phenomenon where a neural network represents far more abstract concepts than it has physical dimensions by storing them in nearly orthogonal directions.
Think of It Like This
Like packing 100 different outfits into a tiny suitcase by carefully folding them so their shapes perfectly interlock without ruining each other.
Because a model might need to understand millions of concepts but only has a hidden dimension of 4096, it uses 'polysemantic' neurons. These neurons fire for multiple unrelated concepts. Resolving this superposition into human-understandable, single-concept features is the primary goal of Sparse Autoencoders in interpretability research.