Sparse Model
A neural network where a vast majority of the weights are zero or dynamically inactive during any given forward pass, drastically reducing active compute.
Think of It Like This
Like a massive university with thousands of specialized professors, but a student only ever talks to three of them for a specific major.
Mixture of Experts (MoE) is the most famous example of a sparse model. By routing tokens to only 2 out of 8 experts, the model maintains the knowledge capacity of a 50-billion parameter network while running as fast as a 12-billion parameter one. It is the primary architectural paradigm scaling modern frontier LLMs.