Mixtral of Experts
Detailed the architecture and training of Mixtral 8x7B, a highly performant open-weight sparse MoE model that proved smaller active parameter counts could rival dense giants.
Paper: Mixtral of Experts
Authors: Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed · 2024
Read the paper
The Problem
While Switch Transformer (2021) proved that sparse Mixture of Experts (MoE) could scale to a trillion parameters, building and deploying production-grade MoE models was mostly restricted to well-funded AI labs (like Google and OpenAI). The open-weight community relied entirely on dense models like LLaMA. Dense models are reliable, but a 70B parameter dense model requires activating 70B parameters for every single token, making inference slow and expensive.
The Idea
Mistral AI wanted to bring the efficiency of sparse MoEs to a state-of-the-art open-weight model. They aimed to create a model that had a large total parameter count (for high capacity and knowledge retention) but a small active parameter count per token (for fast, cheap inference).
How It Works
Mixtral 8x7B is a decoder-only sparse Mixture of Experts model.
- Architecture: Instead of a standard Feed-Forward Network (FFN), every layer contains 8 distinct FFN "experts".
- Routing: A router network computes scores for all 8 experts. Unlike Switch Transformer's top-1 routing, Mixtral uses top-2 routing. The token is sent to the 2 experts with the highest scores.
- Combination: The outputs from the two experts are weighted by their routing probabilities and summed.
Despite the name "8x7B", the model does not have 56 billion parameters. Self-attention layers are shared across all experts, so the total parameter count is 47 billion. Because only 2 experts are active per token, the active parameter count per token is only 13 billion.
Why It Mattered
Mixtral 8x7B was a massive success. It vastly outperformed Llama 2 70B and matched or beat GPT-3.5 on most benchmarks, despite using 5x fewer active parameters during inference than Llama 2 70B.
It popularized MoE for the open-source community, proving that deploying a highly intelligent model didn't require massive GPU clusters if the architecture was sparse. It also validated that top-2 routing struck a perfect balance between computational efficiency and representational stability.
What Came After
Mixtral firmly established MoE as the dominant architecture for frontier models, open or closed. Mistral later released Mixtral 8x22B, scaling the exact same architecture up.
While the standard top-2 MoE approach is still widely used, it is currently being challenged and evolved by models like DeepSeek-V2/V3, which use "fine-grained" MoE (routing to, for example, 8 out of 160 much smaller experts) to achieve even greater parameter isolation and efficiency.