Llama 3
The 2024 Meta technical report detailing a massive scale-up in data and compute, proving that dense models can reach frontier-level capabilities through sheer scale and data quality.
The Problem
By 2024, the frontier of AI was dominated by proprietary models like GPT-4 and Claude 3. While open models like Llama 2 and Mixtral were good, they were decidedly a tier below the frontier. Many believed open-source would permanently lag behind due to the staggering compute costs required to train GPT-4 class models. Furthermore, it was unclear how far standard, dense Transformer architectures could be pushed before hitting diminishing returns.
The Idea
Meta proved that standard, dense Transformers can reach the absolute frontier of AI if fed enough high-quality data and compute. The Llama 3 models (ranging from 8B to 405B parameters) didn't rely on radical architectural changes (they didn't use Mixture-of-Experts). Instead, Meta focused relentlessly on data quality, data scaling, and an incredibly complex alignment pipeline involving Direct Preference Optimization (DPO).
How It Works
Key innovations of the Llama 3 herd:
- Data Scaling (Over-training): The Chinchilla laws suggest an 8B model needs ~160B tokens to be "compute optimal." Meta trained Llama 3 8B on 15 Trillion tokens (nearly 100x over-trained). This resulted in a tiny model with the knowledge and reasoning capacity of much larger legacy models, making it incredibly cheap to run.
- Tokenizer: They expanded the vocabulary size to 128K tokens (from 32K), utilizing a tiktoken-based BPE. This made the model much more efficient at processing text, particularly non-English languages.
- Architecture: They maintained a standard dense Transformer but made Grouped-Query Attention (GQA) standard across all sizes to improve inference speed.
- Alignment: The post-training pipeline was massive, heavily utilizing AI-generated data. They used strong models to generate synthetic training data, aggressively filtered it for quality, and utilized DPO (Direct Preference Optimization) alongside PPO to align the models.
Why It Mattered
Llama 3 405B became the first open-weights model to truly rival GPT-4 and Claude 3.5 Sonnet across a broad spectrum of benchmarks. More importantly, the heavily over-trained 8B and 70B models redefined what was possible on consumer hardware and single-node enterprise servers, cementing Meta's dominance in the open AI ecosystem.
What Came After
Llama 3 set the bar for the current generation of models. The concept of massively "over-training" small models became the industry standard (as seen in Qwen2 and Gemma 2) because inference efficiency is more economically valuable than training efficiency.