Training Compute-Optimal LLMs
The 'Chinchilla paper' that proved models should be scaled equally with training data, overturning the previous consensus that large models could be trained on relatively little data.
Paper: Training Compute-Optimal Large Language Models
Authors: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Nalisnick, Will Millstone, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Arthur Brockschmidt, Oriol Vinyals, Jack W. Rae, Laurent Sifre · 2022
Read the paper
The Problem
Following the Kaplan scaling laws (2020), the industry consensus was to heavily scale model parameters while scaling dataset size much more slowly. This led to models like Gopher (280B parameters) and Megatron-Turing NLG (530B parameters) which were massively oversized relative to the amount of data they were trained on (around 300B tokens). These models were expensive to train, and crucially, incredibly slow and expensive to run during inference because of their massive size.
The Idea
DeepMind researchers suspected that the original Kaplan scaling laws had under-sampled smaller models trained for very long periods, leading to skewed power law coefficients. They hypothesized that you should scale parameters () and data () at an equal rate to achieve the lowest possible loss for a given compute budget.
How It Works
They trained over 400 language models ranging from 70 million to 16 billion parameters on up to 500 billion tokens. By finding the lowest loss for various fixed compute budgets, they derived new, highly accurate scaling laws.
They discovered that the compute-optimal ratio is roughly 20 tokens per parameter. To test their new laws, they trained a new model called Chinchilla. Chinchilla had only 70 billion parameters (1/4 the size of Gopher) but was trained on 1.4 trillion tokens (4x the data of Gopher)—keeping the total training compute budget identical.
Why It Mattered
Chinchilla absolutely crushed Gopher on almost every evaluation metric. It proved the industry had been training oversized, under-trained models.
This was a massive win for efficiency. A 70B model requires far less VRAM and compute during inference (when users actually interact with it) than a 280B model. By trading a smaller parameter count for more training data, Chinchilla achieved better performance at a fraction of the deployment cost.
What Came After
The "Chinchilla-optimal" ratio became the gold standard for model training. Every major model that followed—like Meta's LLaMA (65B parameters trained on 1.4T tokens)—adhered to or exceeded this ratio.
Since then, the industry has pushed even further past Chinchilla-optimality. Researchers realized that while 20 tokens per parameter is optimal for training compute, if you care about inference compute, you should over-train a small model far beyond that ratio (e.g., Llama 3 8B trained on 15T tokens). Thus, pure compute-optimality has been superseded by inference-optimality.