Better & Faster LLMs via Multi-token Prediction

The Problem

Since GPT-1, the fundamental training objective for virtually all language models has been the same: predict the single next token.

While this objective has proven remarkably scalable, it has two drawbacks. First, predicting just the immediate next token encourages models to focus on local, short-term grammar rather than long-term planning and logical structure. Second, during inference, predicting one token at a time is slow. (Speculative decoding speeds this up, but requires training and hosting a completely separate, smaller "draft" model).

The Idea

The Meta FAIR researchers proposed modifying the training objective. Instead of just predicting token $t+1$, what if the model is forced to simultaneously predict tokens $t+1$, $t+2$, $t+3$, and $t+4$?

By forcing the model to predict further into the future, the network must learn deeper, more robust representations of the text that capture long-term structure and logic. As a massive bonus, because the model learns to generate multiple tokens at once, it natively learns to act as its own speculative decoding "draft model."

How It Works

The architecture is a standard Transformer trunk, but with a crucial modification at the very top.

Instead of a single output language-modeling head, the model has $N$ independent output heads (e.g., 4 heads).

• Head 1 predicts token $t+1$.
• Head 2 predicts token $t+2$.
• Head 3 predicts token $t+3$.
• Head 4 predicts token $t+4$.

The loss is calculated across all heads simultaneously during training. The researchers found that applying this objective to small models actually hurts performance, but as model scale increases (e.g., beyond 7B parameters), multi-token prediction consistently outperforms standard next-token prediction, especially on coding and reasoning benchmarks.

At inference time, the model can use its extra heads to simultaneously generate 4 candidate tokens, which are then immediately verified by the main trunk in the next step, resulting in a 3x speedup via self-speculative decoding.

Why It Mattered

This paper provided the first strong empirical evidence at scale that the classic "next token prediction" objective is not the final, optimal way to train language models.

It proved that modifying the loss function to encourage forward-planning yields models that are strictly "better" (higher reasoning scores) and "faster" (built-in speculative decoding) without requiring more training data or larger models.

What Came After

This paper represents the cutting edge of current LLM training research. The multi-token prediction objective is highly likely to become a standard component in the pre-training pipelines of the next generation of frontier models, as labs seek to extract more intelligence and inference speed out of the same compute budget.