Gen AI

Next-Token Prediction

Train a model on one simple task, predict the next token from everything before it, and getting that one prediction right at scale quietly forces grammar, facts, and style into the same weights.

One training signal repeated at every position in a document teaches grammar, facts, and style all at once, because predicting the next word correctly requires all three

Why Does This Exist?

Training a model to do something useful with language usually seems like it should require task-specific labeled data — pairs of questions and answers, articles and summaries, sentences and their sentiment. That data is genuinely scarce relative to the volume of raw text available, and hand-labeling more of it doesn't scale to internet-sized corpora.

Next-token prediction sidesteps the labeling problem entirely by manufacturing its own supervision from ordinary text. Take any document nobody labeled — a book, a webpage, a forum post — and at every single position, the "label" is simply whatever word comes next. No human decided that; it's just what the text already says. A billion words of raw text becomes a billion-plus individual training examples, for free, which is precisely the scale next-token prediction needed to work as well as it does.

Think of It Like This

A game of finish-the-sentence, played on every sentence ever written

Imagine a game where you're shown a sentence with the last word hidden, and you have to guess it — "The capital of France is ___." Play this game on one sentence and you learn almost nothing generalizable. Play it across every sentence in every book in a large library, and something else happens: to keep winning, you're forced to absorb grammar (what part of speech fits here), facts (Paris, specifically), and style (how a travel guide continues versus how a legal document continues) — not because anyone set out to teach you those things separately, but because guessing the hidden word correctly, over and over, across every kind of sentence, requires all of them at once.

Next-token prediction is that game, played on effectively the entire readable internet, at a scale where "just guess the next word" quietly builds something much larger than a next-word guesser.

How It Actually Works

One loss, applied at every position

For a document with tokens $x_1, \ldots, x_n$ , the training objective is to maximize the probability the model assigns to each true next token, given everything before it:

\mathcal{L} = -\sum_{t=1}^{n-1} \log p_\theta(x_{t+1} \mid x_1, \ldots, x_t)

This is ordinary cross-entropy, summed across every position in the document in one pass — not one prediction per document, but one prediction per token. A 2,000-token document supplies roughly 2,000 separate training signals simultaneously, each one graded against the actual next word that was already sitting right there in the text.

Why one objective produces many capabilities

The diagram above makes the mechanism concrete: getting "Seine" right after "...a city on the" isn't answerable from grammar alone — plenty of nouns are grammatically valid there. It requires the geographic fact, and it requires recognizing the travel-guide register the passage is written in, because a legal document would continue differently even in the same grammatical slot. None of these skills is taught as a separate task. They're all necessary conditions for consistently predicting the next token correctly across a large and varied enough corpus, so gradient descent has no choice but to build representations that capture them, as a side effect of optimizing the one loss above.

What makes this scale differently from labeled training

Ordinary supervised learning needs a human (or an existing system) to produce every label, which caps how much labeled data can exist. Next-token prediction's labels are the text itself — every sentence anyone has ever written already comes pre-labeled with what word follows every other word. That's the entire reason this objective could be trained at a scale labeled datasets never approached, and scale, empirically, is a large part of where the resulting capability came from.

The gap this doesn't close

It's worth being precise about what this objective optimizes for: it rewards plausible continuations, not necessarily true ones. A next-token predictor trained purely this way has no built-in mechanism distinguishing "the most statistically likely next word" from "the factually correct next word" when the two diverge — which is a large part of why hallucination is a structural property of this objective rather than a bug introduced somewhere downstream. That's a separate page's worth of detail, but it follows directly from the objective stated here.

Show Me the Code

Computing the next-token loss over one short sequence, position by position, so the "one prediction per token" claim is visible in the loop.

1import numpy as np2
3
4def cross_entropy_at_position(logits: np.ndarray, true_next_id: int) -> float:5    probs = np.exp(logits - logits.max())6    probs /= probs.sum()7    return -np.log(probs[true_next_id] + 1e-12)8
9
10token_ids = [7, 12, 3, 44, 9]                      # a five-token sequence11rng = np.random.default_rng(0)12vocab_size = 5013losses = []14for t in range(len(token_ids) - 1):15    logits = rng.normal(size=vocab_size)            # stand-in for the model's real output16    true_next = token_ids[t + 1]                     # the label is just "the next token in the text"17    losses.append(cross_entropy_at_position(logits, true_next))18
19print(len(losses))          # -> 4 — one loss term per position with a "next token" to predict20print(round(sum(losses), 2))  # -> total loss summed across every position, one document, one pass

Four loss terms from a five-token sequence, with zero labeling effort beyond the text already existing — that ratio is the entire economic argument for this objective at internet scale.

Watch Out For

Confusing plausible with true

Because the objective is graded entirely on how well the model's predicted distribution matches what real text says next, a model trained this way has no separate mechanism for verifying factual accuracy — it's rewarded for statistically likely continuations, and most of the time plausible and true line up, because most training text is true. When they don't line up, or when a prompt pushes into territory the training distribution didn't cover well, the model still produces a plausible-sounding continuation, because that's the only thing the objective ever asked it to do.

Assuming more parameters trained on the same objective always improves everything uniformly

Scaling up a next-token predictor doesn't improve every capability at the same rate — some abilities appear to develop gradually and predictably with scale, while others seem to appear more suddenly at particular scale thresholds, a genuinely debated phenomenon covered on its own page. Treating "bigger model, same objective" as a uniform, linear improvement across every task undersells how unevenly capability actually develops as scale increases.

The Quick Version

Next-token prediction manufactures training labels from raw text itself — the "label" at each position is just whatever word comes next.
The loss is cross-entropy, summed across every position in a document in one pass, not once per document.
Predicting correctly across a large, varied corpus requires grammar, facts, and style simultaneously, which is why one objective builds many capabilities.
This objective scales far past labeled datasets because unlabeled text supplies its own supervision, for free.
The objective rewards plausible continuations, not verified true ones — a structural property, not a downstream bug.