GPT & Autoregressive Models

Why Does This Exist?

Building a language model used to mean picking a task-specific objective for whatever you wanted the model to do — one setup for translation, another for summarization, another for question answering, each with its own labeled dataset. That approach hits a wall fast: labeled task data is expensive and comparatively small, while raw text is nearly unlimited. GPT's founding bet, going back to 2018, was that one simple, unsupervised objective — predict the next word, everywhere, on any text at all — would generalize far better than anyone expected, precisely because it never runs out of training signal.

That objective needs an architecture built for it. GPT is a decoder-only encoder-decoder-architectures-family model: every layer is causally masked, so a position can attend to itself and everything before it, never anything after. That restriction isn't a limitation bolted on afterward — it's what makes the objective coherent at all. Train a model to predict what comes next while letting it see the answer, and it learns nothing; block the answer and you get a genuine prediction task, one you can run at essentially unlimited scale on unlabeled text.

Think of It Like This

How It Actually Works

The objective: one word, over and over

GPT trains on causal language modeling: given tokens $1$ through $t$, predict token $t+1$, for every position in every training sequence at once. There's no separate "understanding" phase and "generation" phase — the exact same forward pass that predicts token 51 from tokens 1 through 50 during training is what runs, one step at a time, during actual generation. The diagram above shows this loop: predict, append the result, feed the whole sequence back in, predict again.

Why one objective generalizes so widely

Predicting the next word accurately across a huge and varied body of text isn't actually a narrow skill. To guess correctly that "the capital of France is ___" ends in "Paris," a model needs a fact. To guess that "she picked up the phone and ___" is more likely to continue with an action than a noun, it needs grammar. To finish a step-by-step math explanation correctly, it needs to have tracked the reasoning so far. None of these were separately labeled as training targets — they emerge as byproducts of getting extremely good at the one stated task, because getting that one task right, at scale, requires learning all of them anyway. This is the mechanism behind what how LLMs work covers at a higher level: a single training signal, applied at enough scale, becoming general capability.

Sampling, not just picking the top word

At inference, GPT doesn't always take the single highest-probability next token — that tends to produce repetitive, flat text. Instead it samples from the predicted probability distribution over the vocabulary, with various strategies controlling how much randomness to allow. The mechanics of that choice belong to next-token prediction; what matters here is that the underlying model output is always a full probability distribution over every possible next token, and generation is a policy for choosing from it.

The one-token-ahead limit

Training this way only ever rewards a model for getting the very next token right — it has no direct signal about whether token 51 sets up a good token 55. That short-horizon pressure is a real limit of the plain next-token objective, and it's the gap that motivates multi-token prediction: predicting several tokens ahead during training, not just one, without changing what runs at inference.

Show Me the Code

A toy autoregressive loop: predict a distribution over a tiny vocabulary from the last token, pick the top choice, append it, repeat.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max(axis=-1, keepdims=True))6    return e / e.sum(axis=-1, keepdims=True)7
8
9def generate_step(seq: np.ndarray, w: np.ndarray) -> np.ndarray:10    logits = seq[-1] @ w                  # predict from the last token only11    probs = softmax(logits)12    next_token = np.zeros_like(seq[-1])13    next_token[np.argmax(probs)] = 1.0     # greedy pick, for a deterministic demo14    return next_token15
16
17rng = np.random.default_rng(0)18w = rng.normal(scale=0.5, size=(4, 4))19seq = rng.normal(size=(3, 4))               # 3-token starting sequence20for step in range(2):21    seq = np.vstack([seq, generate_step(seq, w)])22print(seq.shape)  # -> (5, 4) — two new tokens appended, one at a time

Each new row is generated from everything before it and then appended — exactly the feed-forward, feed-back loop the diagram draws, run twice.

Watch Out For

The Quick Version

• GPT is a decoder-only, causally-masked transformer trained on one objective: predict the next token, everywhere, in any text.
• The same forward pass used in training runs at inference, one step at a time, feeding each generated token back in as input.
• Getting next-token prediction right at scale forces the model to implicitly learn grammar, facts, and reasoning patterns, without any of those being separate training targets.
• Generation samples from a predicted probability distribution rather than always taking the top choice.
• The objective only ever rewards the very next token, which is the gap multi-token prediction exists to address.

What to Read Next

• BERT is the encoder-only counterpart trained on the opposite kind of objective, masked language modeling.
• Multi-Token Prediction addresses the one-token-ahead limit this page names as the objective's real constraint.
• Encoder-Decoder Architectures is where GPT's decoder-only design sits among the other transformer variants.
• How LLMs Work covers the broader picture of how this one training signal becomes general capability.
• Next-Token Prediction covers the sampling strategies this page's generation step glosses over.