T5 & Text-to-Text Transformers

Why Does This Exist?

Before T5, building an NLP system meant building several different systems. A translation model had an encoder plus a decoder that generates a sentence. A classification model had an encoder plus a small linear layer that outputs one of a fixed set of labels. A named-entity tagger had yet another head, predicting a tag per token. Each task got its own output shape, its own loss, and often its own architecture tweak, because "translate a sentence" and "pick one of five labels" look like fundamentally different problems.

That's the wall: every new task needs a new output head, and every output head is bespoke plumbing that has to be designed, trained, and maintained separately, even when the underlying encoder-decoder architecture doing the heavy lifting is identical. T5 — Text-to-Text Transfer Transformer, from a 2019 Google paper — removes the wall with one move: stop treating classification, regression, and generation as different kinds of output. Treat everything as text. The input is a string, the output is a string, always, whatever the task actually is.

Think of It Like This

How It Actually Works

Every task becomes an input string, plus a prefix naming the job

T5 attaches a short natural-language prefix to the input so one model can tell tasks apart without any structural change: "translate English to German: That is good.", "summarize: <article text>", "cola sentence: The cat sat." for a grammar-acceptability check. The prefix is just more text, tokenized and fed in exactly like the rest of the input — there's no special "task ID" field, no branching code path. Whatever tells the model which job this is has to live inside the string itself.

The architecture underneath is an ordinary encoder-decoder

T5 is built on the full encoder-decoder architecture: an unmasked encoder reads the entire prefixed input, and a causally-masked decoder generates the output one token at a time, cross-attending into the encoder's representation at every layer. Nothing about that stack is new or task-text-specific — it's the same shape BERT's encoder half and GPT's decoder half are each one piece of. What changes between T5 and a plain translation model isn't the architecture; it's what gets fed in and what gets asked of the output.

Classification comes out as a generated label word

This is the piece that actually removes the wall. A sentiment classifier normally ends in a small linear layer mapping a hidden vector to a fixed number of classes — two numbers, softmaxed, done. T5 has no such layer anywhere. Instead, the decoder generates text one token at a time, same as it would for a translation, and the label itself is the generated text: the literal word "positive" or "negative," produced through the exact same next-token distribution the decoder always produces. There's no separate classification head to design, because the vocabulary already contains the words "positive" and "negative," and generating one of them is no different, mechanically, than generating any other word.

One model, one loss, mixed training data across tasks

Because every task's target is now a string, every task's loss is the same one — the standard cross-entropy over next-token predictions in the decoder. That lets T5 train on translation examples, summarization examples, and classification examples in the same batches, under the same objective, without any task-specific loss term or task-specific layer to route between. The training signal that used to be split across several different output shapes now flows through one.

Show Me the Code

A minimal illustration of the classification-as-generation trick: restrict the decoder's output distribution to just the label tokens, and the argmax over that smaller set is the "classification."

1import numpy as np2
3
4def softmax(logits: np.ndarray) -> np.ndarray:5    shifted = logits - logits.max()6    return np.exp(shifted) / np.exp(shifted).sum()7
8
9rng = np.random.default_rng(0)10vocab_size = 1011logits = rng.normal(size=vocab_size)          # one score per vocabulary token12label_token_ids = np.array([3, 7])            # token ids for "negative"=3, "positive"=713
14full_probs = softmax(logits)15print(full_probs.argmax())                    # -> 6 -- an unrelated token wins over the full vocabulary16
17label_probs = softmax(logits[label_token_ids])18winner = label_token_ids[label_probs.argmax()]19print(winner)                                 # -> 7 -- "positive" wins once generation is restricted to label words

Nothing here is a classifier in the traditional sense — it's the same decoder, the same softmax over the vocabulary, just read out through the two tokens that happen to spell the labels.

Watch Out For

The Quick Version

• T5 treats every NLP task — translation, classification, summarization — as generating a string from a string, with no per-task output head.
• A short text prefix tells the model which task this is; there's no separate task-ID mechanism.
• The architecture underneath is an ordinary encoder-decoder, unrelated to the text-to-text framing itself.
• Classification is generation restricted to a small set of label words, scored through the same decoder softmax as anything else.
• One shared loss across every task lets mixed-task data train in the same batches.

What to Read Next

• Encoder-Decoder Architectures is the underlying stack T5's text-to-text framing runs on top of.
• GPT shows the same "generate the label as text" trick applied to a decoder-only model instead.
• BERT is the encoder-only alternative that T5's text-to-text approach was designed to replace for a shared multi-task setup.
• Transformer Architecture is the block both the encoder and decoder stacks are built from.