AI360Xpert
Gen AI

Tokenization

How a language model chops your text into bite-sized pieces called tokens, then swaps each one for a number it can actually do math on.

The word 'unbelievable' is split into the subword tokens un, believ, and able, and each token is mapped to an integer id
The word 'unbelievable' is split into the subword tokens un, believ, and able, and each token is mapped to an integer id

Why Does This Exist?

A language model does math on numbers, not letters. So before it can read a single word, your text has to become a list of numbers. Tokenization is that first, unglamorous step, and getting it right matters more than it looks.

The obvious approaches both fail. Split on whole words and your vocabulary explodes: every plural, typo, and rare name needs its own slot, and the model chokes the first time it meets a word it never saw. Split on individual characters and sequences get painfully long, so the model wastes its effort spelling instead of understanding.

Modern models take the middle path with subword tokenization. Common words stay whole, and rare ones break into reusable fragments. That keeps the vocabulary a fixed, manageable size while still being able to represent literally any string, even one it has never encountered.

Think of It Like This

LEGO bricks for language

You can build almost anything from a modest box of LEGO bricks. You don't need a custom piece for every possible shape, just a good set of common bricks you snap together. Subword tokenization gives the model a fixed box of text "bricks": frequent words are single big pieces, and anything unusual gets assembled from smaller ones. "unbelievable" isn't in the box, so it's built from un + believ + able.

How It Actually Works

The most common recipe is Byte Pair Encoding (BPE). It learns its vocabulary from data before any model training happens:

  1. Start from characters. Every word begins as a sequence of individual characters.
  2. Count pairs. Scan a large corpus and find the most frequent adjacent pair of symbols.
  3. Merge it. Fuse that pair into a single new token and add it to the vocabulary.
  4. Repeat. Keep merging the next-most-frequent pair until the vocabulary hits its target size, often 30,000 to 100,000 tokens.

The result: frequent chunks like ing, the, and whole common words become single tokens, while rare words fall back to smaller pieces. At runtime the tokenizer greedily matches your text against this learned vocabulary, then swaps each token for its integer id. Those ids are what the model actually reads, and each one is later turned into an embedding vector.

From text to model input

Step 1 of 3

Split

Break the raw string into the largest subword pieces that exist in the vocabulary.

Show all steps
  1. Split: Break the raw string into the largest subword pieces that exist in the vocabulary.
  2. Map to ids: Look up each token's integer id — the model's actual input is this list of numbers.
  3. Embed: Each id becomes a learned vector, ready for the attention layers to process.

Show Me the Code

def bpe_merge(tokens: list[str], pair: tuple[str, str]) -> list[str]:    merged, i = [], 0    while i < len(tokens):        # If this position matches the learned pair, fuse it into one token.        if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:            merged.append(tokens[i] + tokens[i + 1])            i += 2        else:            merged.append(tokens[i])            i += 1    return merged

chars = list("lower")  # start from charactersprint(bpe_merge(chars, ("e", "r")))  # -> ['l', 'o', 'w', 'er']

That's one merge step. Real training applies thousands of them in order, and encoding just replays the learned merges on new text.

Watch Out For

Token count is not word count

A "4,000 token" context window is not 4,000 words. English averages roughly 0.75 words per token, and code, punctuation, or non-English scripts can be far denser. Whitespace and capitalization also change the split, so " Cat" and "cat" may be different tokens. Always measure prompt length in tokens, not words.

Tokenizers make models bad at spelling

Because the model sees strawberry as a couple of chunks rather than letters, questions like "how many r's are in strawberry?" trip it up. It never saw the individual letters. The tokenizer, not the model's intelligence, is usually the culprit behind these odd character-level failures.

The Quick Version

  • Models do math on numbers, so text must first be split into tokens and mapped to integer ids.
  • Subword tokenization (like BPE) keeps a fixed vocabulary while still representing any string.
  • Common words are single tokens; rare words break into reusable fragments.
  • Token count differs from word count, which is what fills a context window.
  • Because tokens aren't letters, models struggle with spelling and character counting.

What to Read Next