Gen AI

Tokenization

Cut text into discrete units the model can index — word-level, character-level, or subword — and the choice among them trades vocabulary size against sequence length.

The same sentence splits into fewer, larger pieces at word level, many more, tiny pieces at character level, and a middle number of frequency-learned fragments at subword level

Why Does This Exist?

A neural network needs numbers as input, not raw characters, so text has to be split into discrete units before anything else in how an LLM works can happen. The obvious first idea is splitting on whitespace — one token per word. That runs into a wall almost immediately: language keeps producing new words. Names, typos, made-up words, technical jargon, and text in languages the training corpus barely covered all generate words the model has never seen, and a word-level vocabulary has no slot for them. Whatever fixed vocabulary gets built at training time, real text will eventually contain a word outside it.

The opposite extreme — splitting into individual characters — solves the coverage problem completely; there's a finite, small set of characters, so nothing is ever "unseen." But it creates a different problem: a sentence that was 10 words becomes roughly 50 characters, and every one of those 50 positions costs compute in an architecture whose attention cost grows quadratically with sequence length. Character-level tokenization trades away the out-of-vocabulary problem for a much longer, much more expensive sequence.

Think of It Like This

Packing a suitcase with fixed-size containers

Word-level tokenization is packing with large, pre-shaped containers — efficient when your items match the container shapes exactly, but useless the moment you have something oddly shaped that doesn't fit any container you own. Character-level tokenization is packing everything as individual loose items — nothing ever fails to fit, but you now need vastly more containers to hold the same stuff, and unpacking takes forever.

Subword tokenization is packing with a set of medium, reusable modular pieces learned from experience — common shapes get their own dedicated piece, and anything unusual just gets broken down into a few of those reusable pieces instead of failing to pack, or being broken all the way down to individual items. Most of the time you're using large, efficient pieces; only for genuinely novel items do you fall back to smaller ones.

How It Actually Works

Three levels, one tradeoff axis

The diagram above lays out the same sentence at all three granularities. Word-level tokenization splits on whitespace and punctuation: fewest tokens per sentence, but the vocabulary has to grow to cover every word that might appear, and it never fully succeeds — there's always an unseen word waiting. Character-level tokenization has a tiny, fixed vocabulary (every character a script uses) and zero out-of-vocabulary tokens, at the cost of sequences several times longer than the word-level version for the same text. Subword tokenization sits between them: a vocabulary of tens of thousands of frequently occurring fragments, learned from a training corpus rather than hand-specified, where common whole words get their own single token and rare or unseen words decompose into a small number of familiar pieces.

Why subword won

The practical argument for subword tokenization is that it makes the out-of-vocabulary problem disappear without paying character-level's full sequence-length cost. A word the vocabulary has never seen as a whole unit — tokenizing, say — decomposes into pieces the vocabulary does know, like token and izing, rather than failing outright or falling back to dozens of individual characters. Common words stay compact (often one token each), and only genuinely rare or novel text pays the cost of more, smaller pieces. Byte pair encoding is the algorithm most subword vocabularies are actually built with, learning which fragments are frequent enough to deserve their own token directly from a training corpus.

The consequence people underestimate

Token count is not word count, and the gap isn't uniform across languages or content types. A tokenizer's vocabulary is learned from a training corpus that's rarely balanced across scripts — merges learned predominantly from English prose leave less-represented languages and scripts with less efficient coverage, so the same sentence translated into an underrepresented language can cost meaningfully more tokens for identical meaning. Structured formats pay a related cost: JSON's braces, quotes, and colons are themselves tokens, so a data payload can cost noticeably more per character than equivalent prose. Since context windows and generation cost are both measured in tokens, not characters or words, this gap has real, measurable consequences for cost and capacity that a naive "how long is this text" estimate will get wrong.

Show Me the Code

A simplified word-level and character-level split side by side, to make the length difference concrete rather than asserted.

1def word_tokenize(text: str) -> list[str]:2    return text.split()                                     # crude: whitespace only3
4
5def char_tokenize(text: str) -> list[str]:6    return list(text.replace(" ", "_"))                       # every character, spaces marked7
8
9text = "tokenizing rare words costs more"10words = word_tokenize(text)11chars = char_tokenize(text)12print(len(words), "word tokens:", words)13print(len(chars), "char tokens:", "".join(chars))14# -> 5 word tokens: ['tokenizing', 'rare', 'words', 'costs', 'more']15# -> 33 char tokens: tokenizing_rare_words_costs_more

Five word-level tokens against 33 character-level tokens for the identical sentence — a subword tokenizer would land somewhere in between, likely close to the word count for common words like rare and more, and slightly higher only for less frequent ones.

Watch Out For

Estimating cost or context usage in words instead of tokens

Pricing, rate limits, and context-window limits for essentially every LLM API are measured in tokens, not words or characters, and the word-to-token ratio isn't fixed — it varies by language, by how much punctuation or code is present, and by which specific tokenizer a model uses. Estimating "this document is 2,000 words, so it's about 2,000 tokens" is a common, costly assumption; the real number is often 25 to 40 percent higher for English prose, and can be far higher for other languages or structured data.

Mixing tokenizers across a pipeline

A vocabulary and its merge rules are specific to the tokenizer they were trained with, and swapping tokenizers between two components of a pipeline — using one model's tokenizer to estimate token counts for a different model's API call, say — produces silently wrong counts, because the two tokenizers segment the same text differently. Even a version bump to the same tokenizer's normalization rules can change token counts for identical input. Pin the exact tokenizer version alongside the model it belongs to.

The Quick Version

Word-level tokenization has an unavoidable out-of-vocabulary problem; character-level avoids it but multiplies sequence length.
Subword tokenization learns a vocabulary of frequent fragments from a corpus, sitting between the two extremes.
Common words get compact, often single-token representations; rare or unseen words decompose into familiar pieces instead of failing.
Token count is not word count, and the gap is uneven across languages and formats — underrepresented scripts and structured data like JSON both cost more per character.
Context limits and API pricing are measured in tokens, so estimating cost in words or characters is unreliable.