Gen AI

Tokenizer Training

Before a model can learn weights to predict text, the tokenizer must learn a vocabulary from a corpus by repeatedly merging the most frequent symbol pairs.

Why Does This Exist?

Every neural network needs discrete numbers as input, which means text must be sliced into a fixed set of tokens. But who decides what those tokens are? If you hardcode a vocabulary of English words, the model fails on code and foreign languages. If you use single characters, sequences become impossibly long and expensive to process.

The modern solution is to learn the vocabulary directly from data. Tokenizer training is a separate, preliminary process that runs before the actual LLM training starts. It scans a massive corpus of text and uses a frequency-based algorithm to discover which combinations of characters appear most often. By doing this, the tokenizer automatically allocates its limited vocabulary budget to the fragments that are actually useful in the real world, rather than relying on human intuition about what a "word" or "syllable" should be.

Think of It Like This

Designing a shorthand alphabet for a specific job

Imagine you are hired as a court reporter. Before you start transcribing, you review a decade of past court transcripts. You notice the phrase "Your Honor" appears thousands of times, so you create a single shorthand squiggle for it. You notice "objection" is also very common, so it gets its own squiggle. But you never see the word "supercalifragilisticexpialidocious," so you don't bother creating a shortcut for it.

Tokenizer training is this exact process. It's a preparation phase where the system looks at the data it will eventually have to process and builds a custom dictionary of shortcuts. The LLM training is the actual court reporting; the tokenizer training is just designing the shorthand.

How It Actually Works

A separate process from model training

It is crucial to understand that tokenizer training and model training are completely independent. Tokenizer training involves no neural networks, no gradients, and no backpropagation. It is purely a statistical counting algorithm. Once the tokenizer is trained, its vocabulary and merge rules are frozen. Only then does the LLM training begin, using that frozen vocabulary to map text into numbers.

The Byte-Pair Encoding (BPE) algorithm

Most modern LLMs use a variant of Byte-Pair Encoding (BPE) to train their tokenizers. The process starts by breaking the entire training corpus down to its fundamental units—usually individual bytes or Unicode characters. The algorithm then enters a loop:

Scan the corpus and count the frequencies of all adjacent pairs of symbols.
Find the single most frequent pair (e.g., t and h co-occurring to form th).
Merge that pair into a new, single symbol and add it to the vocabulary.
Replace all occurrences of the pair in the corpus with the new symbol.
Repeat the process until the vocabulary reaches a predetermined target size (e.g., 50,000 or 100,000 tokens).

Because this process is entirely frequency-driven, common words like the and ing quickly become single tokens, while rare words remain split into smaller fragments.

The training corpus matters

A tokenizer is uniquely tailored to the data it was trained on. If you train a tokenizer entirely on English text, it will dedicate its vocabulary to English prefixes, suffixes, and common words. If you then ask that tokenizer to process Korean or Python code, it will perform terribly—it won't have the right "shorthand" available, so it will fall back to splitting the text into individual, inefficient characters. This is why multilingual models like GPT-4 require tokenizers trained on a carefully balanced mix of languages and code.

Show Me the Code

Here is a simplified Python script that demonstrates the core logic of training a tokenizer using a counting approach.

1from collections import Counter2import re3
4def get_stats(vocab: dict[str, int]) -> dict[tuple[str, str], int]:5    pairs = Counter()6    for word, freq in vocab.items():7        symbols = word.split()8        for i in range(len(symbols) - 1):9            pairs[symbols[i], symbols[i+1]] += freq10    return pairs11
12def merge_vocab(pair: tuple[str, str], v_in: dict[str, int]) -> dict[str, int]:13    v_out = {}14    bigram = re.escape(' '.join(pair))15    p = re.compile(r'(?<!\S)' + bigram + r'(?!\S)')16    for word in v_in:17        w_out = p.sub(''.join(pair), word)18        v_out[w_out] = v_in[word]19    return v_out20
21# Example corpus frequencies22vocab = {'l o w </w>': 5, 'l o w e r </w>': 2, 'n e w e s t </w>': 6, 'w i d e s t </w>': 3}23
24num_merges = 325for i in range(num_merges):26    pairs = get_stats(vocab)27    best = max(pairs, key=pairs.get)28    vocab = merge_vocab(best, vocab)29    print(f"Step {i+1}: Merged {best} -> {vocab}")30# -> Step 1: Merged ('e', 's')31# -> Step 2: Merged ('es', 't')32# -> Step 3: Merged ('est', '</w>')

In just three steps, the algorithm discovers the common suffix est</w> purely by observing its high frequency across the words "newest" and "widest".

Watch Out For

Assuming tokenizers understand language

Because BPE often produces tokens that look like recognizable prefixes or suffixes (like un or ing), it is tempting to think the tokenizer understands morphology. It does not. It is blindly counting co-occurrences. If a completely nonsensical combination of letters appears frequently enough in the training data, the tokenizer will merge it into a single token without hesitation.

Mismatched tokenizers in production

A trained tokenizer's vocabulary is a hard dependency for its corresponding LLM. If you use Llama 3's tokenizer to preprocess text for a Mistral model, the output will be complete garbage. The model expects token ID 405 to mean "apple", but the mismatched tokenizer might use 405 to mean "the". Always deploy the exact tokenizer artifact that was paired with the model during its training.

The Quick Version

Tokenizer training is a separate, statistical process that runs before the actual LLM training begins.
It learns a vocabulary by scanning a corpus and repeatedly merging the most frequent pairs of symbols.
This frequency-driven approach ensures common words get single tokens, while rare words are broken into smaller, reusable pieces.
A tokenizer is highly biased toward the data it was trained on; an English-only tokenizer will be extremely inefficient at processing code or other languages.