Generative AI
Tokenization (BPE)
How models see text as statistically common subword chunks formed by merging frequent character pairs.
Stage 1 of 2: Characters
Current text has 8 tokens. Next merge rule applies to "aa".
- Token
Language models start by splitting text into characters, the smallest atomic units.
Check your understanding
Type any text and watch it get broken into subword tokens live, showing merge rules being applied step by step.
Byte Pair Encoding (BPE) is the algorithm that most modern large language models use to see the text we type. Instead of breaking text into characters (which is too fine-grained and produces huge context lengths) or into words (which fails on typos, punctuation, and new words), BPE finds the perfect middle ground.
By starting with characters and iteratively merging the most frequently adjacent pairs in the training corpus, the tokenizer builds a vocabulary of subwords. Common words might be a single token, while rare ones are split into pieces.
This lab lets you watch training itself: at each step it counts how often every adjacent pair occurs in your text and merges whichever pair repeats most, exactly like a real BPE trainer would over a corpus. When no pair repeats anymore, there is nothing left to merge.
Reference
- Token
- A chunk of text (character, subword, or word) assigned a unique integer ID.
- BPE
- Byte Pair Encoding: a data compression algorithm adapted to learn subword vocabularies.
- Merge rule
- The single most frequent adjacent pair in the current sequence, replaced everywhere it occurs.
- Tie-break
- When several pairs are tied for the highest count, the one whose first occurrence starts earliest in the sequence is merged.
- Stopping condition
- Merging halts once no adjacent pair occurs more than once — the vocabulary this text can produce is exhausted.
Break it on purpose
Type text where no adjacent character pair repeats (e.g. random characters like "xyz") and the algorithm has nothing to merge, so it stays broken into individual characters.