Visual explainer

Tokenization

How raw text is split into discrete chunks before being processed by neural networks, and why this mechanism creates strange failure modes.

Models read IDs like 234, not letters like A and I.

You type AI is cool and the net cannot touch letters. It needs numbers, so you'll slice that string into pieces and look each piece up as an ID.

Pieces Get Numbers

Your vocab maps each piece to one fixed ID number.

You keep a vocab table that pairs each chunk with one ID. AI becomes 234, is becomes 56, and cool becomes 789, so text turns into a list you can compute on.

Frequent Pairs Merge

Pair e plus r merges 48 times to form er first.

You start from letters and merge the most common neighbor first. E plus r meets 48 times, so er joins before th, and common words soon need one token.

Rare Words Split

Rare unhappiness splits into un plus happi plus ness.

You meet unhappiness and you do not add a whole entry. You reuse un plus happi plus ness as 3 IDs, so rare forms cost pieces instead of vocab rows.

Short Beats Long

Seven letters shrink to 2 tokens instead of 7 IDs.

You compare two counters on playing with 7 letters. Letter IDs need 7 steps, while play plus ing needs 2, so frequent merges buy you shorter runs.

Pick Vocab Size

Pick 30k pieces and grow when splits hide meaning.

You start near 30k pieces for English and grow for code or many tongues. Flip larger when common words still split, since tiny vocabs stretch every sentence.

Where It Breaks

Split strawberry hides letter counts from the model.

You ask how many r's sit in strawberry. It sees 4231 plus 8912, not letters, so you'll watch it miscount what you count at a glance.

The Quick Version

Text must become IDs before math.
AI, is, cool map to 234, 56, 789.
E plus r merges 48 times first.
Unhappiness costs 3 reused pieces.
Playing needs 2 tokens, not 7.
Grow vocab when words still split.
IDs hide letters, so counting breaks.