Gen AI

Tokenization Artifacts

Because tokenizers blindly merge frequent text strings, they create quirks like separate tokens for capitalized words and leading spaces, forcing models to learn redundant concepts.

Trailing spaces and mismatched casing create separate token IDs, forcing the model to learn multiple embeddings for identical semantic meaning

Why Does This Exist?

When a tokenizer learns its vocabulary purely by counting frequencies, it doesn't know what the text actually means. It just sees strings of bytes. As a result, the vocabulary it builds is full of redundant, weird, and sometimes completely broken entries. These are called tokenization artifacts.

Understanding these artifacts is crucial because they directly impact how an LLM behaves. When a model seems to mysteriously struggle with a specific word, or fails at a simple rhyming task, or suddenly outputs bizarre text, the root cause is almost always a quirk in how the text was chopped up before the model ever saw it.

Think of It Like This

A filing system that treats 'Apple' and ' apple' as entirely unrelated

Imagine a physical filing cabinet where folders are organized strictly by exact visual matches. You have a folder for "Apple", a separate folder for "apple", and yet another folder for " apple " (with a leading space). To a human, these are all the same concept. But to the filing system, they are as different as "Apple" and "Zebra".

If you want the filing system to know that "Apple" is a fruit, you have to write that information down in all three folders independently. If you forget to update the " apple " folder, the system will look foolish when someone queries it with a leading space. This is exactly what tokenization artifacts do to an LLM's embedding space.

How It Actually Works

The leading space problem

In most modern tokenizers, the space character is treated just like any other letter. Because words in English are almost always preceded by a space, the tokenizer usually merges the space into the beginning of the word. Therefore, token and token are two completely different entries in the vocabulary.

This forces the LLM to learn two entirely separate embeddings for the same semantic concept. During training, the model has to independently deduce that token ID 4192 ( Apple) and token ID 703 (Apple) behave similarly. If one variant appears much less frequently in the training data, the model might understand the concept well when it starts a sentence, but struggle with it mid-sentence.

Case sensitivity and capitalization

Just like spaces, capitalization creates separate tokens. apple, Apple, and APPLE will all receive unique token IDs. This redundancy bloats the vocabulary and dilutes the model's learning efficiency. A model might be an expert on the token for python, but if someone prompts it with P Y T H O N, the tokenizer shatters that text into individual characters, and the model suddenly has no idea what the topic is, because the embeddings for those individual letters don't carry the semantic weight of the single python token.

Glitch tokens

Because tokenizers are trained on massive, unfiltered internet scrapes, their vocabularies sometimes absorb bizarre strings of text that appeared thousands of times due to automated bots or corrupted data—things like SolidGoldMagikarp or random Reddit usernames. These are known as "glitch tokens."

Because these tokens appeared frequently in the tokenizer's training data, they got their own token ID. But because they were often filtered out of the LLM's training data, the model never learned a proper embedding for them. If a user forces the model to process or generate one of these glitch tokens, the model's internal math goes haywire, often resulting in surreal, evasive, or repetitive outputs.

Show Me the Code

You can easily observe these artifacts by passing variations of the same word through a real tokenizer (like the one used by GPT-4).

1import tiktoken2
3# Load the tokenizer used by GPT-44enc = tiktoken.get_encoding("cl100k_base")5
6variations = [7    "apple",8    "Apple",9    " APPLE",10    " apple"11]12
13for var in variations:14    tokens = enc.encode(var)15    print(f"'{var}' -> {tokens}")16    17# -> 'apple' -> [82531]18# -> 'Apple' -> [16108]19# -> ' APPLE' -> [32050]20# -> ' apple' -> [21560]

Notice that four semantically identical inputs yield four entirely different token IDs. The model has to learn the meaning of all four independently.

Watch Out For

Assuming spelling or rhyming tasks are easy

LLMs are notoriously bad at tasks like "how many 'r's are in strawberry" or "write a poem that rhymes with orange." This is a direct result of tokenization. The model does not see the letters s-t-r-a-w-b-e-r-r-y. It sees a single integer ID representing the whole chunk. Asking an LLM to count letters is like asking a human to describe the chemical makeup of a cake by just looking at a photo of it.

Prompt injection via token fragmentation

Adversaries often bypass safety filters by exploiting tokenization. If a safety system checks for the token ignore, an attacker might write i g n o r e. The tokenizer breaks this into single characters, completely bypassing the safety filter that was looking for the whole-word token. The LLM, however, is smart enough to piece the characters together and follow the malicious instruction.

The Quick Version

Tokenizers group text purely by frequency, creating redundant tokens for words with trailing spaces or varying capitalization.
This forces the LLM to learn multiple, separate embeddings for the exact same semantic concept.
"Glitch tokens" are bizarre, frequent strings that made it into the tokenizer's vocabulary but lack meaningful embeddings, causing erratic model behavior.
Tokenization obscures individual characters, making spelling, counting, and rhyming tasks unnaturally difficult for LLMs.