Byte-Pair Encoding
A widely used subword tokenization algorithm that iteratively merges the most frequently occurring pairs of characters or bytes into a single unique token.
Think of It Like This
Like replacing the frequent letter pair 't' and 'h' with a single new symbol 'th' to shorten the overall length of a written message.
BPE perfectly balances the massive vocabulary size of word-level tokenization with the extremely long sequences of character-level tokenization. It gracefully handles rare words and misspellings by breaking them down into known subword chunks. Almost all modern large language models, including the GPT family, rely on BPE to process raw text into numerical inputs.