BPE · WordPiece · Unigram✓ Mathematical
◆ The PatternBreaking text into the atomic units a model can process
LLMs don't see characters or words — they see tokens. A tokenizer maps raw text to integer IDs from a fixed vocabulary. The dominant algorithm is Byte Pair Encoding (BPE): start with individual bytes, iteratively merge the most frequent adjacent pair until the vocabulary reaches the target size (typically 32k–128k).
BPE: Repeat → find most frequent pair (a, b) → merge into ab → until |V| = target
Each merge creates a new subword token. Rare words get split into multiple tokens.
Compression ratio = bytes / tokens ≈ 3–4× for English
A good tokenizer compresses text efficiently — fewer tokens per sentence means more context fits in the window.
WordPiece (BERT) maximizes likelihood instead of frequency. Unigram (SentencePiece) starts with a large vocabulary and prunes. Byte-level BPE (GPT-2+) operates on UTF-8 bytes, needing no pre-tokenization — handles any language/script.
Vocab size is a key tradeoff: larger vocab → fewer tokens per text but bigger embedding table. GPT-4 uses ~100k tokens; LLaMA 2 uses ~32k.
Interactive — type text to see BPE tokenization
Python — BPE from scratch#
def train_bpe(text, vocab_size):
tokens = list(text.encode('utf-8'))
merges = {}
while len(set(tokens)) < vocab_size:
pairs = {}
for i in range(len(tokens) - 1):
p = (tokens[i], tokens[i+1])
pairs[p] = pairs.get(p, 0) + 1
if not pairs: break
best = max(pairs, key=pairs.get)
new_id = max(set(tokens)) + 1
merges[best] = new_id
# Apply merge
new_tokens, i = [], 0
while i < len(tokens):
if i < len(tokens)-1 and (tokens[i], tokens[i+1]) == best:
new_tokens.append(new_id)
i += 2
else:
new_tokens.append(tokens[i])
i += 1
tokens = new_tokens
return mergesPattern bridge: Splitting text into subword tokens is BPE compression — frequent pairs merge, rare words split. In statistics, binning discretizes continuous data.