NLP✓ Mathematical
◆ The PatternHow text becomes numbers — the first step in every language model
Byte Pair Encoding (BPE) builds a vocabulary by iteratively merging the most frequent pair of tokens. It handles unseen words via subword splitting — no unknown tokens needed.
BPE: repeatedly merge most frequent adjacent pair
"lowest" → ["low", "est"] or ["l", "ow", "est"] depending on learned merges
Vocab size: typically 32k–128k tokens
GPT-2: 50,257 | LLaMA: 32,000 | GPT-4: ~100,000
// BPE merge process — watch vocabulary build up
1
Start with characters
Initial vocabulary = all unique bytes/characters in corpus
2
Count adjacent pairs
Find the most frequent pair of consecutive tokens
3
Merge & add to vocab
Replace all occurrences. New token added to vocabulary.
4
Repeat until vocab_size
Continue until target vocabulary size reached (e.g., 50k)
# tiktoken (OpenAI's fast BPE) import tiktoken enc = tiktoken.encoding_for_model("gpt-4") tokens = enc.encode("Hello, world!") # [9906, 11, 1917, 0] text = enc.decode(tokens) # "Hello, world!" # HuggingFace tokenizers from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")
Why it matters: Tokenization determines the model's "eyesight". Poor tokenization (e.g., splitting numbers digit-by-digit) directly hurts performance. Modern models train their own tokenizer on their specific data.
Pattern bridge: Byte Pair Encoding merges frequent pairs into tokens — the same compression principle behind LLM vocabularies. In statistics, binning into percentiles is discretization of continuous data.