ML Math

Four words with counts: low ×5, lower ×2, newest ×6, widest ×3. BPE merges the most frequent adjacent pair each round: e+s, then es+t, then est+</w>, then l+o. After four merges “newest” is n e w est</w> — the common ending has become one token, learned from counts alone.

• Tokens are frequency artefacts, not morphemes; the same word can split differently with a leading space or capital.
• Languages and scripts under-represented in the training corpus get more tokens per word, costing more and fitting worse.
• Numbers split into irregular chunks, one reason arithmetic is hard for language models.

from collections import Counter
words = {'l o w </w>': 5, 'l o w e r </w>': 2, 'n e w e s t </w>': 6, 'w i d e s t </w>': 3}   # word -> count
merges = []
for _ in range(4):
    pairs = Counter()
    for w, n in words.items():
        s = w.split()
        for a, b in zip(s, s[1:]): pairs[a, b] += n
    best = max(pairs, key=pairs.get)
    merges.append(''.join(best))
    words = {w.replace(' '.join(best), ''.join(best)): n for w, n in words.items()}

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

# tiktoken (OpenAI's fast BPE)
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("Hello, world!")          # [9906, 11, 1917, 0]
text = enc.decode(tokens)                       # "Hello, world!"

# HuggingFace tokenizers
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")