LLM Engineering

Byte-level BPE starts from UTF-8 bytes. English letters are 1 byte each, Danish text averages 1.19 (æ, ø and å take two), Greek 2, and Chinese and Hindi 3. Where the tokenizer saw little of a script, merges are few and text stays close to a token per byte — so the same sentence can cost several times more tokens, context and money in one language than in another.

• Token counts, prices and context limits are per token, not per word; compare languages on tokens actually produced (Petrov et al. 2023 measured up to 15× differences).
• Tokenizers are frozen with the model; text that is rare in the tokenizer’s training data (code in a new language, a new script) stays expensive.
• Character-level questions (“how many r’s in strawberry?”) are hard because the model never sees characters.

samples = {'English': 'pattern', 'Danish': 'mønstre på tværs', 'Greek': 'μοτίβο',
           'Chinese': '模式识别', 'Hindi': 'पैटर्न'}
bytes_per_char = {lang: len(s.encode('utf-8')) / len(s) for lang, s in samples.items()}
# byte-level BPE starts from UTF-8 bytes: scripts it saw little of stay closer to one token per byte

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

def train_bpe(text, vocab_size):
    tokens = list(text.encode('utf-8'))
    merges = {}
    while len(set(tokens)) < vocab_size:
        pairs = {}
        for i in range(len(tokens) - 1):
            p = (tokens[i], tokens[i+1])
            pairs[p] = pairs.get(p, 0) + 1
        if not pairs: break
        best = max(pairs, key=pairs.get)
        new_id = max(set(tokens)) + 1
        merges[best] = new_id
        # Apply merge
        new_tokens, i = [], 0
        while i < len(tokens):
            if i < len(tokens)-1 and (tokens[i], tokens[i+1]) == best:
                new_tokens.append(new_id)
                i += 2
            else:
                new_tokens.append(tokens[i])
                i += 1
        tokens = new_tokens
    return mergesCopy