LLM Engineering

In GPT-2 small the embedding table — 50,257 tokens × 768 dimensions — is 31% of all 124M parameters. In Llama 2 70B one 32,000 × 8,192 table is 0.4%. Tying the input and output tables, as GPT-2 does, saves 38.6M parameters. In small models the vocabulary is a large share of the budget; in large ones it hardly registers.

• Embedding geometry reflects co-occurrence in training text, including its biases.
• Rare tokens get few updates and poor vectors; “glitch tokens” that almost never appeared in training can produce strange behaviour.
• Comparing input embeddings across models is meaningless; each model has its own space.

def embedding_share(vocab, d, total):
    return vocab * d / total

gpt2_small = embedding_share(50_257, 768, 124_439_808)       # GPT-2 small, tied input/output
llama2_70b = embedding_share(32_000, 8_192, 68_976_648_192)   # Llama 2 70B, one of its two tables
tied_saving = 50_257 * 768                                     # parameters saved by weight tying

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

import torch
import torch.nn as nn

vocab_size, d_model = 32000, 4096
embed = nn.Embedding(vocab_size, d_model)

# Forward: token IDs → dense vectors
token_ids = torch.tensor([101, 2054, 2003])  # 3 tokens
vectors = embed(token_ids)  # shape: (3, 4096)

# Cosine similarity between tokens
from torch.nn.functional import cosine_similarity
sim = cosine_similarity(vectors[0], vectors[1], dim=0)
print(f"Similarity: {sim:.3f}")Copy