Learned Representations✓ Mathematical
◆ The PatternMapping discrete token IDs to continuous vector space
Each token ID indexes into a learned embedding table of shape (vocab_size, d_model). The result is a dense vector that captures semantic meaning. Similar tokens end up near each other in this space — "king" and "queen" are close, "cat" and "automobile" are far.
E = Embedding(token_id) ∈ ℝ^d_model
Simple lookup: row token_id from the embedding matrix. No computation, just indexing.
Scaled: E' = E · √d_model
Some architectures scale embeddings so their magnitude matches positional encodings.
Typical dimensions: GPT-2 uses d=768 (small) to d=1600 (XL). LLaMA-70B uses d=8192. The embedding table is often tied with the output projection (weight tying), reducing parameter count.
For GPT-4's ~100k vocab with d_model=12288, the embedding table alone is ~1.2B parameters — a significant fraction of total model size for smaller models.
Interactive — 2D embedding projection
Python — embedding layer#
import torch
import torch.nn as nn
vocab_size, d_model = 32000, 4096
embed = nn.Embedding(vocab_size, d_model)
# Forward: token IDs → dense vectors
token_ids = torch.tensor([101, 2054, 2003]) # 3 tokens
vectors = embed(token_ids) # shape: (3, 4096)
# Cosine similarity between tokens
from torch.nn.functional import cosine_similarity
sim = cosine_similarity(vectors[0], vectors[1], dim=0)
print(f"Similarity: {sim:.3f}")Pattern bridge: Mapping tokens to dense vectors where distance = meaning. Cosine similarity measures the result.