Representation✓ Mathematical
◆ The PatternMapping discrete tokens to continuous vector spaces
An embedding maps each discrete token to a dense vector where semantic similarity = geometric proximity.
Skip-gram: max Σ Σ log P(wₒ | wᵢ)
Predict surrounding words from centre word — forces similar words to nearby vectors
king − man + woman ≈ queen
Learned embeddings encode analogical relationships as linear vector arithmetic
// 2D embedding space — semantic clusters
self.embed = nn.Embedding(vocab_size=50000, embedding_dim=256) x = self.embed(token_ids) # (batch, seq) → (batch, seq, 256)
Modern embeddings: Word2Vec is the ancestor. Today BERT/GPT produce contextual embeddings — the same word gets different vectors depending on context.
Pattern bridge: Mapping tokens to vectors where distance = meaning is the same idea behind LLM token embeddings.