RepresentationMathematical
◆ The PatternMapping discrete tokens to continuous vector spaces
An embedding maps each discrete token to a dense vector where semantic similarity = geometric proximity.
Skip-gram: max Σ Σ log P(wₒ | wᵢ)
Predict surrounding words from centre word — forces similar words to nearby vectors
king − man + woman ≈ queen
Learned embeddings encode analogical relationships as linear vector arithmetic
// 2D embedding space — semantic clusters
Modern embeddings: Word2Vec is the ancestor. Today BERT/GPT produce contextual embeddings — the same word gets different vectors depending on context.
Pattern bridge: Mapping tokens to vectors where distance = meaning is the same idea behind LLM token embeddings.
Worked example
Five hand-made 3-d vectors. king − man + woman lands closest to queen, and then to woman and king. Here the analogy works because the vectors were built for it. In real embeddings the famous result relies partly on a quiet rule: the input words are excluded from the answers (Nissim et al. 2020).
Where it misleads
- Analogy tests overstate what embeddings know; excluding the query words does much of the work.
- Embeddings learn the associations in their training text, including social biases.
- Distances are meaningful only within one trained space; two models’ vectors cannot be compared directly.
In code
E = {'king': [0.9, 0.8, 0.1], 'queen': [0.9, 0.1, 0.8], 'man': [0.1, 0.9, 0.1],
'woman': [0.1, 0.2, 0.8], 'apple': [0.0, 0.3, 0.3]} # hand-made 3-d vectors
E = {w: np.array(v) for w, v in E.items()}
target = E['king'] - E['man'] + E['woman']
cos = lambda u, v: u @ v / (np.linalg.norm(u) * np.linalg.norm(v))
ranked = sorted(E, key=lambda w: -cos(E[w], target))
best_excluding_inputs = [w for w in ranked if w not in ('king', 'man', 'woman')][0]Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
self.embed = nn.Embedding(vocab_size=50000, embedding_dim=256) x = self.embed(token_ids) # (batch, seq) → (batch, seq, 256)
Sources
- T. Mikolov, K. Chen, G. Corrado & J. Dean, “Efficient Estimation of Word Representations in Vector Space”, arXiv:1301.3781, 2013
- M. Nissim, R. van Noord & R. van der Goot, “Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor”, Computational Linguistics 46(2), 2020
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 2