Word Embeddings
How machines learn meaning in a vector space
What are word embeddings?
Picture a huge city where every word is a house. Words with similar meaning live in the same neighbourhood: "dog" and "cat" as next-door neighbours, "Berlin" and "Paris" two streets over in the capitals district.
Here is the magic: directions in this city carry meaning. The walk from "man" to "woman" goes the exact same distance and direction as "king" to "queen". That is exactly what this demo measures: king − man + woman ≈ queen.
Analogy:
Picture a huge city where every word is a house. Words with similar meaning live in the same neighbourhood: "dog" and "cat" as next-door neighbours, "Berlin" and "Paris" two streets over in the capitals district.
Here is the magic: directions in this city carry meaning. The walk from "man" to "woman" goes the exact same distance and direction as "king" to "queen". That is exactly what this demo measures: king − man + woman ≈ queen.
Definition:
Word embeddings are dense vector representations of words, learned from context co-occurrence in large text corpora (Word2Vec, GloVe, FastText). Geometric distance in the vector space correlates with semantic similarity.
Similarity is typically measured by cosine: cos(a, b) = (a · b) / (‖a‖ · ‖b‖). Values close to 1 mean semantically related, close to 0 unrelated, close to −1 opposite.
How do word embeddings work?
A model reads billions of sentences. From each sentence it learns: which words tend to appear in the same context? From those signals it builds one vector per word — usually 50 to 300 dimensions — so that contextually similar words get similar vectors.
- 1Skip-gram / CBOWThe model predicts each word's neighbours (or vice versa) and updates the vectors so the prediction gets better.
- 2Semantic axesDuring training, latent directions emerge — "gender", "capital-of-country", "plural" — without anyone programming them.
- 32D projectionFor display, the 100-dimensional space is flattened to 2D with PCA. Clusters stay visible, but distances on the map are only roughly right — that's why neighbours and analogies are calculated on the full vectors.
- 4Vector arithmeticBecause semantic axes exist as directions,
a − b + cworks as an analogy operation. The classic example isking − man + woman ≈ queen.
Interactive demo — explore the meaning space
This map is the meaning space: every dot is a word, the coloured areas are topic clusters, and spatial closeness means similar meaning. Drag to pan, scroll or pinch to zoom. Pick a task below.
The map in words
This map turns meaning into place: words become locations, and the paths between them become directions. If you cannot see the canvas, here is the shape the numbers below leave out.
- What you see
- Around 190 small dots scattered across the field like a night sky. Dots of the same colour huddle into islands: the animals sit in one corner, the countries in another, the emotions somewhere else again. The closer two dots lie, the more alike their meaning.
- What happens
- Click a word and short lines reach out to its nearest neighbours in the same neighbourhood. In the arithmetic task an arrow appears, and the result drops in as a new dot right where the expected word already sits. Analogies show up as parallel arrows: the step from king to queen points the same way as the step from man to woman.
- What you can do
- Three tasks are ready: Neighbourhood shows a word's closest neighbours, Arithmetic lets you try a − b + c, and Challenge quizzes you on analogies. You can click individual dots, drag the map, and zoom.
- What to watch for
- Watch the parallel arrows. That king-to-queen takes the same path as man-to-woman is no accident — the model has learned a rule like gender as a fixed direction in space. That is exactly why meaning here becomes nearness and direction.
Every word is a point in space. Click one — the highlighted neighbours are the most similar in meaning, measured by cosine similarity.
Click a point in the space or search a word to see its nearest neighbours.
Where word embeddings live
Embeddings are not a gimmick — they are the foundation of almost every modern language AI:
Search engines find documents by meaning, not just exact words — a search for "car" also finds "vehicle".
Similar products, songs or articles are found via the closeness of their vectors — "you might also like".
Multilingual embeddings align different language spaces with each other — the basis of modern translators.
Every token in ChatGPT and friends starts as an embedding — it is the very first layer of every transformer.
Key takeaways
- Words become vectors: every word gets a position in space — learned from the context it appears in.
- Closeness means meaning: semantically similar words sit close together; this is measured with cosine similarity.
- Directions carry meaning: the same step as man→woman leads from king→queen — which is why king − man + woman ≈ queen.
- Learned, not programmed: the semantic axes emerge on their own during training on billions of sentences.
- Limit of static embeddings: one word has exactly one vector; polysemy is only solved by contextual models like BERT.
Did you get it?
- I can explain why words with similar meaning sit close together in the space.
- I understand what cosine similarity between two word vectors measures.
- I can follow why king − man + woman ≈ queen works.
- I know the main limit of classic (static) embeddings: one word, one vector.
Test your knowledge
Why does "king − man + woman ≈ queen" actually work?
1. Why does "king − man + woman ≈ queen" actually work?
- ☐ A) Because during training the model learns latent directions such as a "gender axis", without anyone explicitly programming them.
- ☐ B) Because embeddings are a database with hand-coded analogies.
- ☐ C) Because words are arranged alphabetically in a vector.
- ☐ D) Because the model performs a web search at query time.
2. What does cosine similarity between two word vectors measure?
- ☐ A) The alphabetical distance between the words.
- ☐ B) The angle between the vectors — close to 1 means very similar, close to 0 means unrelated.
- ☐ C) The number of letters they share.
- ☐ D) How often both words appear on Wikipedia.
3. Where does a word-embedding model "learn" the vectors from?
- ☐ A) From a dictionary with definitions.
- ☐ B) From explicit examples that humans label.
- ☐ C) From context co-occurrence: which words tend to appear together in a corpus.
- ☐ D) From the ASCII encoding of the words.
4. What is the main limitation of classic (static) word embeddings like Word2Vec?
- ☐ A) They only work for English words.
- ☐ B) One word gets ONE vector — polysemy (bank as river-bank vs. money-bank) gets blurred. Modern contextual models like BERT produce a fresh vector per occurrence.
- ☐ C) They are too slow for real applications.
- ☐ D) They cannot process special characters.
Word Embeddings Explained
Words as points in space
A computer cannot do anything with the text king — but it can with numbers. A word embedding assigns each word a vector of many numbers (in practice 50 to 300 dimensions). The trick: words with similar meaning get similar vectors and thus lie close together. That turns meaning into a geometry that can be measured and computed with.
The basic idea: you recognize a word by its company
How is the model supposed to know that cat and dog are related? From context. Both often appear next to words like feed, fur or vet. This distributional hypothesis (Firth, 1957: You shall know a word by the company it keeps) is the core idea: words that appear in similar surroundings usually mean similar things. Nobody programs the meanings by hand — they arise purely from raw text.
Skip-gram: predicting neighbors
The word2vec method (Mikolov et al., 2013) slides a context window over the text. In the skip-gram variant, the model tries to predict a target word's neighbors from it; the reverse is called CBOW (predicting the target word from the neighbors). With every hit, the vectors involved are nudged a bit closer together, and for randomly drawn non-neighbors, further apart. This tugging across millions of sentences gradually shapes the vector space. The so-called negative sampling is an efficient approximation of the full softmax, not an exact probability calculation.
Computing with meaning
Because training produces whole directions for concepts like gender or capital-to-country, you can compute with vectors. The famous discovery from 2013: king − man + woman ≈ queen. You shift the vector for king along the gender direction and land near queen. Important: the result is the nearest vector, not an exact equation — it works for many, but not all, analogies, and the input words themselves are excluded from the search.
Properties and limits
- Self-supervised: learns only from raw text, without any labels
- Similarity = cosine: what is measured is the angle between vectors, not their length
- Static: classical embeddings give every word one vector — bank (financial institution) and bank (river bank) collapse into one
- Biases: what is in the text is also in the model — societal biases in the corpus end up measurably in the vectors
Play with the demo! Click a word and see its nearest neighbors along with the cosine value. Then switch to compute mode and try king − man + woman — or Berlin − Germany + France. Watch how well the analogy lands and where it misses.
1957: The distributional hypothesis
Long before any neural network, linguists like John R. Firth and Zellig Harris formulated the idea that a word's meaning results from its typical neighbors. Firth's sentence You shall know a word by the company it keeps remains the foundation of every embedding to this day.
2003: Neural language models
Yoshua Bengio and colleagues showed that a neural network can simultaneously learn a language model and a distributed vector representation for every word. The idea was powerful, but computationally expensive.
2013: word2vec makes it fast
Tomáš Mikolov and his team at Google published word2vec with the variants skip-gram and CBOW. Negative sampling made training so efficient that embeddings could be computed from billions of words. The analogy king − man + woman ≈ queen made the method famous.
2014: GloVe from Stanford
Jeffrey Pennington, Richard Socher and Christopher Manning introduced GloVe: instead of sliding a window, it learns from a global statistic of how often word pairs co-occur. This demo uses real GloVe vectors (100 dimensions, public domain) — the learning path is different, but the resulting geometry is the same as with skip-gram: similar words, similar vectors.
From 2018: Contextual embeddings
Models like ELMo and BERT overcame the biggest weakness: they give the same word a different vector depending on the sentence. This finally makes it possible to distinguish bank (money) from bank (river). Still, the static vectors of word2vec and GloVe remained the building block on which everything else was built.
1
# Learning word embeddings with skip-gram (word2vec, Mikolov 2013)
2
3
# 1. Initialize vocabulary and vectors
4
build vocabulary V from the corpus
5
for each word w in V:
6
vector[w] = random values, D dimensions # D ~ 50..300
7
8
# 2. Slide a context window over the whole text
9
repeat per epoch, for each position in text:
10
target = text[position]
11
context = words in the window (±c) around target
12
13
for each context word ct in context:
14
# 3. Does the real pair (target, ct) fit together?
15
score = dot_product(vector[target], vector[ct])
16
p = sigmoid(score) # close to 1 = fits, close to 0 = doesn't
17
18
# 4. Draw negative examples (approximation instead of full softmax)
19
negatives = draw k random words, frequency-weighted
20
21
# 5. Small step: real pair closer, random pairs further apart
22
move vector[target] and vector[ct] closer together
23
for each neg in negatives:
24
push vector[target] and vector[neg] further apart
25
26
# result: similar contexts -> similar vectors
27
28
# ── using the finished vectors (this is what the demo shows) ──
29
30
# 6. Similarity: nearest neighbors via cosine
31
function similarity(a, b):
32
return cosine(vector[a], vector[b])
33
# cosine = dot_product / (magnitude[a] * magnitude[b])
34
35
# 7. Analogy: king − man + woman ≈ queen
36
function analogy(a, b, c):
37
target = vector[a] − vector[b] + vector[c]
38
return word with largest cosine(vector[word], target)
39
# a, b and c are excluded from the search
40
41
# 8. Visualization: project the high-dim space to 2D via PCA
42
map = pca(all vectors, components = 2)
🎲 Rolling the vectors
A vocabulary is built from the corpus. Every word first gets a vector of purely random numbers in D dimensions (usually 50 to 300). Nothing means anything yet — training is what gives the numbers their meaning.
build vocabulary V from the corpus
for each word w in V:
vector[w] = random values, D dimensions # D ~ 50..300
📚 Corpus
Huge amounts of raw text — the only source of knowledge. No labels, no dictionaries.
🪟 Context window
A window reads target word plus neighbors and delivers real word pairs.
🔧 Adjusting vectors
Real pairs closer together, random pairs further apart — many small learning steps.
🌌 Vector space
After training, similar words lie close together; concepts become directions.
🧮 Using it
Nearest neighbors via cosine and analogies via vector arithmetic — that's what the demo shows.
Related Content
Article
Embeddings & Latent Space
The mathematical space where similar words are neighbors — without anyone telling them.
Large Language Models
The model that basically only predicts the next word — and makes a remarkable career doing it.
Programming vs. Training
How programming changed when people stopped writing every rule themselves.
Time & Sequences (RNNs)
The architecture that gave AI memory — before someone found a better way.
Tokenization: The Machine Alphabet
How letters become tokens, and why LLMs see the world through this lens.
The Breakthrough — Transformers & Attention
What "Attention Is All You Need" (2017) actually changed — and why the world hasn't gone back.
Unsupervised Learning
Learning without an answer key — the more demanding but often more practical variant of machine learning.
Spaces and Directions (Vectors)
Why AI constantly works with arrows in n-dimensional space.
Demo
Attention — where do language models look?
Interactive heatmap of self-attention: precomputed patterns show how a 2-layer transformer wires words together — pronoun resolution, verb-object, disambiguation.
Neural Network Playground
Click layers and neurons together, choose a dataset and activation function, and watch the network learn to separate the data in real time.