Back to Learning Area

Embeddings & Latent Space

The mathematical space where similar words are neighbors — without anyone telling them.

  • Architectures
  • 10 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Words as numbers in space

What are embeddings?

For computers, words are initially numbers. But a number alone has no meaning. Embeddings change that. Embedding means: a word gets a coordinate. Similar words get similar coordinates.

The number space

An embedding is a list of numbers. These numbers describe the meaning of a word. This space is called a latent space. Latent space means: a space for meanings. Similar things lie close together there.

What are embeddings used for?

Embeddings help with many AI tasks. For example, an AI can find similar texts. A search engine uses embeddings. This is how it finds texts with the same meaning.

Your computer reads the word "apple" and sees token ID 402. That number is meaningless — until the machine learns to place it in a space where similar concepts cluster together. Its position in that space is the Embedding — and without it, no ChatGPT, no image generator, no modern AI would exist.

In this article, you will learn three things: how Word2Vec transforms words into dense vectors, what the Latent Space is and why proximity equals semantic similarity, and how RAG pipelines build on Embeddings to retrieve knowledge from documents.

Words as Vectors: The Word2Vec Breakthrough

Before Word2Vec, developers used One-Hot Encoding: each word was represented as a huge vector — 50,000 dimensions, with exactly one 1 and 49,999 zeros. The fatal problem: the distance between "apple" and "banana" is identical to the distance between "apple" and "car engine." One-Hot has no concept of similarity.

One-Hot Encoding

50,000 dimensions. Exactly one 1, rest zeros. No similarity measurable. "Apple" and "banana" equally distant as "apple" and "engine."

Dense Embedding

100-300 dimensions. All values active. Similarity directly measurable. "Apple" and "banana" are close together.

In 2013, Tomas Mikolov and his team at Google introduced Word2Vec. Linguist J.R. Firth (1957) summed up the core idea: "A word is known by the company it keeps." Words that appear in similar contexts receive similar vectors. Word2Vec trained on billions of words and compressed meaning into dense vectors with typically 100 to 300 dimensions.

Definition

Word Embedding — When Words Get Coordinates

Analogy

Imagine a new student arriving at a foreign school without speaking the language. She figures out who "belongs together" by watching who sits with whom at lunch. Kids at the same table probably share interests. Word2Vec does exactly this — it watches billions of sentences and places words that appear in the same neighborhoods close together.

Where the analogy breaks: the student also sees faces, hears tones, and reads body language. Word2Vec only sees text — it has zero sensory grounding. It knows that "apple" and "banana" co-occur with "fruit" but has no idea what either tastes like.

The most famous example: Vector("King") - Vector("Man") + Vector("Woman") ≈ Vector("Queen"). The direction from "Man" to "Woman" in the vector space roughly parallels the shift from "King" to "Queen." Similarly: Vector("Paris") - Vector("France") + Vector("Germany") ≈ Vector("Berlin"). These equations are fascinating — but also idealized. In practice, Embedding spaces are messier than textbook examples suggest.

Caution

Common Misconception: "Embeddings Understand Meaning"

No. Embeddings capture statistical co-occurrence patterns, not meaning.

"Apple" and "banana" are close because they appear in similar sentences — not because the model knows what fruit is. In Google's well-known Word2Vec model, "Doctor" - "he" + "she" ≈ "Nurse" — reflecting societal biases in training data, not medical reality. The example isn't quite that clean, though: the usual analogy method excludes "Doctor" itself as an answer, otherwise "Doctor" would often come out again. The bias in embeddings is still well documented.

Every modern language model starts by converting its input into Embeddings. In a Transformer, the very first layer is the Embedding layer — it converts token IDs into dense vectors before any attention mechanism can operate.

Interactive: Bag-of-Words and TF-IDF

Before words become dense embeddings, a text first has to turn into numbers. The classic route: count how often each word appears (Bag-of-Words), then weight how telling each word is (TF-IDF). Try it on three tiny documents.

The Latent Space: Where Meaning Lives

Definition

Latent Space — The Hidden Space of Representations

Analogy

Imagine an invisible city map where similar concepts "live" in the same neighborhood. There is a "dog quarter" next to the "cat quarter," a "Europe quarter" next to the "Asia quarter." No human drew this map — the model cartographed it entirely on its own.

"Latent" means hidden: the dimensions of the space don't correspond to human-readable properties. Dimension 47 might partially encode "formality" and partially "topic domain" — you can't label it cleanly. That's why researchers use mathematical tools for dimensionality reduction to visualize high-dimensional spaces in 2D.

Word2Vec

300

Dimensions per word vector

GPT-3

12,288

Dimensions in the Latent Space

Stable Diffusion 1.5

4 x64x64

Latent image representation

Where the analogy breaks: a city is two-dimensional. The Latent Space has thousands of dimensions. In 2D, a point can have only a few direct neighbors. In 12,288 dimensions, "dog" can simultaneously be close to "cat" (as a pet), "wolf" (biologically), and "loyalty" (in character). This multi-dimensional proximity is impossible on a flat map.

A concrete example: a text analysis tool embeds movie reviews. "The film was brilliant, outstanding acting" and "Fantastic movie, superb cast" land nearly at the same point — despite sharing no significant words. "The film was terribly boring" lands much farther away.

Caution

Common Misconception: "Each Dimension Has a Readable Meaning"

No. Unlike hand-engineered features ("dimension 1 = size, dimension 2 = weight"), Latent Space dimensions are abstract and entangled.

That's why researchers use dimensionality reduction (t-SNE, PCA) for visualization — but these projections always lose information and can be misleading.

RAG: The Killer App of Embeddings

Retrieval-Augmented Generation connects a pre-trained LLM with external knowledge. The technical foundation: Embeddings. Without translating text into vectors, RAG as we know it today would not exist.

  1. 1. Chunking
  2. 2. Embedding
  3. 3. Store
  4. 4. Query Embed
  5. 5. Search
  6. 6. Generate

The process: documents are split into chunks of typically 200-1000 tokens. Each chunk is converted into a vector by an Embedding model and stored in a vector database. When a user asks a question, the query is also embedded, the most similar chunks are found via similarity search, and passed to the LLM as context.

Keyword Search

Only finds documents containing the exact search words. "Reset password" does not find "renew access credentials."

Semantic Search

Finds documents by meaning. "Reset password" also finds "renew access credentials" and "account recovery."

A company manages 10,000 technical PDF manuals. An employee asks: "How do I reset my password?" Keyword search only finds documents containing "password." Embedding-based search also finds chunks about "renewing access credentials," "credential reset procedure," and "account recovery" — because these phrases sit close together in the Latent Space.

Caution

Common Misconception: "RAG Replaces Fine-Tuning"

RAG and fine-tuning solve different problems. Think of it this way: RAG gives the AI a new book to read, fine-tuning changes the AI's brain.

RAG provides access to external, updatable knowledge — ideal for facts that change. Fine-tuning changes the model's behavior and style — ideal for domain-specific tone. Many production systems use both.

Word2Vec and GloVe produce static Embeddings: a word always gets the same vector regardless of context. The word "bank" has the same vector whether it refers to a financial institution or a riverbank.

ELMo and BERT (both 2018) introduced contextual Embeddings: the same word gets different vectors depending on its sentence context. "The bank by the river" and "The bank transfers money" produce different vectors for "bank" — the model resolves ambiguities.

This evolution was crucial: contextual Embeddings enabled the linguistic depth that characterizes modern LLMs like GPT and Claude.

Cosine Similarity measures the angle between two vectors: cos(θ) = (A·B) / (||A|| x ||B||). The value ranges from -1 (opposite) to +1 (identical direction). It is the usual measure for embeddings because it counts only direction, not length; if the vectors are normalized to length 1, as with many embedding models, Euclidean distance gives the same ranking.

Practical interpretation (rough guide values that vary by model): 0.92 = very similar (e.g., synonyms). 0.5 = weakly related. 0.1 = barely connected. These scores determine which chunks in a RAG pipeline count as "relevant."

Interactive: Words in Embedding Space

Click on the words and observe their position in 2D space. Semantically similar words are close together. Notice the parallel axes: King relates to Queen as Man relates to Woman — exactly the vector arithmetic that made Word2Vec famous.

Click on a concept to learn more. The lines show how the concepts are connected.
GenderGenderRoyaltyRoyaltysimilarKingQueenManWomanDogCatCar

Click on a circle to see details.

Legend

King
Queen
Man
Woman
Dog
Cat
Car

Caution

Warnings and Pitfalls

  • AI does not understand real meaning. It only knows statistical patterns. "Apple" and "banana" are close because they appear in similar sentences — not because the model knows what fruit is.
  • The equation King - Man + Woman ≈ Queen is an idealized textbook example. In practice, Embedding spaces are chaotic. Such clean analogies are the exception.
  • Embeddings are not universal: a model trained on English often works poorly on German texts. A pure text model cannot encode images; for that there are multimodal embedding models that map text, images and in some cases audio and video into a shared space.
  • "Semantic search" does not mean the AI understands meaning. With irony, sarcasm, or subtle nuances, pure distance measurement fails.

Note

Key Takeaways

  1. Embeddings translate concepts into dense numerical vectors. Words with similar contexts get similar vectors — "apple" and "banana" cluster together, "apple" and "carburetor" don't.
  2. The Latent Space is the high-dimensional coordinate system where all representations live. Its hundreds to thousands of dimensions allow a single concept to be simultaneously close to many different related concepts.
  3. RAG makes Embeddings practical: chunk documents, embed them, store in a vector DB, search semantically, feed relevant chunks to an LLM. No Embeddings, no modern RAG.

Now you understand how AI models represent meaning. The next article introduces Computer Vision and CNNs — how machines learn to understand images.

Check

Quiz: Embeddings & Latent Space

Not completed

What problem does Word2Vec solve that One-Hot Encoding cannot?

Select one answer

Answer Key: 1) B · 2) C · 3) C · 4) B

Check

Did you understand everything?

Can you explain this in your own words? Tick what you've got.

0 of 3 done