Attention — where do language models look?
When ChatGPT, Claude or DeepL try to understand a sentence, they have to decide — for every single word — which other words matter right now. That targeted listening is called attention, and you can watch it happen here. Click through heads, layers and sentences.
What is attention?
Picture a loud cocktail party: dozens of voices talking at once — but the moment someone across the room says your name, your head turns automatically. Out of all that noise, your brain picked one piece of information and gave everything else less weight.
That selective act — turning the volume up on what matters, down on what doesn't — is attention.
Reading is the same thing in miniature: the moment you see the word „it", your eye flicks back to the noun it refers to. You weight, in a flash, which earlier words are relevant and tune the rest out.
A transformer language model does this for every single word at the same time — freshly per layer and per attention head.
Example: „The animal didn't cross the street because it was too tired."
What does „it" refer to? You know instantly: the animal — not the street, not „cross". The language model has to make the same call, otherwise the translation breaks or the answer turns to nonsense. In a trained model, a single attention head learns exactly this pattern: pronoun → correct antecedent.
Without attention there would be no ChatGPT, no Claude, no DeepL translator today — they all rest on this one idea, first formulated in this form in the 2017 paper „Attention Is All You Need".
Analogy:
Picture a loud cocktail party: dozens of voices talking at once — but the moment someone across the room says your name, your head turns automatically. Out of all that noise, your brain picked one piece of information and gave everything else less weight.
That selective act — turning the volume up on what matters, down on what doesn't — is attention.
Reading is the same thing in miniature: the moment you see the word „it", your eye flicks back to the noun it refers to. You weight, in a flash, which earlier words are relevant and tune the rest out.
A transformer language model does this for every single word at the same time — freshly per layer and per attention head.
Example: „The animal didn't cross the street because it was too tired."
What does „it" refer to? You know instantly: the animal — not the street, not „cross". The language model has to make the same call, otherwise the translation breaks or the answer turns to nonsense. In a trained model, a single attention head learns exactly this pattern: pronoun → correct antecedent.
Without attention there would be no ChatGPT, no Claude, no DeepL translator today — they all rest on this one idea, first formulated in this form in the 2017 paper „Attention Is All You Need".
Definition:
Self-attention computes, for each token, a weighted sum of all value vectors. The weights come from dot products of query and key vectors, stabilised by \u221Ad_k and normalised with softmax.
Attention(Q,K,V) = softmax(Q·K\u1D40 / \u221Ad_k) · V
Step by step
Here is the picture behind it. Imagine every word in the sentence walking into a library at the same time.
Each word brings a search slip (that's the Query), reads through the index cards of all the other books (the Keys), weights which cards match its search best, and takes home a weighted mixture of the matching books' contents (the Values).
That mixture is the word's new, context-enriched meaning. Technically, this runs in five steps per layer:
- Each token is projected into three vectors: Query (Q), Key (K), Value (V).
- Q·K\u1D40 produces an n×n score matrix — how similar is every query to every key?
- Row-wise softmax turns scores into proper attention weights (each row sums to 1).
- A token's output is the weighted sum of all value vectors — information flows in from wherever it looks hardest.
- Multi-head attention does this in parallel with different projections — each head learns a different relation type (syntactic, semantic, positional).
Layer 1 usually listens locally — „what's right next to me?" Layer 2 builds the bigger-picture relations: pronoun resolution, verb-object binding, sense disambiguation.
Real models like GPT-4 or Claude don't have 2 layers, but 80 to 100 layers stacked on top of each other, each with 32 to 128 heads.
This demo is a deliberate mini-version with only 2 layers and 4 heads — so the principle stays in your head instead of drowning in a fireworks display of numbers.
Query, Key, Value — every token plays three roles
Stick with the library image from above — it carries surprisingly far.
Each word in the sentence is simultaneously searcher, index card, and book content. From a single word embedding (a list of numbers describing the word), the model computes three derived vectors — Q, K and V — through three learned projection matrices.
You can picture them as three different lenses the model puts on each word. Each lens highlights a different aspect:
- Query (Q) — the search slip — „What am I looking for right now?" Q is the search slip a word hands out to all the others.
When the pronoun „it" becomes active, its query asks, roughly: „Which word earlier in the sentence is a singular noun, an animate thing that could be tired?"
The question isn't written in words on the slip — it lives as a direction in a number vector. - Key (K) — the index card — „What do I offer?" K is the index card each word holds up clearly in the library.
The card on „animal" effectively says: „I'm a noun, singular, an animate being."
When the search slip from „it" hits this card, they line up — the dot product Q·K becomes large. The better the question and the card match, the more attention flows to that word. - Value (V) — the book content — „What do I hand over when I get picked?" V is the book content the word releases once it has been given attention.
As soon as the weights are fixed (every row sums to 100 %), all value vectors are added up according to their weights. The result is the query word's new, context-enriched meaning.
Mnemonic: K decides who is heard; V carries what is heard.
Why „self"-attention? Because every word in the same sentence acts simultaneously as Query and Key and Value for every other word (and for itself). The sentence essentially looks at itself — every word querying every other and comparing.
In the library image: every reader is also a book with its own index card. That's what sets self-attention apart from cross-attention, where Q comes from one sequence and K/V from another (classically: encoder→decoder in machine translation).
Why ÷ √d_k? A detail that looks harmless — and rescues training.
At high vector dimensions (d_k = 64 or 128 in real models) the Q·K dot products grow large by nature. Softmax with large inputs peaks sharply: nearly 100 % on a single word, all others near zero, and the gradient dies.
Dividing by √d_k keeps the input variance constant no matter how big the vectors are. A small number with a big effect — and exactly why the method is called scaled dot-product attention.
The formula in detail
Now the math — but don't worry. You don't need any linear algebra to follow along.
The formula looks compact at first because it writes what you've just met — search slip, index card, book content — in mathematical shorthand. Read it line by line, top to bottom.
Every step gets a plain-text sentence next to it explaining what happens. By the time you reach the final expression after six steps, you've effectively read a weighted average with similarity weights — that's really all it is.
Q = X · W_QK = X · W_KV = X · W_VS = (Q · Kᵀ) / √d_kA = softmax(S, axis = -1)output = A · VAttention(Q, K, V) = softmax(QKᵀ / √d_k) · VOne thing to take away: these six lines are essentially the whole mechanism. Everything a modern transformer does is, at its core, the same calculation — only with learned projection matrices, hundreds of dimensions, dozens of heads and dozens of layers stacked on top.
Q·K workshop — how vectors become attention
We deliberately name the dimensions: ref = reference, act = action, mod = modifier, gen = generic content. That helps you build an intuition for what a vector dimension could mean.
Click a pronoun, verb or determiner in the heatmap further down — the workshop will then show you, step by step: query vector → dot product with the candidates → softmax.
Real models use 64 or 128 dimensions that aren't named so human-readably. The principle is exactly the same — only richer and finer.
0.90 · 0.90 + 0.00 · 0.00 + 0.00 · 0.00 + 0.10 · 0.10 = 0.8200.90 · 0.10 + 0.00 · 0.10 + 0.00 · 0.10 + 0.10 · 0.20 = 0.1100.90 · 0.10 + 0.00 · 0.10 + 0.00 · 0.10 + 0.10 · 0.20 = 0.1100.90 · 0.20 + 0.00 · 0.20 + 0.00 · 0.20 + 0.10 · 0.30 = 0.210- Hund32%
- Leonie22%
- gab22%
- er23%
The whole attention game is, at its core, a similarity game in number space. In a real model the vector directions are learned from data and encode much finer aspects than our four demo dimensions — but if you've understood what happens here with just 4 numbers, you've understood the essence of what happens inside a 128-dimensional GPT vector.
Real transformers like GPT-4, Claude or BERT learn very similar patterns — but from billions of sentences of training data, with nobody hard-coding „it → animal". That is the real wow moment: the patterns emerge on their own from the data.
So this demo shows what comes out — so you'll recognise it in a real heatmap later — not how the training itself proceeds.
Interactive heatmap
Advanced view — 4 more patterns + layer/head selector
The crucial head for language understanding: pronouns find their antecedent. "it" → "animal", "sie" → "Aufgabe". Real disambiguation in action.
You are looking at layer L2, head H1 — pattern: Pronoun→antecedent.
- Hund0.89
- er0.06
- Leonie0.006
- gab0.006
- dem0.006
- einen0.006
Key takeaways — three things to remember
- Attention = weighted sum: (the cocktail-party principle): every word aggregates information from every other word, weighted by a learned similarity between query and key vectors. No fixed rules, no hard-coded grammar — everything emerges from comparing number vectors.
- Multi-head means division of labour: different heads learn different relation types because every head has its own Q/K/V projection matrices. One head becomes the pronoun specialist, another spots verb-object bindings, a third watches local neighbourhoods. A model like GPT-4 has 32 to 128 such specialists per layer — and 80 to 100 layers stacked on top of each other.
- Deeper layers, more abstract relations: layer 1 is dominated by positional patterns (previous token, local window) — the library does coarse sorting first. Semantic links like pronoun resolution and sense disambiguation only emerge in layer 2 (and from the middle layers onwards in real models).
- And why this matters: this one mechanism — Q·K similarity, softmax weighting, weighted sum of values — is the foundation underneath every large language model of the past years. Whoever understands attention understands the engine of ChatGPT, Claude, Gemini and DeepL in principle — and that's more than 95 % of users will ever see.
Did you get it?
- I can explain what Query, Key and Value are and how they get combined.
- In a heatmap, I can spot when a pronoun attends to its antecedent.
- I can describe why several attention heads run in parallel and what each can learn.
Attention — how tokens listen to each other
What attention computes
Self-attention lets every token pull weighted information from every other token in the sentence. It does not rely on position or hand-coded rules but on a learned similarity game between three projections:
- Query (Q): what am I looking for? — the token projected as a "question vector"
- Key (K): what do I advertise? — projected as a "keyword vector"
- Value (V): what information would I provide? — the content projection
The formula
Attention(Q, K, V) = softmax( Q · Kᵀ / √d_k ) · V
Step by step: Q · Kᵀ produces an n×n score matrix — entry (i,j) is the dot product between query i and key j, i.e. "how well do they match?". Dividing by √d_k stabilises softmax (dot products grow with dimensionality). Row-wise softmax turns raw scores into proper probabilities (each row sums to 1). Multiplying by V gives every token a weighted sum of all value vectors.
Multi-head — many attentions in parallel
A single attention mechanism can only learn one kind of relation well. Multi-head attention runs h attention computations in parallel, each with its own projection matrices W_Q, W_K, W_V. Their outputs are concatenated and linearly projected. One head may learn pronoun resolution, another verb-object bindings, a third positional patterns.
Properties
- Parallel: all tokens are processed at once (a big win over RNNs)
- Distance-independent: a token can interact with the first or the last word equally well
- Quadratic: complexity O(n²·d) in sequence length — the bottleneck for long contexts
- Interpretable: attention weights show "where the model looks" (read with care)
Try the demo! Pick a sentence with a pronoun and the pattern "Pronoun → antecedent" (layer 2, head 1). The aha moment is immediate: the pronoun looks at its antecedent, not at some random token.
1
# Self-attention — scaled dot product
2
function attention(X, W_Q, W_K, W_V):
3
# 1. Project into Q, K, V
4
Q = X · W_Q
5
K = X · W_K
6
V = X · W_V
7
8
# 2. Score matrix (n×n)
9
S = (Q · Kᵀ) / sqrt(d_k)
10
11
# 3. Row-wise softmax
12
A = softmax(S, axis = -1)
13
14
# 4. Weighted sum of values
15
output = A · V
16
return output
17
18
# Multi-head — h parallel attention computations
19
function multihead(X, [W_Q_i, W_K_i, W_V_i for i in 1..h], W_O):
20
heads = [attention(X, W_Q_i, W_K_i, W_V_i) for i in 1..h]
21
H = concat(heads)
22
return H · W_O
🎚️ Project Q, K, V
Every token (as a vector X) is multiplied by three learned matrices W_Q, W_K, W_V. The result: three views of the same token — Query ("what am I looking for?"), Key ("what do I advertise?") and Value ("what information do I provide?").
# 1. Project into Q, K, V
Q = X · W_Q
K = X · W_K
V = X · W_V
🎚️ Projection
X → Q, K, V via three learned matrices.
🗂️ Scores
Q · Kᵀ / √d_k gives an n×n matrix.
📊 Softmax
Row-wise softmax → attention weights.
↔️ Weighted sum
A · V mixes value vectors.
📚 Multi-head
h heads in parallel, then concat + project.
Test your understanding
What does self-attention compute for every token?
1. What does self-attention compute for every token?
- ☐ A) Only the next word.
- ☐ B) A weighted sum of all value vectors.
- ☐ C) A probability for each class.
- ☐ D) The gradient of the loss.
2. Why is the score matrix divided by \u221Ad_k?
- ☐ A) To stabilise the softmax inputs and avoid saturation.
- ☐ B) To save memory.
- ☐ C) So values become negative.
- ☐ D) Because tokens cannot be divided by zero.
3. What do different attention heads accomplish?
- ☐ A) They are pure copies of each other.
- ☐ B) They each store one example.
- ☐ C) They learn different kinds of relations (positional, syntactic, semantic).
- ☐ D) They replace tokenisation.
4. In which layer does pronoun resolution typically appear most clearly?
- ☐ A) In the embedding layer.
- ☐ B) In later layers (often layer 2 or deeper).
- ☐ C) Only in the decoder.
- ☐ D) Only at inference, not during training.
Related Content
Article
The Breakthrough — Transformers & Attention
What "Attention Is All You Need" (2017) actually changed — and why the world hasn't gone back.
Tokenization: The Machine Alphabet
How letters become tokens, and why LLMs see the world through this lens.
Large Language Models
The model that basically only predicts the next word — and makes a remarkable career doing it.
Embeddings & Latent Space
The mathematical space where similar words are neighbors — without anyone telling them.
Time & Sequences (RNNs)
The architecture that gave AI memory — before someone found a better way.
Sampling & Temperature
The mini-roulette after every token — and the dials you can turn.
Transfer Learning & Fine-Tuning
How to win over an AI model instead of raising it from scratch — invaluable from the start.