What "Attention Is All You Need" (2017) actually changed — and why the world hasn't gone back.
Architectures 12 min Expert June 15, 2026
RNNs process language one word at a time. For a 500-word paragraph, word 1 must survive 499 steps of compression before influencing the output — often, it does not survive. In 2017, Vaswani et al. proposed a radical alternative: what if every word could directly attend to every other word, all at once?
2014 Papers
Attention Mechanism: The Key to Modern LLMs
September 2014: Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio published a paper that would lastingly change the NLP world. 'Neural Machine Translation by Jointly Learning to Align and Translate' solved a fundamental problem in sequence-to-sequence models. Previous encoder-decoder architectures compressed every input sentence into a single fixed-length vector — an information bottleneck for long sentences. Bahdanau attention was a significant step forward: instead of a fixed vector, the model used dynamic attention over different parts of the input sentence. Like the human eye jumping while reading, AI attention moves between relevant words. This 'additive attention' became the conceptual precursor to modern NLP systems. The later Transformer (2017) built on the attention idea, but replaced the additive variant with the more efficient Scaled Dot-Product Attention. Without Bahdanau's attention concept, no Transformers; without Transformers, no GPT family or BERT. This breakthrough happened three years before 'Attention Is All You Need.'
That idea — Self-Attention — eliminated the sequential bottleneck and unleashed a wave of scaling that produced GPT, BERT, and virtually every major AI system in use today.
Self-Attention — Every Token Sees Every Other Token
The central innovation of Transformers: parallel processing, direct connections, dynamic weights. Why this solves the RNN problems:
Self-Attention
AnalogyDefinition
Imagine a classroom discussion. In the RNN approach, students sit in a long line and can only whisper to their immediate neighbor — a message from student 1 must pass through students 2, 3, 4, ... before reaching student 30. Information gets lost. In the Self-Attention approach, every student can see and hear every other student directly. Each student decides independently how much attention to pay to each speaker based on what is currently being discussed.
Example
In a real classroom, students have limited cognitive bandwidth and cannot truly attend to 30 speakers simultaneously. Self-Attention computes attention to ALL tokens mechanically and in parallel — there is no cognitive limit, only computational cost.
Analogy:
Imagine a classroom discussion. In the RNN approach, students sit in a long line and can only whisper to their immediate neighbor — a message from student 1 must pass through students 2, 3, 4, ... before reaching student 30. Information gets lost. In the Self-Attention approach, every student can see and hear every other student directly. Each student decides independently how much attention to pay to each speaker based on what is currently being discussed.
Example
In a real classroom, students have limited cognitive bandwidth and cannot truly attend to 30 speakers simultaneously. Self-Attention computes attention to ALL tokens mechanically and in parallel — there is no cognitive limit, only computational cost.
Definition:
Self-Attention computes, for every token in a sequence, how relevant every other token is to it — all at once, in parallel. Unlike RNNs that compress everything into a single hidden state, every token has a direct connection to every other token. The attention weights are dynamic: they are computed fresh for each input sequence.
RNN (sequential)
Tokens are processed one by one. Information from token 1 must pass through 499 hidden states to reach token 500. The vanishing gradient problem causes long-range dependencies to disappear.
Transformer (parallel)
Every token has direct access to every other — the distance is always exactly one step. No information loss over distance. Massive GPU parallelization possible.
The word "bank" in two sentences: "The bank is by the river" vs. "The bank is bankrupt." Without attention, "bank" has the same representation in both. With Self-Attention: in sentence 1, "bank" attends strongly to "river" — its representation shifts toward "bench/seat." In sentence 2, "bank" attends strongly to "bankrupt" — representation shifts toward "financial institution." Same word, different meaning — computed automatically.
The attention weights are dynamic: they are computed fresh for each input — unlike static CNN filters or fixed dense layer weights.
Watch Out: Transformers Don't "Understand" Language
Transformers compute statistical patterns, not meaning. The attention weights show correlations, not real comprehension. The impressive results come from pattern recognition on massive datasets — not from cognition.
Try It Out: Self-Attention Step by Step
See how Self-Attention computes attention weights for the token "Fuchs" in the example sentence "Der schlaue Fuchs":
Step 1 / 6Tokens ready
Three tokens are in the sequence. Self-Attention will now compute how much attention each token pays to each other — all at once, in parallel.
Query, Key, Value — The Mechanics of Attention
How attention is actually computed — step by step:
Query, Key, Value
AnalogyDefinition
Think of a library search. You walk in with a specific research question (your Query). Every book has an index card describing its content (its Key). You compare your question against every index card — some match well (high dot product), others don't. For the matching books, you read the relevant passages (their Values). The final answer is a blend of information from all relevant books, weighted by how well each card matched your question.
Example
In a real library, you typically pick one or two best-matching books. Self-Attention computes a weighted combination of ALL values — every token contributes something, though most contributions are near-zero after softmax.
Analogy:
Think of a library search. You walk in with a specific research question (your Query). Every book has an index card describing its content (its Key). You compare your question against every index card — some match well (high dot product), others don't. For the matching books, you read the relevant passages (their Values). The final answer is a blend of information from all relevant books, weighted by how well each card matched your question.
Example
In a real library, you typically pick one or two best-matching books. Self-Attention computes a weighted combination of ALL values — every token contributes something, though most contributions are near-zero after softmax.
Definition:
Each token is projected into three vectors: Query (what information does this token seek?), Key (what information does it offer?), and Value (what actual content does it carry?). Attention scores are computed as scaled dot products: score = (Q · K^T) / sqrt(d_k). Softmax converts scores to probabilities. The output is a probability-weighted sum of all Value vectors.
Attention Formula
Attention(Q, K, V) = softmax(Q · KT / √dk) · V
Mini-attention for "Der schlaue Fuchs" (3 tokens, d_k = 4).
Scaling Divided by sqrt(d_k) = sqrt(4) = 2: Result [1, 1, 4]
3
Softmax Softmax([1, 1, 4]) = [0.05, 0.05, 0.90]
4
Weighted Values Assume the value vectors are: V_Der=[1,0], V_schlaue=[0,1], V_Fuchs=[1,1]. Weighted sum: 0.05·[1,0] + 0.05·[0,1] + 0.90·[1,1] = [0.95, 0.95]
5
Result "Fuchs" attends primarily to itself — its own information dominates the output.
Deep Dive: The Scaling Factor sqrt(d_k)
Without dividing by sqrt(d_k), dot products become very large in high dimensions. Large values push softmax to extreme values (near 0 or 1), causing vanishing gradients — so-called "softmax saturation." Dividing by sqrt(d_k) keeps values in a moderate range so softmax produces differentiable probabilities and training remains stable.
Interactive: The Attention Formula
Click on Q, K, or V to understand what each vector does in the attention computation:
Attention Formula
Q·Kᵀ→V
Q — Query
The Query vector encodes what information this token is looking for. Each token generates its own Query by multiplying its embedding with the learned weight matrix W_Q. The Query is then compared against all Keys to determine relevance.
Concrete Example
Formula: Attention(Q, K, V) = softmax(Q·Kᵀ/√dₖ) · V
2Divide by √d_k to prevent softmax saturation → [1, 1, 4]
3Softmax normalizes to probabilities → [0.05, 0.05, 0.90]
4Weighted sum of V vectors → output [0.95, 0.95]
The Transformer Block — Assembling the Machine
The three components of a Transformer Block in detail:
Transformer Block
AnalogyDefinition
Think of a Transformer Block as a committee meeting in three phases. Phase 1 (Multi-Head Attention): multiple expert panels discuss the input simultaneously — one analyzes grammar, another meaning, another tracks references. Phase 2 (Add & Norm): each participant writes a brief combining the new discussion insights with their original notes (residual connection = "don't forget what you already knew"). Phase 3 (Feed-Forward): each participant privately processes and consolidates their updated understanding before the next round.
Example
Committee members influence each other during discussion. In Multi-Head Attention, the heads operate completely independently in parallel — they do not communicate with each other within the same block.
Analogy:
Think of a Transformer Block as a committee meeting in three phases. Phase 1 (Multi-Head Attention): multiple expert panels discuss the input simultaneously — one analyzes grammar, another meaning, another tracks references. Phase 2 (Add & Norm): each participant writes a brief combining the new discussion insights with their original notes (residual connection = "don't forget what you already knew"). Phase 3 (Feed-Forward): each participant privately processes and consolidates their updated understanding before the next round.
Example
Committee members influence each other during discussion. In Multi-Head Attention, the heads operate completely independently in parallel — they do not communicate with each other within the same block.
Definition:
A Transformer Block has three components: (1) Multi-Head Attention — multiple parallel attention computations learning different relationship patterns. (2) Add & Norm — a residual connection adds the original input to the attention output (prevents vanishing gradients), followed by Layer Normalization for training stability. (3) Feed-Forward Network — two dense layers with ReLU, applied independently to each token.
Transformer Block: Data Flow
Multi-Head Attention
Multiple parallel attention computations — each head learns different patterns (grammar, semantics, references)
Add & Norm
Residual connection + Layer Normalization — stabilizes gradient flow in deep networks
Feed-Forward Network
Two dense layers with ReLU — processes the information gathered by attention, per token
Add & Norm
Second residual connection + normalization — output ready for the next block
Modern models stack dozens to hundreds of such blocks:
96blocks
Transformer Blocks GPT-3: 96 blocks, each with 96 attention heads
The Paper "Attention Is All You Need" — Vaswani et al.
2018 Papers
BERT Significantly Improves Language Understanding
A key advance in bidirectional language models and the birth of modern NLP. In October 2018, Jacob Devlin and his team at Google Research published the paper on BERT — Bidirectional Encoder Representations from Transformers. This innovation substantially changed language processing by training deep bidirectional representations from unlabeled text for the first time. Unlike earlier models, BERT considers both left and right context simultaneously across all layers. The results were noteworthy: BERT achieved new state-of-the-art results on eleven NLP tasks and improved the GLUE score by a notable 7.7 percentage points to 80.5%. The pre-training itself took several days on many TPUs — but the open-source release democratized cutting-edge technology: the ready-trained model could be fine-tuned for a custom task on a single cloud TPU in about 30 minutes. BERT established the pre-training/fine-tuning paradigm that today forms the foundation of all large language models.
Watch Out: More Attention Heads ≠ Automatically Better
More heads mean more parallel perspectives, but not necessarily better results. The heads share the total dimension — with the same model size, individual heads become smaller. The optimal number of heads depends on the task and model size.
Deep Dive: O(n²) — The Quadratic Cost of Attention
Self-Attention compares every token with every other — that's n × n comparisons. Double the sequence length, quadruple the cost. At 1,000 tokens: 1 million comparisons. At 10,000 tokens: 100 million. This is why context length is a hard problem: GPT-3 was limited to 2,048 tokens. Modern models use various optimizations (Flash Attention, Sparse Attention) to enable longer contexts — but the fundamental quadratic complexity remains.
Key Takeaways
Self-Attention gives every token direct access to every other token — in parallel, without sequential bottleneck and without vanishing gradient over distance. The weights are dynamic (computed fresh for each input), not static like CNN filters.
The QKV mechanism structures attention as an information retrieval system: Query asks "what do I need?", Key offers "what do I have?", Value delivers the actual content. The dot product Q·K measures relevance, softmax normalizes to probabilities.
A Transformer Block combines Multi-Head Attention, residual connections with Layer Normalization, and a Feed-Forward Network. GPT-3 stacks 96 such blocks with 96 heads each — 175 billion parameters total.
Checkpoint
What two fundamental problems of RNNs does Self-Attention solve, and how exactly does it do so?
Walk through the steps of a QKV attention computation using a simple example: dot products, scaling, softmax, and weighted output.
What are the three components of a Transformer Block, and why does this architecture enable massive GPU parallelization?
Quiz: Transformers & Attention
Question 1 / 4
Not completed
What fundamental problem of RNNs does Self-Attention solve by giving every token direct access to every other token?
1. What fundamental problem of RNNs does Self-Attention solve by giving every token direct access to every other token?
☐ A) RNNs are too slow to train on GPUs.
☐ B) Information from early tokens gets lost over long sequences (vanishing gradient), because it must pass through every intermediate hidden state.
☐ C) RNNs cannot process text, only images.
☐ D) RNNs produce too many parameters.
2. In the QKV attention computation, the dot product Q·K for token A and token B equals 12, and d_k = 64. What is the scaled score before softmax?
☐ A) 12
☐ B) 12/64 = 0.1875
☐ C) 12/8 = 1.5
☐ D) 12 × 8 = 96
3. A Transformer model has 12 blocks, each with 12 attention heads. How many different attention patterns are computed in total across all blocks?
☐ A) 12 (one per block)
☐ B) 24 (12 blocks + 12 heads)
☐ C) 144 (12 blocks × 12 heads)
☐ D) 48 (12 blocks × 4 tokens)
4. Self-Attention has O(n²) computational complexity where n is the sequence length. If processing a 1,000-token document requires X computation, approximately how much more does a 10,000-token document require?