How letters become tokens, and why LLMs see the world through this lens.
Architectures 12 min Intermediate June 15, 2026
Ask ChatGPT how many R's are in "strawberry" and it often gets it wrong. Not because it is bad at counting, but because it never sees the letters at all. It sees three chunks — "Str", "aw", "berry" — glued together by an algorithm that runs before the conversation.
Everything a language model does starts with how it splits text into pieces. This invisible first step is called tokenization, and it determines what the model can see, how much it can remember, and what it will never be able to do.
Tokens — Why Not Just Use Words?
Token
AnalogyDefinition
Imagine three toy systems. System A gives you individual atoms — you can build anything, but assembling a house takes millions of pieces. System B gives you pre-built houses — fast to place, but if you need a shape that was not pre-manufactured, you are stuck. System C — LEGO — gives you a few thousand standardized bricks of varying sizes. Common structures come as larger pre-built pieces; unusual shapes are assembled from smaller bricks. That is exactly the trade-off subword tokenization makes: characters are atoms, words are pre-built houses, and tokens are LEGO bricks — the useful middle ground.
Example
LEGO bricks have physical shapes that humans recognize. Tokens, however, are arbitrary character sequences like "tion", " the", or "ishable" — with no inherent meaning. The analogy oversells the "meaningfulness" of individual pieces.
Analogy:
Imagine three toy systems. System A gives you individual atoms — you can build anything, but assembling a house takes millions of pieces. System B gives you pre-built houses — fast to place, but if you need a shape that was not pre-manufactured, you are stuck. System C — LEGO — gives you a few thousand standardized bricks of varying sizes. Common structures come as larger pre-built pieces; unusual shapes are assembled from smaller bricks. That is exactly the trade-off subword tokenization makes: characters are atoms, words are pre-built houses, and tokens are LEGO bricks — the useful middle ground.
Example
LEGO bricks have physical shapes that humans recognize. Tokens, however, are arbitrary character sequences like "tion", " the", or "ishable" — with no inherent meaning. The analogy oversells the "meaningfulness" of individual pieces.
Definition:
A token is the fundamental unit that a language model processes — not a word, not a character, but a subword unit determined by a statistical algorithm. Modern language models universally use subword tokenization with vocabulary sizes of 32,000 to 100,000 entries.
Three Granularity Levels
There are three ways to convert text into numbers. Each comes at a price:
Word-Level
Enormous vocabulary, unknown words impossible, typos crash the system
Subword-Level (Tokens)
Manageable vocabulary (32k-100k), any input processable, unknown words get split into pieces
The third option — character-level — produces a tiny vocabulary of only ~256 entries, but extremely long sequences. Subword tokenization solves both problems at once: the vocabulary stays practical, and no input is truly "unknown".
The Strawberry Problem
When you ask an LLM "How many R's are in 'strawberry'?", it often answers "2" instead of "3". The reason is architectural: the tokenizer splits "strawberry" into tokens like "Str", "aw", "berry" before the model ever sees the text.
The model processes these three chunks as atomic units — it cannot look inside a token to count individual characters. This is not a bug that will be fixed with more training data; it is a fundamental consequence of subword tokenization.
Misconception: "Tokens are basically just words"
If tokens were words, every language in the world would need its own enormous vocabulary, unknown words would crash the system, and typos would be unprocessable. Subword tokenization was invented precisely to avoid these problems. A token can be a full word ("the"), a word fragment ("tion"), a single character ("x"), or even a space plus characters (" the"). The boundaries are determined by statistical frequency in the training data, not by linguistic rules.
Interactive: Tokenize Text Yourself
Type any text and watch it being split into tokens. Switch between character, word, and subword tokenization and compare: How many tokens does each method produce? Pay special attention to how subword tokenization breaks unknown words into known parts — that is exactly what BPE does, which you will learn about in the next section.
Live Tokenizer
Common syllables and subwords are merged (like GPT does). Best compromise between vocabulary size and token count.
LLMs don't process words but tokens. A longer text requires more tokens, which means more computation and cost. Subword tokenizers (like BPE) offer the best compromise: fewer tokens than character-level, but more flexible than word-level splitting.
Character-levelEach character = 1 token. Very many tokens, but no unknown vocabulary.
Word-levelEach word = 1 token. Few tokens, but unknown words are a problem.
Subword (BPE)Common syllables and words are merged. Best compromise — standard for GPT & Co.
Byte-Pair Encoding — How the Vocabulary Learns Itself
Byte-Pair Encoding (BPE)
AnalogyDefinition
Imagine a new employee reading thousands of company emails. The first week, they notice everyone writes "as soon as possible" constantly, so they start writing "ASAP". The next week, "end of day" becomes "EOD". Over months, they build a personal dictionary of abbreviations for the most common phrases. BPE does the same thing: it watches which character pairs appear most often and creates a shorthand (a merged token) for each one.
Example
The crucial difference: the employee abbreviates based on meaning ("as soon as possible" is a semantic unit). BPE abbreviates purely by character frequency — it would merge "th" and "e" into "the" not because "the" is a meaningful word, but because the characters happen to appear next to each other extremely often.
Analogy:
Imagine a new employee reading thousands of company emails. The first week, they notice everyone writes "as soon as possible" constantly, so they start writing "ASAP". The next week, "end of day" becomes "EOD". Over months, they build a personal dictionary of abbreviations for the most common phrases. BPE does the same thing: it watches which character pairs appear most often and creates a shorthand (a merged token) for each one.
Example
The crucial difference: the employee abbreviates based on meaning ("as soon as possible" is a semantic unit). BPE abbreviates purely by character frequency — it would merge "th" and "e" into "the" not because "the" is a meaningful word, but because the characters happen to appear next to each other extremely often.
Definition:
Byte-Pair Encoding (BPE) is the dominant tokenization algorithm used by models like GPT-4 and Llama. Starting with a base vocabulary of individual characters or bytes, BPE scans a training corpus, finds the most frequently co-occurring character pair, merges it into a new single token, and repeats. The final vocabulary size (typically 32,000 to 100,000 tokens) is a hyperparameter.
BPE on a Mini-Corpus
Corpus: "low lower lowest new newer newest". Initial tokens: individual characters [l, o, w, e, r, s, t, n].
1
Most frequent pair: l + o → new token "lo"
2
Most frequent pair: lo + w → new token "low"
3
Most frequent pair: e + r → new token "er"
4
Most frequent pair: e + s → new token "es"
5
Most frequent pair: es + t → new token "est"
6
Most frequent pair: n + e → new token "ne"
7
Most frequent pair: ne + w → new token "new"
After 7 merges: "lowest" = ["low", "est"]. "newer" = ["new", "er"]. Each word is assembled from reusable pieces, and the pieces emerged from pure co-occurrence statistics.
32k–100k
Typical vocabulary size of modern tokenizers
Deep Dive: The Multilingual Penalty
Tokenizers trained predominantly on English text split German or Hindi text into significantly more tokens — roughly 30% more for German. The reason: efficient merges already exist for English words, while rarer languages are fragmented more heavily. The consequence is directly felt: 30% more tokens mean 30% higher API costs and 30% less space in the context window — for the same semantic content.
~30%
More tokens for German text vs. English
Misconception: "The tokenizer understands word boundaries and morphology"
BPE has no concept of linguistic structure. It does not know that "un-" is a prefix or that "-ing" is a suffix. If "ung" appears more frequently than "un" + "g" in the training corpus (as it does in German-heavy corpora), then "ung" becomes a single token regardless of whether it is a suffix or part of a stem. This purely statistical process also explains the multilingual penalty.
BPE live: watch the vocabulary grow
The live tokenizer above shows the finished result. Here you see the training step before it: how BPE builds that vocabulary in the first place.
The Context Window — How Much a Model Can Remember
Context Window
AnalogyDefinition
Imagine writing a research paper at a desk that physically fits exactly 50 pages. You can spread out all 50 pages and reference any of them freely. But the moment you add page 51, page 1 falls off the desk and is gone — not filed, not summarized, just gone. To reference it again, you need someone to fetch it from the archive (this is RAG: Retrieval-Augmented Generation).
Example
The desk also has a quirk: you naturally focus on the pages nearest to you (the most recent) and the ones pinned at the top (the first), but pages in the middle of the stack tend to get overlooked. Limitation: a desk is spatial while a context window is sequential — the analogy does not capture that tokens are processed in order.
Analogy:
Imagine writing a research paper at a desk that physically fits exactly 50 pages. You can spread out all 50 pages and reference any of them freely. But the moment you add page 51, page 1 falls off the desk and is gone — not filed, not summarized, just gone. To reference it again, you need someone to fetch it from the archive (this is RAG: Retrieval-Augmented Generation).
Example
The desk also has a quirk: you naturally focus on the pages nearest to you (the most recent) and the ones pinned at the top (the first), but pages in the middle of the stack tend to get overlooked. Limitation: a desk is spatial while a context window is sequential — the analogy does not capture that tokens are processed in order.
Definition:
The context window (also called context length) is the maximum number of tokens a language model can process in a single processing step. GPT-4 Turbo has a context window of 128,000 tokens; Claude 3 supports up to 200,000 tokens. This is a hard ceiling: once the window is full, older tokens are dropped entirely.
128k
GPT-4 Turbo Context Window (Tokens)
200k
Claude 3 Context Window (Tokens)
2020 Papers
GPT-3: The 175-Billion-Parameter Model
The breakthrough to few-shot learning and emergent AI capabilities. On May 28, 2020, OpenAI's team led by Tom Brown presented the significant paper 'Language Models are Few-Shot Learners' - GPT-3 with 175 billion parameters, more than 100 times larger than GPT-2. The scale revealed emergent capabilities: the model could solve new tasks with just a few examples, without any fine-tuning. From translations to word puzzles to three-digit arithmetic, GPT-3 demonstrated impressive versatility. Human evaluators could barely distinguish news articles generated by GPT-3 from real ones. Through in-context learning alone, GPT-3 approached the state of the art on individual SuperGLUE subtasks - though on the overall benchmark it remained well behind the fine-tuned top models (around 89 points) with a score of roughly 71.8. Thirty-one OpenAI researchers (Tom Brown and 30 co-authors) demonstrated that massive parameter scaling can produce qualitatively new capabilities. GPT-3 laid the foundation for ChatGPT and the modern LLM era.
A Novel in Tokens
An average novel contains 80,000-130,000 words, which translates to roughly 100,000-160,000 tokens (tokens > words because punctuation, spaces, and subword splits add overhead). A model with a 128k-token window can just barely fit one complete novel.
Lost-in-the-Middle Effect
Researchers have demonstrated that when a key fact is placed in the middle of a long context (say, page 150 of 300), models retrieve it significantly less reliably than facts placed at the beginning or end. The context window is not a uniform memory — it has a U-shaped attention curve where the edges are privileged.
Misconception: "A 200k context window means the model remembers everything perfectly"
Two problems undermine this assumption. First, the Lost-in-the-Middle effect means retrieval quality degrades for information in the center of very long contexts, even though it technically fits. Second, longer contexts are exponentially more expensive in compute (quadratic scaling), so many applications deliberately limit context usage far below the theoretical maximum. A 200k window is a ceiling, not a guarantee of quality.
Interactive: How Does an LLM Choose the Next Token?
A language model does not simply predict the most likely token — it samples from a distribution. The temperature parameter controls how random this choice is. Move the slider: at low temperature, the most frequent token dominates. At high temperature, even unlikely tokens get chosen. Click "Sample" repeatedly and observe how the empirical frequency converges toward the theoretical probability.
An LLM has generated the beginning "Das Wetter heute ___" ("The weather today ___") and computes probabilities for the next word. The most natural continuation is "ist" ("is") — but the temperature determines whether the model always picks the safe choice or dares more unusual continuations.
0.1 (focused)2.0 (creative)
Standard (T≈1.0): The original logit probabilities are used. Balance between precision and variety.
Probability Distribution (at T=1.0)
ist
73.3%
war
13.4%
wird
8.1%
soll
2.7%
bleibt
1.5%
kann
1.0%
Results (0 Samples)
No samples yet — click "Sample token"
Start the Experiment
Click "Sample token" to see how the LLM samples at the current temperature. Observe how the distribution of results approaches the theoretical probability with more samples.
Key Takeaways
Tokens are not words — they are statistical subword units created by the tokenizer before the model sees the text.
BPE builds the vocabulary through pure frequency statistics, not linguistic understanding.
The context window is a hard limit measured in tokens — when full, older information is completely lost.
Even within the context window, models struggle with information in the middle (Lost-in-the-Middle effect).
Next article: Embeddings — how these tokens become vectors with meaning.
Checkpoint: Tokenization
Explain in your own words why ChatGPT often fails when asked about the letters in 'strawberry' — and what this has to do with tokenization.
Take the mini-corpus 'make, makes, making'. Start with individual characters [m, a, k, e, s, i, n, g] and perform the BPE merges step by step. Which tokens emerge at the end?
A model has a 128k token context window. A user submits a 300-page document. Which information is lost — and why not evenly?
Quiz: Tokenization
Question 1 / 5
What is the main advantage of subword tokenization over word-level tokenization?
1. What is the main advantage of subword tokenization over word-level tokenization?
☐ A) Subwords are faster to compute
☐ B) Subwords create a manageable vocabulary while still handling unknown words by splitting them into smaller pieces
☐ C) Subwords preserve the exact meaning of each word
☐ D) Subwords eliminate the need for a vocabulary entirely
2. In BPE, the character pair "t" + "h" is merged into the token "th". What determined this merge?
☐ A) "th" is a linguistically meaningful unit (digraph)
☐ B) A human linguist marked "th" as important
☐ C) "t" and "h" co-occurred more frequently than any other remaining pair in the training corpus
☐ D) The model learned during text generation that "th" is useful
3. A BPE tokenizer trained predominantly on English text processes the German word "Handlungsempfehlung" (18 characters). What is the most likely outcome compared to the English word "recommendation" (14 characters)?
☐ A) Both words produce roughly the same number of tokens
☐ B) The German word produces significantly more tokens because the tokenizer has fewer German-specific merges
☐ C) The German word produces fewer tokens because German compounds are more efficient
☐ D) Both words are tokenized character by character
4. You are building a customer support chatbot with a 16k-token context window. A customer's conversation history contains 20,000 tokens. What happens to the earliest messages, and what engineering solution could help?
☐ A) The model summarizes old messages automatically
☐ B) The earliest messages are silently dropped; RAG could retrieve relevant past messages on demand
☐ C) The model compresses old tokens to fit everything
☐ D) The context window automatically expands to accommodate the conversation
5. Researchers found that when a critical fact is placed on page 150 of a 300-page document fed into a 200k-token model, the model retrieves it less reliably than facts on page 1 or page 300. What explains this?
☐ A) The model runs out of memory at page 150
☐ B) The attention mechanism distributes focus non-uniformly, favoring the beginning and end of the sequence (Lost-in-the-Middle effect)
☐ C) Pages in the middle are tokenized differently than pages at the edges
☐ D) The model only processes the first and last 50 pages