Tokenization: The Machine Alphabet
How letters become tokens, and why LLMs see the world through this lens.
Ask ChatGPT how many R's are in "strawberry" — for a long time, it often got it wrong. Not because it is bad at counting, but because it never sees the letters at all. It sees three chunks — "str", "aw", "berry" (with GPT-4's tokenizer) — glued together by an algorithm that runs before the conversation.
Everything a language model does starts with how it splits text into pieces. This invisible first step is called tokenization, and it determines what the model can see, how much it can remember, and what it finds difficult.
Tokens — Why Not Just Use Words?
Definition
Token
Analogy
Example
Definition
Three Granularity Levels
There are three ways to convert text into numbers. Each comes at a price:
Enormous vocabulary, unknown words impossible, typos become "unknown"
Manageable vocabulary (32k-200k), any input processable, unknown words get split into pieces
The third option — character or byte level — produces a tiny vocabulary (exactly 256 entries at the byte level), but extremely long sequences. Subword tokenization solves both problems at once: the vocabulary stays practical, and no input is truly "unknown".
Caution
The Strawberry Problem
When asked "How many R's are in 'strawberry'?", older LLMs often answered "2" instead of "3". The reason is architectural: the tokenizer splits "strawberry" into tokens like "str", "aw", "berry" before the model ever sees the text.
The model processes these three chunks as atomic units — it cannot look inside a token to count individual characters. This is not an ordinary bug that can simply be fixed; it is a fundamental consequence of subword tokenization.
Caution
Misconception: "Tokens are basically just words"
If tokens were words, every language in the world would need its own enormous vocabulary, unknown words could not be represented by the system, and typos would be unprocessable. Subword tokenization was invented precisely to avoid these problems. A token can be a full word ("the"), a word fragment ("tion"), a single character ("x"), or even a space plus characters (" the"). The boundaries are determined by statistical frequency in the training data, not by linguistic rules.
Interactive: Tokenize Text Yourself
Type any text and watch it being split into tokens. Switch between character, word, and subword tokenization and compare: How many tokens does each method produce? Pay special attention to how subword tokenization breaks unknown words into known parts — that is exactly what BPE does, which you will learn about in the next section.
Live Tokenizer
Common syllables and subwords are merged (like GPT does). Best compromise between vocabulary size and token count.
Show token list (23 Tokens)
LLMs don't process words but tokens. A longer text requires more tokens, which means more computation and cost. Subword tokenizers (like BPE) offer the best compromise: fewer tokens than character-level, but more flexible than word-level splitting.
Byte-Pair Encoding — How the Vocabulary Learns Itself
Definition
Byte-Pair Encoding (BPE)
Analogy
Example
Definition
BPE on a Mini-Corpus
Corpus: "low lower lowest new newer newest". Initial tokens: individual characters [l, o, w, e, r, s, t, n]. The order is simplified: real BPE would merge w + e first here, because that pair occurs four times.
- 1. Merge: l + o → new token "lo"
- 2. Merge: lo + w → new token "low"
- 3. Merge: e + r → new token "er"
- 4. Merge: e + s → new token "es"
- 5. Merge: es + t → new token "est"
- 6. Merge: n + e → new token "ne"
- 7. Merge: ne + w → new token "new"
After 7 merges: "lowest" = ["low", "est"]. "newer" = ["new", "er"]. Each word is assembled from reusable pieces, and the pieces emerged from pure co-occurrence statistics.
Typical vocabulary size of modern tokenizers
32 k–200k
Tokenizers trained predominantly on English text split German or Hindi text into significantly more tokens — depending on the tokenizer, roughly 30% (GPT-4o) to 50% (GPT-4) more for German. The reason: efficient merges already exist for English words, while rarer languages are fragmented more heavily. The consequence is directly felt: 30% more tokens mean 30% higher API costs for the same content, and the context window holds just under a quarter less text.
More tokens for German text vs. English (GPT-4o's tokenizer)
~30 %
Caution
Misconception: "The tokenizer understands word boundaries and morphology"
BPE has no concept of linguistic structure. It does not know that "un-" is a prefix or that "-ing" is a suffix. If "un" + "g" appear next to each other very often in the training corpus (as they do in German-heavy corpora), then "ung" becomes a single token regardless of whether it is a suffix or part of a stem. This purely statistical process also explains the multilingual penalty.
BPE live: watch the vocabulary grow
The live tokenizer above shows the finished result. Here you see the training step before it: how BPE builds that vocabulary in the first place.
The Context Window — How Much a Model Can Remember
Definition
Context Window
Analogy
Example
Definition
GPT-4 Turbo (shut down) Context Window (Tokens)
128 k
Claude 3 (2024, shut down) Context Window (Tokens)
200 k
GPT-3: The 175-Billion-Parameter Model
The breakthrough to few-shot learning and emergent AI capabilities. On May 28, 2020, OpenAI's team led by Tom Brown presented the significant paper 'Language Models are Few-Shot Learners' - GPT-3 with 175 billion parameters, more than 100 times larger than GPT-2. The scale revealed emergent capabilities: the model could solve new tasks with just a few examples, without any fine-tuning. From translations to word puzzles to three-digit arithmetic, GPT-3 demonstrated impressive versatility. Human evaluators could barely distinguish news articles generated by GPT-3 from real ones. Through in-context learning alone, GPT-3 approached the state of the art on individual SuperGLUE subtasks - though on the overall benchmark it remained well behind the fine-tuned top models (around 89 points) with a score of roughly 71.8. Thirty-one OpenAI researchers (Tom Brown and 30 co-authors) demonstrated that massive parameter scaling can produce qualitatively new capabilities. GPT-3 laid the foundation for ChatGPT and the modern LLM era.
A Novel in Tokens
An average novel contains 80,000-130,000 words, which translates to roughly 100,000-160,000 tokens (tokens > words because punctuation and subword splits add overhead). A model with a 128k-token window can fit a shorter novel; a long one at 160,000 tokens needs a bigger window, like Claude 3's 200,000 tokens.
Caution
Lost-in-the-Middle Effect
Researchers have demonstrated that when a key fact is placed in the middle of a long context (say, page 150 of 300), models retrieve it significantly less reliably than facts placed at the beginning or end. The context window is not a uniform memory — it has a U-shaped retrieval curve where the edges are privileged.
Caution
Misconception: "A 200k context window means the model remembers everything perfectly"
Two problems undermine this assumption. First, the Lost-in-the-Middle effect means retrieval quality degrades for information in the center of very long contexts, even though it technically fits. Second, longer contexts cost disproportionately more compute (attention grows quadratically with length), so many applications deliberately limit context usage far below the theoretical maximum. A 200k window is a ceiling, not a guarantee of quality.
Interactive: Token Costs and the Context Window
A model does not count in words but in tokens, and space and price depend on them: every token takes up room in the context window and is billed by the API. This calculator estimates both. It counts the words and converts them into tokens with a rule of thumb, preset from the novel example above (80,000 words ≈ 100,000 tokens). The article names no price: enter your provider's; the preset is 1 US dollar per million tokens.
Note
Key Takeaways
- Tokens are not words — they are statistical subword units created by the tokenizer before the model sees the text.
- BPE builds the vocabulary through pure frequency statistics, not linguistic understanding.
- The context window is a hard limit measured in tokens — when full, older information is completely lost.
- Even within the context window, models struggle with information in the middle (Lost-in-the-Middle effect).
Next article: Embeddings — how these tokens become vectors with meaning.
Check
Checkpoint: Tokenization
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: Tokenization
What is the main advantage of subword tokenization over word-level tokenization?
Select one answer
1. What is the main advantage of subword tokenization over word-level tokenization?
2. In BPE, the character pair "t" + "h" is merged into the token "th". What determined this merge?
3. A BPE tokenizer trained predominantly on English text processes the German word "Handlungsempfehlung" (19 characters). What is the most likely outcome compared to the English word "recommendation" (14 characters)?
4. You are building a customer support chatbot with a 16k-token context window. A customer's conversation history contains 20,000 tokens. What happens to the earliest messages, and what engineering solution could help?
5. Suppose a 200k-token model retrieves a critical fact on page 150 of a 300-page document less reliably than facts on page 1 or page 300. What explains this?