Large Language Models
The model that basically only predicts the next word — and makes a remarkable career doing it.
You've seen how Transformers use self-attention to process sequences. Now imagine scaling that architecture to 175 billion parameters and feeding it more text than any human could read in a hundred lifetimes. The result: a system that was only asked to predict the next word — yet learned to write poetry, debug code, and explain quantum physics.
This article explains how LLMs work, why scaling is so predictable, and what transforms a raw text generator into the assistant you know as ChatGPT.
The Simplest Task That Changes Everything
Definition
Large Language Model (LLM)
Analogy
Definition
The training objective is strikingly simple: given a sequence of tokens, predict the next one. The entire training text provides the labels — every token is simultaneously input and target. That is why no human annotators are needed.
Example: The sentence "The cat sits on the ___". The model assigns high probabilities to words like "roof", "table", or "sofa", and low probabilities to "highway" or "refrigerator". From billions of such predictions, a deep statistical model of language structure emerges.
Note
Emergent Abilities
At sufficient scale, abilities appear that were never explicitly trained: summarization, translation, code generation, and logical reasoning. None of these were taught to the model — they emerge as a byproduct of scaling. How abruptly these abilities really appear is disputed, though: Schaeffer et al. (2023) showed that some apparent jumps turn into smooth improvements when measured differently.
Caution
Do LLMs Understand Language?
Whether LLMs actually understand language or merely reproduce statistical patterns is one of the central open questions in AI research and philosophy. There is no scientific consensus yet.
GPT-3 marked a milestone: 175 billion parameters, trained on 300 billion tokens, estimated training cost approximately $4.6 million. Trained on more text than a human could read in a hundred lifetimes.
Caution
Misconception
"LLMs understand language the way humans do."
They compute statistical distributions over token sequences. Whether this pattern matching constitutes genuine understanding is actively debated. The system produces human-like results without processing meaning the way biological brains do. The key difference to the analogy: a human needs a connection to the physical world to learn meaning — the model builds a purely statistical map without knowing how a word feels.
Interactive: Try Next-Token Prediction
Move the temperature slider and observe how the probability distribution over possible next tokens changes. Sample tokens and see how the actual frequency distribution approaches the theoretical probability with more samples.
An LLM has generated the beginning "Das Wetter heute ___" ("The weather today ___") and computes probabilities for the next word. The most natural continuation is "ist" ("is") — but the temperature determines whether the model always picks the safe choice or dares more unusual continuations.
Probability Distribution (at T=1.0)
Results (0 Samples)
No samples yet — click "Sample token"
Click "Sample token" to see how the LLM samples at the current temperature. Observe how the distribution of results approaches the theoretical probability with more samples.
The Recipe for Scale
Kaplan et al. (2020) showed empirically that LLM performance improves as a power law when scaling three factors together: model parameters, training data (tokens), and compute. The improvement is predictable and continuous — at least for the prediction error (loss); individual abilities can still become visible only above a certain size. From this, people first concluded that extra compute should mainly go into bigger models — and the models grew ever more gigantic. Since 2024 there has been a fourth lever: compute at answer time (test-time compute). If you let a model think longer before answering, it can beat a much larger model on suitable tasks.
GPT-3 Parameters
175 B
Largest dense language model in 2020
Chinchilla Ratio
20:1
Tokens per parameter (optimal)
GPT-3 Training Cost
$4.6 M
Estimate for one training run
Llama 3 Training Data
15 T tokens
Meta, 2024 (405B model)
GPT-2 (OpenAI)
1.5B parameters. First evidence that scaling alone produces remarkable text generation.
GPT-3 (OpenAI)
175B parameters, 300B tokens. Demonstrates few-shot learning: the model solves new tasks with just a few examples in the prompt.
Chinchilla (DeepMind)
70B parameters, 1.4T tokens. Matches or exceeds GPT-3 with 60% fewer parameters — the optimal ratio is ~20 tokens per parameter.
GPT-4 (OpenAI)
Estimated ~1.8T parameters (Mixture of Experts — a technique where the model consists of specialized sub-networks). Multimodal — processes text and images. Exact architecture not published.
Llama 3 (Meta)
8B/70B/405B variants, 15T tokens. Open-weight models: the weights are free to download, but the license comes with conditions. The smaller variants are deliberately overtrained for inference efficiency (8B: roughly 1,900 tokens per parameter).
Concrete comparison: GPT-3 (175B parameters, 300B tokens, undertrained) versus Chinchilla (70B parameters, 1.4T tokens, optimal) — Chinchilla does clearly better on most benchmarks and is much cheaper to run, even though its training took almost twice the compute.
Hoffmann et al. (2022) showed: the optimal ratio for compute-efficient training is approximately 20 tokens per parameter. GPT-3 with 175B parameters and 300B tokens had only 1.7 tokens per parameter — massively undertrained.
Chinchilla with 70B parameters and 1.4T tokens (20 tokens/parameter) matched or exceeded GPT-3 on most benchmarks — at a fraction of the model size.
Smaller modern models like Llama 3 8B (15T tokens = roughly 1,900 tokens/parameter) are deliberately trained beyond the Chinchilla optimum. Reason: smaller, overtrained models are cheaper at inference because fewer parameters need to be loaded.
Caution
Misconception
"Bigger models are always better."
A 70B model with the right amount of data can match or exceed a 175B model with too little data. Size without proportional data is wasted compute.
From Text Generator to Assistant
A classic LLM like the first ChatGPT goes through three training phases. Each phase adds a different capability. The cost and time required drop drastically after pre-training.
- 1. Pre-training$M, months
- 2. Instruction Tuning$, days
- 3. RLHF$$, weeks
Phase 1: Pre-training. The most expensive phase (millions of dollars). The model reads massive amounts of text and learns general language structures, facts, and relationships. Result: a text completion engine that can continue any sentence — but does not follow instructions.
GPT-1: The Birth of Generative Pre-Training
The foundation of all modern large language models through unsupervised pre-training. On June 11, 2018, Alec Radford and his OpenAI team published the landmark paper 'Improving Language Understanding by Generative Pre-Training.' This work combined the Transformer architecture with unsupervised pre-training for the first time and established a two-stage paradigm: first, generative training on large text corpora, then fine-tuning for specific tasks. With 117 million parameters and training on the BooksCorpus dataset of over 7,000 unpublished books across various genres, GPT-1 demonstrated that transfer learning works for language understanding. The twelve-layer, decoder-only Transformer architecture with masked self-attention set the template for the entire GPT series. This innovation turned the 2017 Transformer architecture into a practical tool for a wide range of NLP tasks and launched the era of large language models.
Phase 2: Instruction Tuning. Supervised fine-tuning on (prompt, response) pairs. The model learns to follow instructions: question leads to helpful answer instead of arbitrary text completion. Relatively cheap and fast.
Phase 3: RLHF (Reinforcement Learning from Human Feedback). Human raters rank different model outputs. A reward model is trained from this and guides further optimization. The model becomes more helpful, safer, and more honest. Since 2024, reasoning models such as OpenAI o1 or DeepSeek-R1 add another phase: reinforcement learning on tasks with checkable answers, such as math or code. The model learns to think longer before answering. This phase is anything but cheap: xAI says it ran it at pre-training scale for Grok 4.
Prompt: "Explain quantum computing". Response: Continues the text arbitrarily, possibly a Wikipedia-style paragraph, often drifts off-topic. No structure, no addressing the user.
Prompt: "Explain quantum computing". Response: Structured, helpful explanation addressing the user. Same pre-training as the base, different behavior through additional training steps.
The Foundation Model concept: a universally pre-trained model becomes the basis for specialized applications. Medicine, law, customer service — via fine-tuning, a general model becomes an expert. You don't train from scratch every time.
LLaMA: Foundation Model with a Research License
The democratization of Large Language Models through open research models. On February 24, 2023, Meta AI released LLaMA (Large Language Model Meta AI) — a collection of foundation models ranging from 7B to 65B parameters, trained exclusively on publicly available data. The landmark paper 'LLaMA: Open and Efficient Foundation Language Models' demonstrated that state-of-the-art performance is achievable without proprietary datasets. LLaMA enabled researchers without access to large infrastructure to study advanced language models. The inference code was released under a GPLv3 license, while model access was granted on a case-by-case basis for academic research. Trained on trillions of tokens and offered in various model sizes, LLaMA addressed diverse hardware requirements. This work catalyzed a wave of open LLM research and inspired numerous follow-up models in the open-source community.
Meta published the model weights with Llama 3, as did DeepSeek and, since 2025, even OpenAI (gpt-oss). Almost anyone can download, adapt, and deploy the models; with Llama, only very large providers need an extra license. Only a few companies (such as Meta, Google, OpenAI, Anthropic, or DeepSeek) can afford pre-training — at the top end it costs hundreds of millions of dollars and requires thousands of specialized chips.
But fine-tuning is affordable. Democratization happens because anyone can build on a pre-trained foundation model. This fundamentally changes the economic structure of AI: pre-training is an oligopoly, fine-tuning is accessible to all.
Caution
Misconception
"Fine-tuning means retraining the entire model from scratch."
Fine-tuning starts from the finished weights and changes them only slightly (often using parameter-efficient methods like LoRA — a technique that trains only a few additional parameters instead of the entire model). It is like adding a specialization on top of a broad education — not redoing the education itself.
The next article on Transfer Learning & Fine-Tuning dives deeper into the fine-tuning process and shows how to create a specialized model for just a few hundred dollars.
Interactive: Explore the Attention Formula
Click on individual terms of the attention formula to understand their meaning. Notice how Query, Key, and Value work together to weigh relevant information.
Attention — Compact Form
The attention output: a weighted sum of value vectors. For each token, the mechanism computes which other tokens are relevant and combines their information accordingly.
Concrete Example
Given: Attention(Q, K, V) = softmax(QKᵀ/√dₖ) · V
Components: S = QKᵀ/√dₖ (Scoring Function) , A = softmax(S) · V (Aggregation)
Note
Key Takeaways
- LLMs learn by predicting the next token — no human labels needed. The training data IS the supervision signal.
- Scaling laws show empirically that performance improves predictably with more parameters, data, and compute — but only when all three scale together (Chinchilla rule: ~20 tokens per parameter).
- A raw LLM is just a text completion engine. Instruction tuning and RLHF transform it into a helpful assistant.
- Bigger is not always better. A well-trained 70B model can outperform a poorly trained 175B model.
- LLMs compute statistical probabilities over tokens. Whether this constitutes "understanding" remains an open scientific and philosophical question.
Check
Quiz: Large Language Models
What training objective is used during the pre-training phase of LLMs?
Select one answer
1. What training objective is used during the pre-training phase of LLMs?
2. GPT-3 has 175 billion parameters and was trained on 300 billion tokens. According to the Chinchilla rule (~20 tokens per parameter), how many tokens should it ideally have been trained on?
3. A company wants to build a customer support chatbot. They have a pre-trained 70B base model. What additional training steps are needed to make it function as a helpful, safe assistant?
4. Model A has 175B parameters trained on 300B tokens. Model B has 70B parameters trained on 1.4T tokens. Both use the same Transformer architecture. Which model likely performs better, and why?
5. Why does self-supervised learning eliminate the need for human-annotated training data in LLM pre-training?
6. An LLM was trained exclusively to predict the next word. Yet it can write code, summarize texts, and translate between languages. Which explanation is most accurate?
Check
Comprehension Check
Can you explain this in your own words? Tick what you've got.
0 of 3 done