Back to Learning Area

Large Language Models

The model that basically only predicts the next word — and makes a remarkable career doing it.

  • Architectures
  • 12 min
  • Expert

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: How large AI language models work

What is an LLM?

LLM means: large language model. This is a very large AI language model. It can understand and write text. ChatGPT is a famous LLM.

How an LLM learns

An LLM learns from very many texts. It reads more texts than a human ever could. Training has a simple goal. The model should predict the next word. It learns language and many facts while doing this.

Parameters: the strength of the model

An LLM has billions of parameters. Parameters are settings in the model. The more parameters, the more knowledge. ChatGPT had 175 billion parameters. This enables complex answers.

You've seen how Transformers use self-attention to process sequences. Now imagine scaling that architecture to 175 billion parameters and feeding it more text than any human could read in a hundred lifetimes. The result: a system that was only asked to predict the next word — yet learned to write poetry, debug code, and explain quantum physics.

This article explains how LLMs work, why scaling is so predictable, and what transforms a raw text generator into the assistant you know as ChatGPT.

The Simplest Task That Changes Everything

Definition

Large Language Model (LLM)

Analogy

Imagine someone learning a foreign language purely by reading millions of books — no teacher, no dictionary, no grammar rules. Just by predicting what word comes next, they gradually absorb grammar, facts, style, and even cultural knowledge. At some point, they can write original essays in that language, even though they were never explicitly taught how.

The training objective is strikingly simple: given a sequence of tokens, predict the next one. The entire training text provides the labels — every token is simultaneously input and target. That is why no human annotators are needed.

Example: The sentence "The cat sits on the ___". The model assigns high probabilities to words like "roof", "table", or "sofa", and low probabilities to "highway" or "refrigerator". From billions of such predictions, a deep statistical model of language structure emerges.

Note

Emergent Abilities

At sufficient scale, abilities appear that were never explicitly trained: summarization, translation, code generation, and logical reasoning. None of these were taught to the model — they emerge as a byproduct of scaling. How abruptly these abilities really appear is disputed, though: Schaeffer et al. (2023) showed that some apparent jumps turn into smooth improvements when measured differently.

Caution

Do LLMs Understand Language?

Whether LLMs actually understand language or merely reproduce statistical patterns is one of the central open questions in AI research and philosophy. There is no scientific consensus yet.

GPT-3 marked a milestone: 175 billion parameters, trained on 300 billion tokens, estimated training cost approximately $4.6 million. Trained on more text than a human could read in a hundred lifetimes.

Caution

Misconception

"LLMs understand language the way humans do."

They compute statistical distributions over token sequences. Whether this pattern matching constitutes genuine understanding is actively debated. The system produces human-like results without processing meaning the way biological brains do. The key difference to the analogy: a human needs a connection to the physical world to learn meaning — the model builds a purely statistical map without knowing how a word feels.

Interactive: Try Next-Token Prediction

Move the temperature slider and observe how the probability distribution over possible next tokens changes. Sample tokens and see how the actual frequency distribution approaches the theoretical probability with more samples.

An LLM has generated the beginning "Das Wetter heute ___" ("The weather today ___") and computes probabilities for the next word. The most natural continuation is "ist" ("is") — but the temperature determines whether the model always picks the safe choice or dares more unusual continuations.

Controls
Standard (T≈1.0): The original logit probabilities are used. Balance between precision and variety.
Samples0Most frequent–Expectedist · 73 %

Probability Distribution (at T=1.0)

ist
73.3%
war
13.4%
wird
8.1%
soll
2.7%
bleibt
1.5%
kann
1.0%

Results (0 Samples)

No samples yet — click "Sample token"

Start the Experiment

Click "Sample token" to see how the LLM samples at the current temperature. Observe how the distribution of results approaches the theoretical probability with more samples.

The Recipe for Scale

Kaplan et al. (2020) showed empirically that LLM performance improves as a power law when scaling three factors together: model parameters, training data (tokens), and compute. The improvement is predictable and continuous — at least for the prediction error (loss); individual abilities can still become visible only above a certain size. From this, people first concluded that extra compute should mainly go into bigger models — and the models grew ever more gigantic. Since 2024 there has been a fourth lever: compute at answer time (test-time compute). If you let a model think longer before answering, it can beat a much larger model on suitable tasks.

GPT-3 Parameters

175 B

Largest dense language model in 2020

Chinchilla Ratio

20:1

Tokens per parameter (optimal)

GPT-3 Training Cost

$4.6 M

Estimate for one training run

Llama 3 Training Data

15 T tokens

Meta, 2024 (405B model)

2019

GPT-2 (OpenAI)

1.5B parameters. First evidence that scaling alone produces remarkable text generation.

2020

GPT-3 (OpenAI)

175B parameters, 300B tokens. Demonstrates few-shot learning: the model solves new tasks with just a few examples in the prompt.

2022

Chinchilla (DeepMind)

70B parameters, 1.4T tokens. Matches or exceeds GPT-3 with 60% fewer parameters — the optimal ratio is ~20 tokens per parameter.

2023

GPT-4 (OpenAI)

Estimated ~1.8T parameters (Mixture of Experts — a technique where the model consists of specialized sub-networks). Multimodal — processes text and images. Exact architecture not published.

2024

Llama 3 (Meta)

8B/70B/405B variants, 15T tokens. Open-weight models: the weights are free to download, but the license comes with conditions. The smaller variants are deliberately overtrained for inference efficiency (8B: roughly 1,900 tokens per parameter).

Concrete comparison: GPT-3 (175B parameters, 300B tokens, undertrained) versus Chinchilla (70B parameters, 1.4T tokens, optimal) — Chinchilla does clearly better on most benchmarks and is much cheaper to run, even though its training took almost twice the compute.

Hoffmann et al. (2022) showed: the optimal ratio for compute-efficient training is approximately 20 tokens per parameter. GPT-3 with 175B parameters and 300B tokens had only 1.7 tokens per parameter — massively undertrained.

Chinchilla with 70B parameters and 1.4T tokens (20 tokens/parameter) matched or exceeded GPT-3 on most benchmarks — at a fraction of the model size.

Smaller modern models like Llama 3 8B (15T tokens = roughly 1,900 tokens/parameter) are deliberately trained beyond the Chinchilla optimum. Reason: smaller, overtrained models are cheaper at inference because fewer parameters need to be loaded.

Caution

Misconception

"Bigger models are always better."

A 70B model with the right amount of data can match or exceed a 175B model with too little data. Size without proportional data is wasted compute.

From Text Generator to Assistant

A classic LLM like the first ChatGPT goes through three training phases. Each phase adds a different capability. The cost and time required drop drastically after pre-training.

  1. 1. Pre-training$M, months
  2. 2. Instruction Tuning$, days
  3. 3. RLHF$$, weeks

Phase 1: Pre-training. The most expensive phase (millions of dollars). The model reads massive amounts of text and learns general language structures, facts, and relationships. Result: a text completion engine that can continue any sentence — but does not follow instructions.

Phase 2: Instruction Tuning. Supervised fine-tuning on (prompt, response) pairs. The model learns to follow instructions: question leads to helpful answer instead of arbitrary text completion. Relatively cheap and fast.

Phase 3: RLHF (Reinforcement Learning from Human Feedback). Human raters rank different model outputs. A reward model is trained from this and guides further optimization. The model becomes more helpful, safer, and more honest. Since 2024, reasoning models such as OpenAI o1 or DeepSeek-R1 add another phase: reinforcement learning on tasks with checkable answers, such as math or code. The model learns to think longer before answering. This phase is anything but cheap: xAI says it ran it at pre-training scale for Grok 4.

GPT-3 (Base Model)

Prompt: "Explain quantum computing". Response: Continues the text arbitrarily, possibly a Wikipedia-style paragraph, often drifts off-topic. No structure, no addressing the user.

ChatGPT (Instruction Tuning + RLHF)

Prompt: "Explain quantum computing". Response: Structured, helpful explanation addressing the user. Same pre-training as the base, different behavior through additional training steps.

The Foundation Model concept: a universally pre-trained model becomes the basis for specialized applications. Medicine, law, customer service — via fine-tuning, a general model becomes an expert. You don't train from scratch every time.

Meta published the model weights with Llama 3, as did DeepSeek and, since 2025, even OpenAI (gpt-oss). Almost anyone can download, adapt, and deploy the models; with Llama, only very large providers need an extra license. Only a few companies (such as Meta, Google, OpenAI, Anthropic, or DeepSeek) can afford pre-training — at the top end it costs hundreds of millions of dollars and requires thousands of specialized chips.

But fine-tuning is affordable. Democratization happens because anyone can build on a pre-trained foundation model. This fundamentally changes the economic structure of AI: pre-training is an oligopoly, fine-tuning is accessible to all.

Caution

Misconception

"Fine-tuning means retraining the entire model from scratch."

Fine-tuning starts from the finished weights and changes them only slightly (often using parameter-efficient methods like LoRA — a technique that trains only a few additional parameters instead of the entire model). It is like adding a specialization on top of a broad education — not redoing the education itself.

The next article on Transfer Learning & Fine-Tuning dives deeper into the fine-tuning process and shows how to create a specialized model for just a few hundred dollars.

Interactive: Explore the Attention Formula

Click on individual terms of the attention formula to understand their meaning. Notice how Query, Key, and Value work together to weigh relevant information.

Attention — Compact Form

Attention(Q,K,V)softmax(S)V
Attention(Q,K,V) — Result

The attention output: a weighted sum of value vectors. For each token, the mechanism computes which other tokens are relevant and combines their information accordingly.

Concrete Example

Given: Attention(Q, K, V) = softmax(QKᵀ/√dₖ) · V

Components: S = QKᵀ/√dₖ (Scoring Function) , A = softmax(S) · V (Aggregation)

1Compute similarity: Q · Kᵀ, scaled by √dₖ → QKᵀ/√dₖ
2Create weights: softmax normalizes the scores → softmax(S)
3Weighted sum: attention weights × V → A

Note

Key Takeaways

  1. LLMs learn by predicting the next token — no human labels needed. The training data IS the supervision signal.
  2. Scaling laws show empirically that performance improves predictably with more parameters, data, and compute — but only when all three scale together (Chinchilla rule: ~20 tokens per parameter).
  3. A raw LLM is just a text completion engine. Instruction tuning and RLHF transform it into a helpful assistant.
  4. Bigger is not always better. A well-trained 70B model can outperform a poorly trained 175B model.
  5. LLMs compute statistical probabilities over tokens. Whether this constitutes "understanding" remains an open scientific and philosophical question.

Check

Quiz: Large Language Models

Not completed

What training objective is used during the pre-training phase of LLMs?

Select one answer

Answer Key: 1) B · 2) C · 3) C · 4) B · 5) B · 6) B

Check

Comprehension Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done