Large Language Models
The model that basically only predicts the next word — and makes a remarkable career doing it.
You've seen how Transformers use self-attention to process sequences. Now imagine scaling that architecture to 175 billion parameters and feeding it more text than any human could read in a hundred lifetimes. The result: a system that was only asked to predict the next word — yet learned to write poetry, debug code, and explain quantum physics.
This article explains how LLMs work, why scale matters so precisely, and what transforms a raw text generator into the assistant you know as ChatGPT.
The Simplest Task That Changes Everything
Large Language Model (LLM)
The training objective is strikingly simple: given a sequence of tokens, predict the next one. The entire training text provides the labels — every token is simultaneously input and target. That is why no human annotators are needed.
Example: The sentence "The cat sits on the ___". The model assigns high probabilities to words like "roof", "table", or "sofa", and low probabilities to "highway" or "refrigerator". From billions of such predictions, a deep understanding of language structure emerges.
Emergent Abilities
Do LLMs Understand Language?
GPT-3 marked a milestone: 175 billion parameters, trained on 300 billion tokens, training cost approximately $4.6 million. Trained on more text than a human could read in a hundred lifetimes.
Misconception
Interactive: Try Next-Token Prediction
Move the temperature slider and observe how the probability distribution over possible next tokens changes. Sample tokens and see how the actual frequency distribution approaches the theoretical probability with more samples.
An LLM has generated the beginning "Das Wetter heute ___" ("The weather today ___") and computes probabilities for the next word. The most natural continuation is "ist" ("is") — but the temperature determines whether the model always picks the safe choice or dares more unusual continuations.
Probability Distribution (at T=1.0)
Results (0 Samples)
No samples yet — click "Sample token"
Click "Sample token" to see how the LLM samples at the current temperature. Observe how the distribution of results approaches the theoretical probability with more samples.
The Recipe for Scale
Kaplan et al. (2020) proved empirically that LLM performance improves as a power law when scaling three factors together: model parameters, training data (tokens), and compute. The improvement is predictable and continuous — no sudden jumps.
GPT-2 (OpenAI)
1.5B parameters. First evidence that scaling alone produces remarkable text generation.
GPT-3 (OpenAI)
175B parameters, 300B tokens. Demonstrates few-shot learning: the model solves new tasks with just a few examples in the prompt.
Chinchilla (DeepMind)
70B parameters, 1.4T tokens. Matches or exceeds GPT-3 with 60% fewer parameters — the optimal ratio is ~20 tokens per parameter.
Llama 3 (Meta)
8B/70B/405B variants, 15T tokens. Open-source models with published weights. 37 tokens per parameter — deliberately overtrained for inference efficiency.
GPT-4 (OpenAI)
Estimated ~1.8T parameters (Mixture of Experts — a technique where the model consists of specialized sub-networks). Multimodal — processes text and images. Exact architecture not published.
Concrete comparison: GPT-3 (175B parameters, 300B tokens, undertrained) versus Chinchilla (70B parameters, 1.4T tokens, optimal) — comparable performance at a fraction of the cost.
Deep Dive: The Chinchilla Correction
Misconception
From Text Generator to Assistant
An LLM goes through three training phases. Each phase adds a different capability. The cost and time required drop drastically from phase to phase.
Phase 1: Pre-training. The most expensive phase (millions of dollars). The model reads massive amounts of text and learns general language structures, facts, and relationships. Result: a text completion engine that can continue any sentence — but does not follow instructions.
GPT-1: The Birth of Generative Pre-Training
The foundation of all modern large language models through unsupervised pre-training. On June 11, 2018, Alec Radford and his OpenAI team published the landmark paper 'Improving Language Understanding by Generative Pre-Training.' This work combined the Transformer architecture with unsupervised pre-training for the first time and established a two-stage paradigm: first, generative training on large text corpora, then fine-tuning for specific tasks. With 117 million parameters and training on the BooksCorpus dataset of over 7,000 unpublished books across various genres, GPT-1 demonstrated that transfer learning works for language understanding. The twelve-layer, decoder-only Transformer architecture with masked self-attention set the template for the entire GPT series. This innovation turned the 2017 Transformer architecture into a practical tool for a wide range of NLP tasks and launched the era of large language models.
Phase 2: Instruction Tuning. Supervised fine-tuning on (prompt, response) pairs. The model learns to follow instructions: question leads to helpful answer instead of arbitrary text completion. Relatively cheap and fast.
Phase 3: RLHF (Reinforcement Learning from Human Feedback). Human raters rank different model outputs. A reward model is trained from this and guides further optimization. The model becomes more helpful, safer, and more honest.
Prompt: "Explain quantum computing". Response: Continues the text arbitrarily, possibly a Wikipedia-style paragraph, often drifts off-topic. No structure, no addressing the user.
Prompt: "Explain quantum computing". Response: Structured, helpful explanation addressing the user. Same underlying weights, different behavior through additional training layers.
The Foundation Model concept: a universally pre-trained model becomes the basis for specialized applications. Medicine, law, customer service — via fine-tuning, a general model becomes an expert. You don't train from scratch every time.
LLaMA: Open-Source Foundation Model
The democratization of Large Language Models through open research models. On February 24, 2023, Meta AI released LLaMA (Large Language Model Meta AI) — a collection of foundation models ranging from 7B to 65B parameters, trained exclusively on publicly available data. The landmark paper 'LLaMA: Open and Efficient Foundation Language Models' demonstrated that state-of-the-art performance is achievable without proprietary datasets. LLaMA enabled researchers without access to large infrastructure to study advanced language models. The inference code was released under a GPLv3 license, while model access was granted on a case-by-case basis for academic research. Trained on trillions of tokens and offered in various model sizes, LLaMA addressed diverse hardware requirements. This work catalyzed a wave of open LLM research and inspired numerous follow-up models in the open-source community.
Deep Dive: Open Source vs. Closed Source
Misconception
The next article on Transfer Learning & Fine-Tuning dives deeper into the fine-tuning process and shows how to create a specialized model with just a few hundred examples.
Interactive: Explore the Attention Formula
Click on individual terms of the attention formula to understand their meaning. Notice how Query, Key, and Value work together to weigh relevant information.
Attention — Compact Form
The attention output: a weighted sum of value vectors. For each token, the mechanism computes which other tokens are relevant and combines their information accordingly.
Concrete Example
Given: Attention(Q, K, V) = softmax(QKᵀ/√dₖ) · V
Components: S = QKᵀ/√dₖ (Scoring Function) , A = softmax(S) · V (Aggregation)
Key Takeaways
Quiz: Large Language Models
Comprehension Check
- Why does pre-training not require manually labeled data, and where do the training answers come from instead?
- How did the Chinchilla paper change the understanding of scaling laws compared to Kaplan et al.?
- What is the exact difference between a base model like GPT-3 and an interactive chat model like ChatGPT, and which training step is responsible?