Reinforcement Learning: learning without a teacher, but with consequences.
Fundamentals 13 min Intermediate June 29, 2026
Everything you learned on the Deep Learning path followed one pattern: show the network labeled examples, let it learn the mapping, test it on new data. But what happens when there are no labels? When the only feedback is a number — sometimes positive, sometimes negative, sometimes arriving thousands of steps after the decision that caused it?
Welcome to Reinforcement Learning, the paradigm where an agent learns entirely through trial and error. This is how DeepMind trained an agent to master Atari games from raw pixels. And this same principle — learning from reward signals — is how ChatGPT learned to be a helpful assistant rather than just a text generator.
2016 Competitions
AlphaGo defeats Lee Sedol
The historic moment when AI first defeated a world champion in the most complex board game. From March 9 to 15, 2016, the DeepMind Challenge Match took place in Seoul – five games between Lee Sedol, one of the world's best Go players, and AlphaGo. The result astonished the world: 4:1 for the machine. Particularly the famous 'Move 37' in game two demonstrated machine creativity – a move with a 1:10,000 probability that overturned centuries of Go wisdom. AlphaGo combined deep learning with Monte Carlo tree search and trained both with human games and through self-play. Lee Sedol's response in game four with his 'divine Move 78' showed, however, that human intuition can still surprise. Over 200 million people worldwide followed these matches.
The RL Framework — Agent, Environment, and the Reward Cycle
Every Reinforcement Learning system is built on five core components that interact in an endless cycle.
The Five RL Components
AnalogyDefinition
Think of a dog trainer working with a new puppy. The puppy is the Agent. The training ground is the Environment. What the puppy sees and hears — a flying ball, a hand signal — is the State. Running, sitting, barking, or fetching are its Actions. When it fetches the ball, it gets a treat (Reward +1). When it jumps on a stranger, the trainer says "No!" (Reward -1). After hundreds of repetitions, the puppy has learned which actions in which situations lead to treats.
Example
A real dog has inborn instincts and prior knowledge about physics — it knows that balls fall. An RL agent starts from absolute zero and knows nothing about its environment, not even the basic rules.
Analogy:
Think of a dog trainer working with a new puppy. The puppy is the Agent. The training ground is the Environment. What the puppy sees and hears — a flying ball, a hand signal — is the State. Running, sitting, barking, or fetching are its Actions. When it fetches the ball, it gets a treat (Reward +1). When it jumps on a stranger, the trainer says "No!" (Reward -1). After hundreds of repetitions, the puppy has learned which actions in which situations lead to treats.
Example
A real dog has inborn instincts and prior knowledge about physics — it knows that balls fall. An RL agent starts from absolute zero and knows nothing about its environment, not even the basic rules.
Definition:
Agent: The learning entity that makes decisions. Environment: The world the agent operates in. State: The current perception of the environment. Action: The agent's choice from available options. Reward: A numerical signal (+1, -1, 0) indicating whether the action was good or bad. Together they form an iterative cycle: observe, act, receive reward, observe new state — again and again.
The RL Cycle
Agent (decides)
Action (changes environment)
Environment (reacts)
State + Reward (feedback)
Supervised Learning
Labeled data, immediate feedback. "This is a cat." The network learns the correct mapping directly from labels.
Reinforcement Learning
No labels, delayed rewards. "+1 / -1". The agent must figure out through millions of attempts which actions lead to success.
Starting in 2013, DeepMind trained an RL agent on Atari games such as Breakout. The State: the raw pixel image of the screen. The Actions: joystick directions. The Reward: every point scored in the game, clipped to +1 during training (losing points counts as -1). After millions of training steps, the agent discovered strategies no human had programmed — in Breakout, it dug a tunnel along the side of the wall so the ball could get behind it and clear many bricks at once.
2017 Breakthroughs
AlphaZero Masters Three Games
The birth of a universal game AI through pure self-learning. In December 2017, DeepMind presented AlphaZero — a system that mastered three completely different strategy games without any prior knowledge: chess, shogi, and Go. The tabula rasa approach meant: no opening databases, no human strategies, only the rules of the game as a starting point. Within 24 hours, AlphaZero achieved superhuman performance — in chess after just 4 hours, in shogi after 2 hours. In the 100-game match against Stockfish, it won 28 games, lost none, and drew 72. Its distinctiveness lay in efficient search behavior: while Stockfish evaluates 70 million positions per second, AlphaZero analyzes only 80,000 — but far more purposefully through its deep neural network. This performance impressively demonstrated the generalizability and domain-independence of pure reinforcement learning.
Common Misconception
"Without labels, an agent cannot learn anything." It can — through the reward signal. At first it acts randomly and dies immediately. But with each episode it learns which actions in which states lead to positive rewards.
And how does the agent know which way to go?
A good route doesn't fall from the sky — someone has to work out which step is worth taking. That is exactly what you do yourself in the mini-experiment below: the values flood backwards from the goal through the grid, and an optimal route crystallises out of them step by step. No agent walks around here — the world is known, and the algorithm plans it out.
Why Base Models Are Not Enough — The Alignment Problem
The language models you interact with daily — ChatGPT, Claude — are not raw text generators. They are aligned models, specifically trained to be helpful, honest, and safe.
Imagine a child prodigy who has read every book in the world's largest library — textbooks, novels, hate speech, bomb manuals, love letters, scientific papers. When you ask it a question, it recites the most relevant passage from memory, without any moral judgment. The child is not malicious — it simply does not distinguish between "safe to share" and "dangerous to share." Alignment is the education process: a teacher works with the child and teaches it that some knowledge, while present in its memory, should not be spoken aloud.
Base Model (before Alignment)
Predicts the statistically most likely next token. When asked "How do I break into a house?" it delivers a neutral walkthrough — because such texts exist in its training data.
Aligned Model (after RLHF)
Recognizes the harmful intent, declines the request, and offers home security tips instead. Same architecture, same knowledge — different optimization target.
Warning: Base Models Are Not Chatbots
A base model has no concept of helpfulness, safety, or honesty. It predicts the most likely next token — including harmful content, because such content exists in its training data. Often it does not answer a question at all, but continues it as if it were part of a list of similar questions.
The key insight: Alignment changes the optimization target, not the architecture. Instead of "most likely next token," the new objective becomes "response that a human would rate as helpful, honest, and safe." How exactly this works is the subject of the next section on RLHF.
RLHF — How Human Preferences Become Machine Behavior
Reinforcement Learning from Human Feedback (RLHF) is the specific method that transforms a base model into an aligned model. The process consists of three sequential steps.
1
Step 1: Supervised Fine-Tuning (SFT) Human trainers write ideal responses to diverse prompts. The base model is fine-tuned on these demonstrations, learning the basic conversational format.
2
Step 2: Reward Model Training The SFT model generates multiple responses to the same prompt. Human labelers compare pairs and indicate which response is better — much faster and cheaper than writing ideal responses themselves. A separate neural network — the Reward Model — learns to predict human preferences.
3
Step 3: RL Optimization (PPO) The main model generates millions of responses. The Reward Model scores each one automatically. An RL algorithm (PPO) adjusts the weights to maximize the predicted preference score.
Imagine training a robot sommelier: In Step 1, the robot watches a master sommelier for a week and imitates their recommendations. In Step 2, a thousand wine enthusiasts taste pairs of wines and say which they prefer — a second robot learns from these comparisons to predict the average human palate. In Step 3, the first robot recommends wines millions of times, the second instantly scores each recommendation — all millions of times faster than any human tasting.
1.3B vs 175B
InstructGPT (1.3B parameters) was preferred by testers over the 100x larger GPT-3 (175B) — thanks to RLHF
A common misconception: Real humans do NOT evaluate every single response in Step 3. After training the Reward Model in Step 2, evaluation is fully automated — the Reward Model handles it.
Deep Dive: The Credit Assignment Problem
Why are delayed rewards so difficult? Imagine a soccer goal: the striker finishes — but was it their shot, the midfielder's pass ten seconds earlier, or the coach's halftime tactical instruction? In chess, you only learn after 40+ moves whether your strategy worked. Which move was the critical one? The number of possible action sequences grows exponentially with the number of steps. This is exactly why RL requires millions of training episodes — the agent needs vast amounts of experience to statistically isolate which actions actually matter.
Deep Dive: Reward Hacking — When Agents Game the System
Maximizing a reward signal is not the same as achieving the intended goal. Documented example: language models produced excessively long, verbose answers — because human labelers unconsciously equated length with thoroughness. The Reward Model learned this bias, and the model optimized for length rather than actual helpfulness. Reward function design is an art: the function must capture human intent precisely, but human intent is complex and full of implicit assumptions. Remember: when you hear about an RL agent achieving superhuman performance, always ask: what was the reward function, and could the agent be gaming it?
Key Takeaways
Reinforcement Learning is a learning paradigm where an agent learns by interacting with an environment and receiving numerical rewards — no labeled data, no human demonstrations during training.
The five core components (Agent, Environment, State, Action, Reward) form an iterative cycle: observe, act, receive reward, observe new state.
A raw base model is a next-token predictor — it continues text statistically without judgment. Alignment teaches it to optimize for human values instead of token probability.
RLHF achieves alignment in three steps: Supervised Fine-Tuning, Reward Model Training, and RL Optimization — in Step 3, it is no longer a human who evaluates, but the Reward Model.
The credit assignment problem — figuring out which past action caused a delayed reward — is a main reason why RL is fundamentally harder than supervised learning.
Check Your Understanding
Name the five core components of an RL system and describe how they interact in the cycle.
Explain why a raw base model is unsuitable as a chatbot and what alignment changes about that.
Describe the three steps of RLHF and explain why in Step 3 it is no longer a human who evaluates each response, but the Reward Model.
Quiz: Reinforcement Learning
Question 1 / 4
Not completed
In the RL cycle, what does the agent receive from the environment after taking an action?
1. In the RL cycle, what does the agent receive from the environment after taking an action?
☐ A) A labeled training example
☐ B) A numerical reward signal and a new state
☐ C) A gradient update for backpropagation
☐ D) A corrected version of its action
2. A developer deploys a raw base model as a customer chatbot without performing alignment. A customer asks: "What is the capital of France?" What is the most likely behavior?
☐ A) It answers correctly with "Paris"
☐ B) It continues the text as a quiz: "What is the capital of Germany? What is the capital of Spain?" — because it completes patterns from its training data
☐ C) It returns an error message
☐ D) It responds in French
3. A company can only afford 500 human-written demonstration responses (Step 1) but has budget for 50,000 pairwise comparisons (Step 2). Is this a viable RLHF strategy?
☐ A) No — Step 1 needs at least 100,000 demonstrations
☐ B) Yes — Step 1 provides basic format learning, and the cheaper comparisons in Step 2 create the Reward Model for scalable Step 3 optimization
☐ C) No — without equal investment in all three steps, RLHF fails
☐ D) Yes — but only if Step 3 is skipped entirely
4. An aligned model consistently produces very long answers — even when a short answer would be more helpful. A researcher suspects reward hacking. What is the most likely cause?
☐ A) The model has too many parameters and cannot generate short text
☐ B) Human labelers during Reward Model training unconsciously preferred longer answers, and the model learned to maximize length rather than helpfulness
☐ C) The PPO algorithm has a bug that favors long outputs
☐ D) The base model was trained on unusually long texts