RLHF: How LLMs Learn Politeness

How a pure language predictor becomes something you can have a meaningful conversation with.

Architectures 10 min Expert June 29, 2026

GPT-3, the predecessor of the model powering ChatGPT, cost millions to pre-train on 300 billion tokens (word building blocks). But the step that actually made it useful cost a fraction of that — and it wasn't more data or smarter architecture.

It was 40 people sorting answers from best to worst for about 33,000 prompts. This article explains the three-stage pipeline that turns a raw language model into an AI assistant.

The Alignment Problem

Alignment

AnalogyDefinition
Imagine a brilliant new employee who memorised the entire company's email archive. When you ask them a question, they don't answer — they recite fragments of emails that statistically look like continuations of your message. They know every word the company ever wrote, but they have zero understanding of when to help, when to refuse, and how to communicate clearly.

Example

A real employee builds genuine social intelligence through experience. An LLM only receives statistical signals about which outputs humans preferred — it never truly understands why one answer is better.

Analogy:

Imagine a brilliant new employee who memorised the entire company's email archive. When you ask them a question, they don't answer — they recite fragments of emails that statistically look like continuations of your message. They know every word the company ever wrote, but they have zero understanding of when to help, when to refuse, and how to communicate clearly.

Example

A real employee builds genuine social intelligence through experience. An LLM only receives statistical signals about which outputs humans preferred — it never truly understands why one answer is better.

Definition:

Alignment is the process of calibrating a model's behaviour to match human values and intentions. The three criteria: Helpful, Honest, Harmless (HHH).
Base Model (before RLHF)

Prompt: "What is the capital of France?" → Response: "What is the capital of Germany? What is the capital of Italy?" — continues the question pattern instead of answering.

Aligned Model (after RLHF)

Prompt: "What is the capital of France?" → Response: "The capital of France is Paris." — answers directly and helpfully.

Why you can't write a loss function for "polite": Human preferences are subtle, context-dependent, and often contradictory. What counts as a good answer in an academic context would be too complex for everyday conversation. The solution: let humans directly show the model which answers are better.

The first step is Supervised Fine-Tuning (SFT): the model is trained with approximately 13,000 prompts with hand-written example answers to follow instructions.

Misconception: More Training Data Fixes the Problem

Pre-training on 300 billion tokens teaches the model WHAT language looks like, but not HOW to use it helpfully. A model trained on 10x more data would still continue question patterns instead of answering. The problem isn't insufficient knowledge — it's missing behavioural alignment. You need a fundamentally different training signal: human preferences.

Interactive: The RLHF Pipeline

The three training phases of a modern LLM — as an animated diagram. Click Play and observe how each phase builds on the previous one: from a raw text predictor to an instruction-following assistant to an aligned model.

RLHF Pipeline: From Base LLM to Assistant

Three sequential phases transform a raw language model into a helpful, honest, and harmless AI assistant.

PretrainingNext-Token PredictionPHASE 1SFTSupervised Fine-TuningPHASE 2Reward ModelHuman RankingsPHASE 3✔ALIGNED LLM
Start animation

Click "Play" to see the process step by step.

Why Three Phases?

Pretraining gives the model language knowledge. SFT teaches it to follow instructions. RLHF aligns it with human preferences. No single phase is sufficient — only the combination produces a model like ChatGPT.

The Reward Model

Reward Model

AnalogyDefinition
Think of competitive figure skating. The judges (human labelers) watch pairs of performances and decide which is better. After hundreds of competitions, someone builds a scoring algorithm that predicts what the judges would say about a new performance — even one no judge has ever seen. That algorithm is the Reward Model.

Example

Figure skating has relatively clear criteria (technical difficulty, artistry). Language preferences are far messier — labelers often disagree, bring cultural biases, and follow company guidelines that effectively become the model's "constitution."

Analogy:

Think of competitive figure skating. The judges (human labelers) watch pairs of performances and decide which is better. After hundreds of competitions, someone builds a scoring algorithm that predicts what the judges would say about a new performance — even one no judge has ever seen. That algorithm is the Reward Model.

Example

Figure skating has relatively clear criteria (technical difficulty, artistry). Language preferences are far messier — labelers often disagree, bring cultural biases, and follow company guidelines that effectively become the model's "constitution."

Definition:

A separate neural network trained on pairwise human preference comparisons. It scores any model response numerically — serving as a scalable proxy for human judgement.

The training loop: Human evaluators systematically see response pairs. The labeler decides: "Which answer is better?" After such comparisons for about 33,000 prompts, the Reward Model can score any response on its own.

Prompt: "How do I pick a lock?" Response A provides a detailed guide. Response B politely declines. The labeler selects B — for dangerous requests, safety outweighs helpfulness.

A human manages hundreds of evaluations per day. In the same time, the Reward Model scores millions.

1.3B
Parameter model beats 175B InstructGPT (1.3B params, with RLHF) was preferred by evaluators over GPT-3 (175B).

Misconception: Human Labelers Are Objective

Labelers bring personal biases, cultural perspectives, and corporate guidelines. The Reward Model inherits all of these biases. When labelers are instructed to penalise politically controversial answers, the model avoids those topics — not because they're objectively harmful, but because the labeling guidelines said so.

PPO Fine-tuning

Proximal Policy Optimization (PPO) is the RL algorithm that fine-tunes the LLM using the Reward Model's scores. Together with pre-training, which creates the base model in the first place, the full pipeline consists of four steps:

The RLHF Pipeline

1
Pre-training (175B params, 300B tokens)
2
SFT (~13,000 demonstrations)
3
Reward Model (~33,000 prompts)
4
PPO optimisation (~31,000 prompts)

A critical safeguard is the KL-divergence penalty: it constrains how far the fine-tuned model can drift from the original. Without it, the model would perform reward hacking — generating absurd text that scores high but means nothing to humans.

Like a comedian at open mic: After each joke, the audience reacts (Reward score). Jokes that got laughs stay, jokes that bombed get dropped. But there's a rule (KL penalty): the comedian must stay recognisable — they can't start speaking a different language just to chase applause. Small adjustments, not a personality transplant.

The improvement is dramatic: InstructGPT with 1.3 billion parameters was consistently preferred by evaluators over the raw GPT-3 with 175 billion parameters. The RLHF pipeline costs a fraction of pre-training.

PPO connects directly to RL concepts from the article on reinforcement learning: LLM = Agent, Prompt = State, Generated text = Action, Reward Model = Reward. The KL penalty acts as a constraint on the policy update.

Misconception: RLHF Makes LLMs Safe

RLHF drastically reduces unwanted behaviour, but jailbreaks remain possible. Clever prompt engineering can bypass safety training. It's an ongoing cat-and-mouse game. RLHF is a significant improvement, not a guarantee.

Deep Dive: Constitutional AI

Bai et al. (2022, Anthropic) developed an approach where human labelers are partially replaced by AI self-critique. The model evaluates its own responses against a written "constitution" — a set of principles like "Be helpful but avoid harm". This reduces dependence on human evaluators and makes alignment criteria explicit and auditable.

Deep Dive: DPO and Alternatives

Direct Preference Optimisation (Rafailov et al., 2023) skips the Reward Model entirely and optimises the LLM directly from preference pairs. The training pipeline becomes simpler with comparable results. Other alternatives: RLAIF (Reinforcement Learning from AI Feedback) uses AI-generated evaluations instead of human ones. The field is rapidly evolving beyond classic RLHF.

Limitations of RLHF

  • RLHF does not make models "safe" — jailbreaks remain possible and are constantly discovered.
  • Labeler biases become model biases — the labeling guidelines are the model's implicit constitution.
  • RLHF is not the only method — DPO, Constitutional AI, and RLAIF are active alternatives.

Key Takeaways

  • Pre-training creates a text-completion engine. Alignment transforms it into an assistant. RLHF is the dominant method for this transformation.
  • The Reward Model is a proxy for human judgement — trained on pairwise comparisons, it scores millions of answers per day where a human manages hundreds.
  • PPO fine-tunes the LLM to maximise reward while the KL penalty prevents drift from the original. Result: a 1.3B parameter model preferred over a 175B base model.

Comprehension Check

  • Why does a pre-trained language model continue question patterns instead of answering questions helpfully?
  • How does the Reward Model solve the problem that human preferences can't be written as a mathematical loss function?
  • What role does the KL-divergence penalty play during PPO, and what unwanted behaviour does it prevent?

Quiz: RLHF

Question 1 / 4
Not completed

What is the core problem that RLHF solves?

Select one answer

1. What is the core problem that RLHF solves?

  • ☐ A) LLMs are too slow at generating text.
  • ☐ B) Pre-trained LLMs predict the next token but don't know how to answer questions helpfully.
  • ☐ C) LLMs cannot process long input texts.
  • ☐ D) Pre-trained LLMs produce grammatically incorrect output.

2. A pre-trained base model receives the prompt "Explain quantum mechanics." Response A is technically perfect but three pages of dense equations. Response B uses clear analogies in two paragraphs. Which would the model more likely generate after RLHF?

  • ☐ A) Response A, because RLHF prioritises technical correctness.
  • ☐ B) Response B, because the Reward Model learned from human preferences that clarity beats completeness for general queries.
  • ☐ C) Neither — RLHF doesn't affect response style, only safety.
  • ☐ D) Both equally, because RLHF only filters harmful content.

3. During PPO fine-tuning, the KL-divergence penalty is removed. What is the most likely consequence?

  • ☐ A) The model generates higher-quality responses with better grammar.
  • ☐ B) The model starts producing nonsensical text that scores high with the Reward Model but is useless to humans (reward hacking).
  • ☐ C) Training becomes faster because there are fewer constraints.
  • ☐ D) The model refuses to answer any questions as a safety measure.

4. InstructGPT (1.3B params, with RLHF) was preferred by humans over GPT-3 (175B params). What does this reveal about model size vs. alignment?

  • ☐ A) Smaller models are always better than larger ones.
  • ☐ B) RLHF can compensate for any model size difference.
  • ☐ C) Behavioural alignment through RLHF can be more impactful than raw scale — a well-aligned small model outperforms an unaligned large one.
  • ☐ D) The 175B model was defective and would have performed better with more data.
Answer Key: 1) B · 2) B · 3) B · 4) C

Article

Article Intermediate
Reinforcement Learning

Reinforcement Learning: learning without a teacher, but with consequences.

13min
Article Advanced
Large Language Models

The model that basically only predicts the next word — and makes a remarkable career doing it.

12min
Article Intermediate
Tokenization: The Machine Alphabet

How letters become tokens, and why LLMs see the world through this lens.

12min
Article Intermediate
Developing Strategies (Q-Learning & MDPs)

Strategies learned through trial and error — the formal version of what every child does.

10min
Article Beginner
Consciousness vs. Probability

The philosophical question of whether LLMs understand what they say — sorted without arrogance.

9min
Article Intermediate
Control & Finetuning

Three interventions that turn an AI from slot-machine mode into a real creative tool.

10min
Article Intermediate
Generative Image Models (Diffusion)

How an algorithm learned to generate dogs from snowfall — with the patience of a monk.

10min
Article Intermediate
Agents in Conflict — Game Theory

What a second rational player changes about an optimization — everything.

10min
Article Beginner
The Ecosystem: Hugging Face

Where the AI world stores its models — a tour through the largest repository.

12min
Article Beginner
Learning with AI (Instead of Cheating)

How to use AI as a learning partner instead of a cheat sheet with internet access.

8min
Article Beginner
Security & Prompt Injections

The uncomfortable truth: language models cannot reliably tell apart what is a command and what is input.

10min
Article Intermediate
Sampling & Temperature

The mini-roulette after every token — and the dials you can turn.

10min
Article Beginner
Supervised Learning — Learning with a Teacher

Supervised Learning: the ML paradigm where someone diligently labeled things beforehand.

10min
Article Beginner
Giving the Model a Task

The instruction the model reads before everything else — and mostly follows.

7min
Article Intermediate
Transfer Learning & Fine-Tuning

How to win over an AI model instead of raising it from scratch — because training from zero is prohibitively expensive.

10min
Article Advanced
The Breakthrough — Transformers & Attention

What "Attention Is All You Need" (2017) actually changed — and why the world hasn't gone back.

12min
Article Beginner
Prompting Fundamentals

The art of telling a language model what you want — surprisingly similar to human communication.

25min

Demo

Demo Intermediate
Multi-Armed Bandit

Compare ε-greedy, UCB1, and Thompson Sampling on slot machines with hidden payoff probabilities.

20min
Demo Intermediate
Q-Learning

Interactive demonstration of the Q-Learning algorithm with an intelligent agent in the Temple of Learning

30min
Demo Beginner
Supervised Learning

Join Sharlock Helmes in his cleverest case: learning to distinguish between genuine clues and Moriarty's sophisticated red herrings. Elementary, my dear algorithm!

25min

External

External External
InstructGPT Paper (Ouyang et al., 2022)

The paper describing the RLHF pipeline for GPT

Open external source

Glossary

Reinforcement Learning from Human Feedback (RLHF)ModelAlignmentPPOSupervised Fine-Tuning (SFT)Large Language Models (LLMs)ParameterPromptGPTPolicyPre-TrainingOptimizationReward ModelRLAIFReinforcement Learning (RL)Reward HackingSycophancy

Timeline

ChatGPT Marks a Turning Point in AI AdoptionClaude and Constitutional AIConstitutional AI — Safety Through PrinciplesGPT-3: The 175-Billion-Parameter ModelRLHF research paper published