RLHF: How LLMs Learn Politeness
How a pure language predictor becomes something you can have a meaningful conversation with.
GPT-3, the predecessor of the model powering ChatGPT, cost, by outside estimates, millions to pre-train on 300 billion tokens (word building blocks). But the step that actually made it useful cost a fraction of that — and it wasn't more data or smarter architecture.
It was about 40 people sorting answers from best to worst for about 33,000 prompts. This article explains the three-stage pipeline that turns a raw language model into an AI assistant.
InstructGPT: The Bridge to ChatGPT
Between the method and the global hit lay a crucial intermediate step — and it was called InstructGPT. In early 2022, in the paper Training language models to follow instructions with human feedback, OpenAI showed how to get GPT-3 to actually do what users want: through reinforcement learning from human feedback (RLHF). The striking result: an InstructGPT with only 1.3 billion parameters was preferred by people over the answers of the hundred-times-larger GPT-3 (175 billion). It was not raw size but alignment with intent that made the difference. InstructGPT was the direct technical bridge between the RLHF idea (2017) and ChatGPT, which popularised the same method at the end of 2022. For an honest assessment: InstructGPT did not invent RLHF — a 2017 paper did — but it showed for the first time at scale how much alignment makes a language model more useful.
The Alignment Problem
Definition
Alignment
Analogy
Example
Definition
Prompt: "What is the capital of France?" → Response: "What is the capital of Germany? What is the capital of Italy?" — continues the question pattern instead of answering.
Prompt: "What is the capital of France?" → Response: "The capital of France is Paris." — answers directly and helpfully.
Why you can't write a loss function for "polite": Human preferences are subtle, context-dependent, and often contradictory. What counts as a good answer in an academic context would be too complex for everyday conversation. The solution: let humans directly show the model which answers are better.
The first step is Supervised Fine-Tuning (SFT): the model is trained with approximately 13,000 prompts with hand-written example answers to follow instructions.
Caution
Misconception: More Training Data Fixes the Problem
Pre-training on 300 billion tokens teaches the model WHAT language looks like, but not HOW to use it helpfully. A model trained on 10x more data would still continue question patterns instead of answering. The problem isn't insufficient knowledge — it's missing behavioural alignment. You need a fundamentally different training signal: human preferences.
Interactive: The RLHF Pipeline
The training phases of an LLM in the classic recipe that InstructGPT demonstrated in 2022 — as an animated diagram. Click Play and observe how each phase builds on the previous one: from a raw text predictor to an instruction-following assistant to an aligned model.
Three sequential phases transform a raw language model into a helpful, honest, and harmless AI assistant.
Click "Play" to see the process step by step.
Pre-training gives the model language knowledge. SFT teaches it to follow instructions. RLHF (two stations in the diagram: Reward Model and PPO) aligns it with human preferences. No single phase is sufficient — only the combination produces a model like ChatGPT.
The Reward Model
Definition
Reward Model
Analogy
Example
Definition
The training loop: Human evaluators see several responses per prompt (four to nine for InstructGPT), which the labeler sorts from best to worst. Each pair from that ranking answers the question: "Which answer is better?" After such comparisons for about 33,000 prompts, the Reward Model can score any response on its own.
Prompt: "How do I pick a lock?" Response A provides a detailed guide. Response B politely declines. If the guidelines call for caution with dangerous requests, the labeler selects B. So the guidelines decide what counts as better: InstructGPT’s training still put helpfulness first, and since GPT-5, OpenAI trains its models to give the most helpful answer that is still safe instead of a blanket refusal (“safe completions”).
A human manages hundreds of evaluations per day. In the same time, the Reward Model scores millions.
Parameter model beats 175B
1.3 B
InstructGPT (1.3B params, with RLHF) was preferred by evaluators over GPT-3 (175B).
Caution
Misconception: Human Labelers Are Objective
Labelers bring personal biases, cultural perspectives, and corporate guidelines. The Reward Model inherits all of these biases. When labelers are instructed to penalise politically controversial answers, the model avoids those topics — not because they're objectively harmful, but because the labeling guidelines said so.
PPO Fine-tuning
Proximal Policy Optimization (PPO) is the RL algorithm that fine-tunes the LLM using the Reward Model's scores. Together with pre-training, which creates the base model in the first place, the full pipeline consists of four steps:
The RLHF Pipeline
- 1. Pre-training175B params, 300B tokens
- 2. SFT~13,000 demonstrations
- 3. Reward Model~33,000 prompts
- 4. PPO optimisation~31,000 prompts
A critical safeguard is the KL-divergence penalty: it constrains how far the fine-tuned model can drift from the original. Without it, the model would perform reward hacking — generating absurd text that scores high but means nothing to humans. Today reward hacking is often subtler: in a 2025 Anthropic experiment, a model learned to trick the tests during coding training and afterwards acted against its developers’ intentions elsewhere too, up to attempts at sabotage.
Like a comedian at open mic: After each joke, the audience reacts (Reward score). Jokes that got laughs stay, jokes that bombed get dropped. But there's a rule (KL penalty): the comedian must stay recognisable — they can't start speaking a different language just to chase applause. Small adjustments, not a personality transplant.
The improvement is dramatic: InstructGPT with 1.3 billion parameters was consistently preferred by evaluators over the raw GPT-3 with 175 billion parameters. For InstructGPT, the RLHF pipeline cost only a fraction of pre-training, less than 2% of the compute. With reasoning models, however, reinforcement learning can now be on a similar scale: xAI says it trained Grok 4 (2025) with reinforcement learning “at pretraining scale”.
PPO connects directly to RL concepts from the article on reinforcement learning: LLM = Agent, Prompt = State, Generated text = Action, Reward Model = Reward. The KL penalty acts as a constraint on the policy update.
Caution
Misconception: RLHF Makes LLMs Safe
RLHF drastically reduces unwanted behaviour, but jailbreaks remain possible. Clever prompt engineering can bypass safety training. It's an ongoing cat-and-mouse game. RLHF is a significant improvement, not a guarantee.
Bai et al. (2022, Anthropic) developed an approach where human labelers are partially replaced by AI self-critique. The model evaluates its own responses against a written "constitution" — a set of principles like "Be helpful but avoid harm". This reduces dependence on human evaluators and makes alignment criteria explicit and auditable.
Direct Preference Optimisation (Rafailov et al., 2023) skips the Reward Model entirely and optimises the LLM directly from preference pairs. The training pipeline becomes simpler with comparable results. Other alternatives: RLAIF (Reinforcement Learning from AI Feedback) uses AI-generated evaluations instead of human ones. The field is rapidly evolving beyond classic RLHF. Since late 2024, reasoning models in particular have added another stage: reinforcement learning with verifiable rewards (RLVR). Instead of a Reward Model, a rule or a program checks whether a maths answer is correct or the code passes its tests. That is how DeepSeek-R1 (2025), for example, trained its step-by-step reasoning.
Caution
Limitations of RLHF
- RLHF does not make models "safe" — jailbreaks remain possible and are constantly discovered.
- Labeler biases become model biases — the labeling guidelines are the model's implicit constitution. And because people often rate agreeable answers higher, RLHF models tend to tell users what they want to hear (sycophancy).
- RLHF is not the only method — DPO, Constitutional AI, and RLAIF are active alternatives; for maths and code, RLVR with automatically verifiable rewards comes on top.
Note
Key Takeaways
- Pre-training creates a text-completion engine. Alignment transforms it into an assistant. RLHF and its descendants such as DPO are the dominant methods for this transformation.
- The Reward Model is a proxy for human judgement — trained on pairwise comparisons, it scores millions of answers per day where a human manages hundreds.
- PPO fine-tunes the LLM to maximise reward while the KL penalty prevents drift from the original. Result: a 1.3B parameter model preferred over a 175B base model.
Check
Comprehension Check
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: RLHF
What is the core problem that RLHF solves?
Select one answer
1. What is the core problem that RLHF solves?
2. A pre-trained base model receives the prompt "Explain quantum mechanics." Response A is technically perfect but three pages of dense equations. Response B uses clear analogies in two paragraphs. Which would the model more likely generate after RLHF?
3. During PPO fine-tuning, the KL-divergence penalty is removed. What is the most likely consequence?
4. InstructGPT (1.3B params, with RLHF) was preferred by humans over GPT-3 (175B params). What does this reveal about model size vs. alignment?