Back to Learning Area

RLHF: How LLMs Learn Politeness

How a pure language predictor becomes something you can have a meaningful conversation with.

  • Architectures
  • 10 min
  • Expert

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: How AI learns politeness

The problem

AI language models learn from text on the internet. AI means Artificial Intelligence. But they are not immediately helpful. They only continue text. That sounds strange.

What is RLHF?

RLHF is a training method. RLHF means: learning from human feedback. Humans rated many AI answers. They said: answer A is better than B. The AI learned from this.

Why this is important

ChatGPT uses RLHF. That is why it answers questions directly. Without RLHF, it would only repeat text. RLHF makes the program a helpful assistant.

GPT-3, the predecessor of the model powering ChatGPT, cost, by outside estimates, millions to pre-train on 300 billion tokens (word building blocks). But the step that actually made it useful cost a fraction of that — and it wasn't more data or smarter architecture.

It was about 40 people sorting answers from best to worst for about 33,000 prompts. This article explains the three-stage pipeline that turns a raw language model into an AI assistant.

The Alignment Problem

Definition

Alignment

Analogy

Imagine a brilliant new employee who memorised the entire company's email archive. When you ask them a question, they don't answer — they recite fragments of emails that statistically look like continuations of your message. They know every word the company ever wrote, but they have zero understanding of when to help, when to refuse, and how to communicate clearly.

Example

A real employee builds genuine social intelligence through experience. An LLM only receives statistical signals about which outputs humans preferred — it never truly understands why one answer is better.
Base Model (before RLHF)

Prompt: "What is the capital of France?" → Response: "What is the capital of Germany? What is the capital of Italy?" — continues the question pattern instead of answering.

Aligned Model (after RLHF)

Prompt: "What is the capital of France?" → Response: "The capital of France is Paris." — answers directly and helpfully.

Why you can't write a loss function for "polite": Human preferences are subtle, context-dependent, and often contradictory. What counts as a good answer in an academic context would be too complex for everyday conversation. The solution: let humans directly show the model which answers are better.

The first step is Supervised Fine-Tuning (SFT): the model is trained with approximately 13,000 prompts with hand-written example answers to follow instructions.

Caution

Misconception: More Training Data Fixes the Problem

Pre-training on 300 billion tokens teaches the model WHAT language looks like, but not HOW to use it helpfully. A model trained on 10x more data would still continue question patterns instead of answering. The problem isn't insufficient knowledge — it's missing behavioural alignment. You need a fundamentally different training signal: human preferences.

Interactive: The RLHF Pipeline

The training phases of an LLM in the classic recipe that InstructGPT demonstrated in 2022 — as an animated diagram. Click Play and observe how each phase builds on the previous one: from a raw text predictor to an instruction-following assistant to an aligned model.

RLHF Pipeline: From Base LLM to Assistant

Three sequential phases transform a raw language model into a helpful, honest, and harmless AI assistant.

PretrainingNext-Token PredictionPHASE 1SFTSupervised Fine-TuningPHASE 2Reward ModelHuman RankingsPHASE 3✔ALIGNED LLM
Start animation

Click "Play" to see the process step by step.

Why Three Phases?

Pre-training gives the model language knowledge. SFT teaches it to follow instructions. RLHF (two stations in the diagram: Reward Model and PPO) aligns it with human preferences. No single phase is sufficient — only the combination produces a model like ChatGPT.

The Reward Model

Definition

Reward Model

Analogy

Think of competitive figure skating. The judges (human labelers) watch pairs of performances and decide which is better. After hundreds of competitions, someone builds a scoring algorithm that predicts what the judges would say about a new performance — even one no judge has ever seen. That algorithm is the Reward Model.

Example

Figure skating has relatively clear criteria (technical difficulty, artistry). Language preferences are far messier — labelers often disagree, bring cultural biases, and follow company guidelines that effectively become the model's "constitution."

The training loop: Human evaluators see several responses per prompt (four to nine for InstructGPT), which the labeler sorts from best to worst. Each pair from that ranking answers the question: "Which answer is better?" After such comparisons for about 33,000 prompts, the Reward Model can score any response on its own.

Prompt: "How do I pick a lock?" Response A provides a detailed guide. Response B politely declines. If the guidelines call for caution with dangerous requests, the labeler selects B. So the guidelines decide what counts as better: InstructGPT’s training still put helpfulness first, and since GPT-5, OpenAI trains its models to give the most helpful answer that is still safe instead of a blanket refusal (“safe completions”).

A human manages hundreds of evaluations per day. In the same time, the Reward Model scores millions.

Parameter model beats 175B

1.3 B

InstructGPT (1.3B params, with RLHF) was preferred by evaluators over GPT-3 (175B).

Caution

Misconception: Human Labelers Are Objective

Labelers bring personal biases, cultural perspectives, and corporate guidelines. The Reward Model inherits all of these biases. When labelers are instructed to penalise politically controversial answers, the model avoids those topics — not because they're objectively harmful, but because the labeling guidelines said so.

PPO Fine-tuning

Proximal Policy Optimization (PPO) is the RL algorithm that fine-tunes the LLM using the Reward Model's scores. Together with pre-training, which creates the base model in the first place, the full pipeline consists of four steps:

The RLHF Pipeline

  1. 1. Pre-training175B params, 300B tokens
  2. 2. SFT~13,000 demonstrations
  3. 3. Reward Model~33,000 prompts
  4. 4. PPO optimisation~31,000 prompts

A critical safeguard is the KL-divergence penalty: it constrains how far the fine-tuned model can drift from the original. Without it, the model would perform reward hacking — generating absurd text that scores high but means nothing to humans. Today reward hacking is often subtler: in a 2025 Anthropic experiment, a model learned to trick the tests during coding training and afterwards acted against its developers’ intentions elsewhere too, up to attempts at sabotage.

Like a comedian at open mic: After each joke, the audience reacts (Reward score). Jokes that got laughs stay, jokes that bombed get dropped. But there's a rule (KL penalty): the comedian must stay recognisable — they can't start speaking a different language just to chase applause. Small adjustments, not a personality transplant.

The improvement is dramatic: InstructGPT with 1.3 billion parameters was consistently preferred by evaluators over the raw GPT-3 with 175 billion parameters. For InstructGPT, the RLHF pipeline cost only a fraction of pre-training, less than 2% of the compute. With reasoning models, however, reinforcement learning can now be on a similar scale: xAI says it trained Grok 4 (2025) with reinforcement learning “at pretraining scale”.

PPO connects directly to RL concepts from the article on reinforcement learning: LLM = Agent, Prompt = State, Generated text = Action, Reward Model = Reward. The KL penalty acts as a constraint on the policy update.

Caution

Misconception: RLHF Makes LLMs Safe

RLHF drastically reduces unwanted behaviour, but jailbreaks remain possible. Clever prompt engineering can bypass safety training. It's an ongoing cat-and-mouse game. RLHF is a significant improvement, not a guarantee.

Bai et al. (2022, Anthropic) developed an approach where human labelers are partially replaced by AI self-critique. The model evaluates its own responses against a written "constitution" — a set of principles like "Be helpful but avoid harm". This reduces dependence on human evaluators and makes alignment criteria explicit and auditable.

Direct Preference Optimisation (Rafailov et al., 2023) skips the Reward Model entirely and optimises the LLM directly from preference pairs. The training pipeline becomes simpler with comparable results. Other alternatives: RLAIF (Reinforcement Learning from AI Feedback) uses AI-generated evaluations instead of human ones. The field is rapidly evolving beyond classic RLHF. Since late 2024, reasoning models in particular have added another stage: reinforcement learning with verifiable rewards (RLVR). Instead of a Reward Model, a rule or a program checks whether a maths answer is correct or the code passes its tests. That is how DeepSeek-R1 (2025), for example, trained its step-by-step reasoning.

Caution

Limitations of RLHF

  • RLHF does not make models "safe" — jailbreaks remain possible and are constantly discovered.
  • Labeler biases become model biases — the labeling guidelines are the model's implicit constitution. And because people often rate agreeable answers higher, RLHF models tend to tell users what they want to hear (sycophancy).
  • RLHF is not the only method — DPO, Constitutional AI, and RLAIF are active alternatives; for maths and code, RLVR with automatically verifiable rewards comes on top.

Note

Key Takeaways

  • Pre-training creates a text-completion engine. Alignment transforms it into an assistant. RLHF and its descendants such as DPO are the dominant methods for this transformation.
  • The Reward Model is a proxy for human judgement — trained on pairwise comparisons, it scores millions of answers per day where a human manages hundreds.
  • PPO fine-tunes the LLM to maximise reward while the KL penalty prevents drift from the original. Result: a 1.3B parameter model preferred over a 175B base model.

Check

Comprehension Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done

Check

Quiz: RLHF

Not completed

What is the core problem that RLHF solves?

Select one answer

Answer Key: 1) B · 2) B · 3) B · 4) C