RLHF — Reinforcement Learning from Human Feedback — Explained Simply

What RLHF Actually Is

You Are Raising a Child

Your three-year-old is brilliant — talks, walks, reads simple words. Raw intelligence impressive. But they grab food from others' plates, say whatever comes to mind, and confidently make things up when they do not know something.

The intelligence is there. The behaviour is not yet shaped.

You cannot give them a rulebook — too abstract. Instead you teach through feedback. When they do something good — express approval. When they behave badly — express disapproval. Over thousands of these moments their behaviour shapes toward what you consider good.

Not because you gave rules. Because they experienced which behaviours get approved.

That is RLHF.

Why Raw LLMs Behave Badly Without RLHF

Pretraining: predict the next word on internet text

Internet contains:
  ✅ Helpful, accurate information
  ❌ Misinformation, harmful content, rude responses
  ❌ Confident wrong answers

Raw model reproduces all of it equally.
RLHF teaches it to consistently produce helpful, honest, harmless responses.

The Three Stages

Stage 1 — Pretraining

Train on billions of words.
Learns language, facts, reasoning, coding.
Behaviour: uncontrolled — brilliant but unsocialised.
Like the three-year-old before any guidance.

Stage 2 — Supervised Fine-Tuning (SFT)

Human trainers write ideal prompt-response pairs.
Model fine-tunes on thousands of examples.
Learns to imitate good behaviour in seen situations.
Still limited — cannot generalise deeply.

Stage 3a — Collect Preference Data

Same prompt → two different model responses
Human rater: which is better?

Prompt: "Explain quantum computing simply"
Response A: "Quantum bits exist in superposition states..."
Response B: "Like a magic coin that is heads and tails until you look at it..."
Human rater: B is better. Thousands of these comparisons collected.

### Stage 3b — Train a Reward Model

A separate neural network trained on human comparisons. Learns to predict what humans prefer. Now a proxy for human judgment — scalable to millions of responses.

Reward model learns: Clear explanations → high score Confident wrong answers → low score Helpful specific advice → high score Harmful content → very low score

### Stage 3c — Optimise the LLM Against the Reward Model

LLM generates response → reward model scores → LLM adjusts weights Repeat millions of times.

LLM learns: ✅ Be clear and accessible ✅ Admit uncertainty rather than making things up ✅ Actually answer the question asked ✅ Avoid harmful outputs ❌ Do not be verbose for no reason ❌ Do not refuse things unnecessarily

---

## Constitutional AI — Anthropic's Version

Standard RLHF depends on human raters — inconsistent, biased, unscalable. Constitutional AI gives the model principles and has it critique and revise its own outputs.

Anthropic's Constitution includes: "Be helpful, harmless, and honest" "Respect people's autonomy" "Be transparent about uncertainty"

Process:

1. Model generates a response
2. Model critiques it against the constitution
3. Model revises based on its critique
4. Revised responses train the reward model

More scalable. More consistent. Reduces human rater bias. This is what makes Claude specifically different from other LLMs.

---

## Why KL Divergence Appears Here

```python
# Total RLHF objective
objective = reward_score - beta * KL(fine_tuned_model || original_model)
# KL measures drift from original pretraining
# Without it: model finds ways to fool reward model (reward hacking)
# With it: genuine improvement while preserving pretrained knowledge

Conceptual Code

def rlhf_step(prompt, policy_model, reward_model, reference_model, beta=0.1):
    response           = policy_model.generate(prompt)
    reward_score       = reward_model(prompt, response)
    policy_log_prob    = policy_model.log_prob(prompt, response)
    reference_log_prob = reference_model.log_prob(prompt, response)
    kl_penalty         = policy_log_prob - reference_log_prob
    objective          = reward_score - beta * kl_penalty
    loss               = -objective
    loss.backward()
    optimizer.step()

The Real Words Mapped to the Story

The One Thing to Remember

RLHF teaches an LLM to behave well through millions of human preference comparisons rather than explicit rules. A reward model learns what humans prefer. The LLM trains to generate responses the reward model scores highly. Constitutional AI — Anthropic's contribution — teaches the model to critique its own outputs against principles. This is why Claude behaves the way it does.