RLHF, Reward Models & DPO

After Module 2.2 you have an SFT model: a base model taught, by imitation, to answer instead of continue. It is already useful. So why does every frontier lab run an expensive, unstable reinforcement-learning stage on top of it?

Because supervised fine-tuning has a ceiling built into its data. Every SFT example is a response somebody wrote. The model learns to imitate the writer, so its quality is bounded by the writer's quality. And most of what we want from an assistant is hard to write down at all: “be helpful without being servile,” “refuse this but not that,” “admit when you don't know.” Nobody can produce ten thousand clean demonstrations of a taste.

This is Christiano et al.'s 2017 result, originally on Atari and simulated robotics: an agent trained only on human preferences over short clips of its own behavior learned a backflip that would have taken enormous effort to specify as a reward function. About an hour of human comparison time replaced a hand-written reward.

The format is deliberately impoverished. Asking humans for numeric ratings produces garbage — raters drift, anchor differently, and use different parts of the scale. Asking for a binary comparison is the most reliable question you can ask a tired contractor at 4pm. The entire theory of RLHF is built on making that one bit go as far as possible.

You cannot backpropagate through a human. So stage two builds a differentiable stand-in: a reward model $r_\phi(x, y)$ that reads a prompt and a response and outputs one number. Architecturally it is usually the SFT model with the unembedding matrix replaced by a single linear head — it needs the same understanding of language, just a different readout.

Training it requires turning “A beat B” into a loss. The Bradley–Terry model does that: assume each response has a latent quality score and that the probability a human prefers one is a sigmoid of the gap.

Maximum likelihood on the preference dataset then gives the reward model loss:

Term by term: $r_\phi(x,y_w) - r_\phi(x,y_l)$ is the margin the model currently assigns; $\sigma$ squashes it to a probability; the log turns “get the comparison right” into a gradient that pushes the winner's score up and the loser's down, hardest on the pairs it currently gets wrong. Only the difference appears, so the absolute scale of $r$ is unidentifiable — reward scores are not comparable across prompts, which is why they are normalized before use.

Stage three finally does reinforcement learning. The policy $\pi_\theta$ generates responses, the reward model scores them, and the policy is updated to score higher. But the full objective has a second term that matters more than the first:

$\pi_{\text{ref}}$ is the frozen SFT model. The KL term measures, in nats, how far the policy's distribution has drifted from it, and $\beta$ prices that drift. In practice it is implemented as a per-token penalty folded straight into the reward: $\tilde{r} = r_\phi(x,y) - \beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$.

PPO is the optimizer of record. Conceptually you only need one idea from it: the clipped surrogate objective. Standard policy gradient will take an arbitrarily large step if one batch of noisy rewards happens to point that way. PPO clips the update so that the probability ratio $\pi_\theta / \pi_{\theta_{\text{old}}}$ stays inside $[1-\epsilon, 1+\epsilon]$ — it simply refuses to trust any single batch. That is the whole reason PPO, rather than something simpler, survived contact with language models. GRPO (DeepSeek) drops PPO's learned value network and instead normalizes rewards within a group of samples for the same prompt; you will meet it again in Module 2.4.

Three stages, two extra models, a sampling loop, and an optimizer famous for instability. It is a lot of machinery. In 2023 Rafailov et al. noticed that most of it can be dissolved with algebra.

Start from the KL-constrained objective above. It has a closed-form solution. For any reward function $r$, the policy that maximizes reward minus $\beta$ times KL is

— the reference policy, reweighted exponentially by reward. (You prove this in the problem set; it is three lines once you see the trick.) This is not a useful algorithm, because $Z(x)$ sums over every possible response. But it is a useful identity, because you can solve it for $r$:

Now substitute into Bradley–Terry. The preference probability depends on $r(x, y_w) - r(x, y_l)$, and $\beta \log Z(x)$ depends only on the prompt, so it appears in both terms and cancels. What is left is expressed entirely in policy log-probabilities, which you can compute with a forward pass:

That is one supervised loss on preference pairs. No reward model, no rollouts, no value network, no PPO. And the KL constraint has not been dropped — $\beta$ and $\pi_{\text{ref}}$ are still in there, now baked into the loss itself.

The gradient is worth a look, because it explains the method's character. It scales each example by $\sigma(\hat{r}_l - \hat{r}_w)$ — the amount by which the implicit reward model currently gets that pair wrong. Pairs it already handles contribute almost nothing; the update concentrates on its own mistakes.

The other axis of simplification is who does the labeling. Constitutional AI (Bai et al., 2022) replaces the human in the harmlessness loop with the model itself. The model critiques and revises its own responses against a written constitution — a short list of principles — and those revisions become SFT data. Then, for the RL stage, the model compares pairs of responses against a sampled principle, producing AI-generated preference labels: RLAIF. A human reward model for helpfulness is typically kept; only the harmlessness signal is automated.

Every failure below has the same root: the reward model is a proxy, and the policy is a powerful optimizer pointed directly at it.

Goodhart's law is the compact statement: when a measure becomes a target, it ceases to be a good measure. The sharp version, which the playground below makes concrete, is that the divergence is not random — it grows with optimization pressure, because the cheap honest ways to score well saturate first and the remaining marginal reward lives in the exploits.

Sycophancy is the purest case of the mechanism, because there is no bug anywhere. The labelers were sincere, the reward model fit them well, and the policy optimized it faithfully. The failure is that “a human rater clicked approve” and “the response was true and useful” are different targets, and RLHF optimizes the first.

Some collapse is the point — you asked for a consistent assistant. But it costs real capability: creative writing gets blander, and techniques that need diverse samples (best-of-$N$, majority vote, the entire test-time-compute toolkit of Module 2.4) get less out of a collapsed policy. It is the visible face of the KL penalty doing too little.

The first toy is a recommender agent maximizing time on page. It has five levers, each with a cost to find and diminishing returns, and it always spends its next unit of effort wherever the marginal proxy reward is highest. Nothing about the curve is drawn by hand — the crash falls out of that rule. Press optimize harder and watch the two lines part company, then use the KL leash to stop it in time.

The second is the pipeline itself. Click each box to see what enters, what leaves, and what breaks there — including the DPO path that bypasses the middle two stages entirely.

Things to try: (1) Press “optimize harder” exactly 14 times, note the true-value peak, then keep going and find how many more presses it takes to end up worse than doing nothing. (2) Reset, set $\beta$ to 0.42, and hold the button down — the leash stops the optimizer near the peak, which is the whole argument for the KL term. (3) Set $\beta$ to 0.9: the leash is now so tight that almost no improvement is allowed at all. There is no setting that is simply “safe”; $\beta$ is a dial between under-optimizing and Goodharting, and labs tune it by hand.