9.2 RLHF

DATA 443/643 · Week 9 · PDF · Slides

Three stages. Only PPO samples the current policy. β=0 is reward hacking.

These notes match the lecture slides. Use (slides) on the course hub for the deck.

RLHF (InstructGPT / ChatGPT-style) is a three-stage stack: SFT, a reward model, then a policy optimized with RL (usually PPO) so sampled replies score high under the RM without wandering too far from SFT.

1. What RLHF is

RLHF is three stages, not one loss.

1. SFT on instruction pairs (note 8.1). Call this ${\pi }_{\text{SFT}}$.
2. RM $r_{\varphi }$ on chosen/rejected pairs (note 9.1).
3. PPO: sample $y~{\pi }_{\theta }(·\mid x)$, score with $r_{\varphi }$, update ${\pi }_{\theta }$.

After SFT and an RM, you sample from the current policy, score with $r$, and update to raise reward minus KL to a frozen reference. Only stage 3 needs samples from the current policy. Stages 1–2 use a fixed dataset. If your code loads a CSV of completions and never calls generate inside the PPO loop, it is not on-policy RLHF.

PPO is on-policy: you need fresh rollouts from the current ${\pi }_{\theta }$. That is expensive (generate, reward, critic). The critic (value head) estimates advantages so the policy gradient has lower variance. You do not need to derive PPO in this course; you need the picture and the KL term. InstructGPT is this stack at company scale; a seminar project usually stops at best-of-$n$ or DPO.

2. Why we use it

SFT does not explore. It copies answers that were already in the file. Best-of-$n$ uses $r$ only at decode: the policy itself never learns. RLHF trains $\pi$ to look high-reward as it samples, so the next draw is already better, not merely reranked.

PPO still exists because on-policy sampling can find replies that were never in the pair file. That is the conceptual parent of the week: preferences $\to$ a scalar of “better” $\to$ a policy that maximizes it with a leash. It works at company scale and is finicky in a seminar: sampling, KL, reward scale, and a moving policy. DPO (next note) skips the RM and the loop. Keep the three-box picture even if you never run PPO.

On-policy also means last week’s samples are stale: ${\pi }_{\theta }$ moved, so you generate again. That is the cost people are avoiding with DPO. For a project, running full PPO on an LLM is usually out of scope. Best-of-$n$ with a tiny RM, or DPO on a small classifier, is enough to show you understood the stack.

3. Architecture

Three frozen-or-moving pieces sit together at the PPO stage:

• Policy ${\pi }_{\theta }$ (moving).
• Reference ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$ (usually SFT, frozen).
• RM $r_{\varphi }$ (frozen in the PPO stage).
• Optional critic / value head for advantages.

Why ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}={\pi }_{\mathrm{S}\mathrm{F}\mathrm{T}}$ not the raw pretrained LM: the RM was trained on SFT-style answers. Leashing to the base model pulls you back to web-text, not to the assistant you already paid for. The three boxes are SFT, RM, and RL. Best-of-$n$ is already a strong baseline in AlpacaFarm-style studies; it is not a fourth box in this loop.

4. How it works, step by step

1. Run SFT. Freeze that checkpoint as ${\pi }_{\text{SFT}}$ and as ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$.
2. Train the RM on pairs. Freeze $r_{\varphi }$.
3. PPO loop. Sample a batch of new $y$ from the current ${\pi }_{\theta }$. Score with $r_{\varphi }$. Estimate advantages with the critic. Update the policy with a clipped objective. Subtract a KL penalty toward ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$.
4. Read $\beta$. Large $\beta$ stays near SFT; small $\beta$ chases reward. In code this is often a per-token penalty $\beta \log {\pi }_{\theta }-\beta \log {\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$.
5. If $\beta =0$, there is no leash. Expect reward to rise while KL explodes and the text becomes a caricature of the raters: long, flattering, or degenerate. That is reward hacking, not “learning faster.”

Without a constraint, ${\pi }_{\theta }$ hacks $r_{\varphi }$. The KL term is the method, not a footnote. You do not derive the PPO clip in this course; you need to know that a clip plus an advantage estimate is how the policy step is usually written, and that $\beta =0$ is the failure mode to name.

5. Mathematical formulas

The usual RLHF objective looks like

or, sampling from the current policy,

KL cartoon, two-word vocab $\{a,b\}$. ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}=(0.7,0.3)$, ${\pi }_{\theta }=(0.99,0.01)$:

Penalty $\beta ·0.31$. If $\beta =0.1$, you subtract $0.031$ from reward; if $\beta =0$, you subtract nothing and ${\pi }_{\theta }$ can spike. If ${\pi }_{\theta }={\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$, KL is 0 and the leash is silent at that instant.

a goes 0.7 $\to$ 0.99" src="files/graphics/9.2-rlhf/kl-numeric.png" />

6. Positive points and negative points

Positive.

• Can beat SFT on preference evals because the policy learns while it samples, not only at decode.
• On-policy rollouts can find replies that were never in the pair file.
• The KL leash is an explicit knob between “stay an assistant” and “chase the RM.”
• Best-of-$n$ remains a fair baseline on the same RM; PPO has to beat it.

Negative.

• Unstable and expensive: generate, reward, critic, and stale samples every time ${\pi }_{\theta }$ moves.
• RM misspecification becomes policy misspecification. Reward hacking is what $\beta =0$ looks like.
• Using the pretrained base as ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$ after SFT already moved the policy pulls you back to web-text.
• Drawing RLHF as one loss hides that only stage 3 is RL.
• Full PPO on an LLM is usually out of scope for a seminar project, which is why people look to DPO.

When not to. If you have pairs and a GPU but no appetite for a PPO stack, skip to DPO (next note) or stop at best-of-$n$. If $\beta =0$ “to learn faster,” you are not running RLHF; you are maximizing a misspecified RM.

7. Teaching this note

About 35 minutes at the board: three boxes (SFT, RM, PPO), then write the KL objective and set $\beta =0$ vs large $\beta$. Play the same Karpathy RLHF window 21:05–25:43 (and a minute after if he is still on comparisons). DPO is next; do not derive PPO.

8. Worked example

Stage check: a batch of instruction pairs $\to$ SFT. A batch of $(y_w,y_l)$ $\to$ RM. A batch of new samples from ${\pi }_{\theta }$ $\to$ PPO. If your code loads a CSV of completions and never calls generate inside the PPO loop, it is not on-policy RLHF.

KL cartoon: two distributions on a 2-word vocab $\{a,b\}$. ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}=(0.7,0.3)$, ${\pi }_{\theta }=(0.99,0.01)$.

Penalty $\beta ·0.31$. If $\beta =0.1$, you subtract $0.031$ from reward; if $\beta =0$, you subtract nothing and ${\pi }_{\theta }$ can spike.

The three boxes are SFT, RM, and RL. Best-of-$n$ is already a strong baseline in AlpacaFarm-style studies. PPO still exists because on-policy sampling can find replies that were never in the pair file. InstructGPT is this stack at company scale; a seminar project usually stops at best-of-$n$ or DPO.

Why ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}={\pi }_{\mathrm{S}\mathrm{F}\mathrm{T}}$ not the raw pretrained LM: the RM was trained on SFT-style answers. Leashing to the base model pulls you back to web-text, not to the assistant you already paid for.

9. Where students get stuck

• Drawing RLHF as one loss. It is three stages; only the last is RL.
• Setting $\beta =0$ “to learn faster” and then being surprised by reward hacking.
• Using the pretrained base as ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$ after SFT already moved the policy.

10. Video

Karpathy — Intro to Large Language Models. Play 21:05–25:43 (RLHF, comparison labels). Pause on the three-stage story: write, compare, then a loop that uses those comparisons. This clip will not derive PPO; the board does the KL leash.

11. Practice

1. Which of the three stages needs samples from the current policy, not a fixed dataset?

2. If $\beta =0$, what failure mode should you expect?

3. Why is ${\pi }_{\text{ref}}$ the SFT model rather than the raw pretrained LM?

4. ${\pi }_{\theta }=(0.8,0.2)$, ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}=(0.8,0.2)$. What is $\mathrm{K}\mathrm{L}({\pi }_{\theta }\Vert {\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}})$, and what does that say about the leash at this instant?

5. Reward $r=5$, $\beta =2$, $\mathrm{K}\mathrm{L}=1.5$. What scalar is the objective $r-\beta \mathrm{K}\mathrm{L}$? If you raise $\beta$ to 4, who wins: the RM or the reference?