Week 9: RLHF and direct preference optimization

DATA 443/643 · Week 9

Module 3 — RLHF and direct preference optimization

Notes

• 9.1 Preference Data and Reward Models(slides)
• 9.2 RLHF(slides)
• 9.3 Direct Preference Optimization(slides)

Labs

• Lab 9: Pairwise Preferences and a Toy DPO Loss(PDF)

Videos

• Karpathy: Intro to Large Language Models (RLHF / preferences, 21:05–25:43)
• Karpathy: Deep Dive into LLMs like ChatGPT (preference / RLHF, 2:48:26–3:09:39)

Additional readings

• Stanford CS224N 2025 L10: RM, RLHF, DPO (SFT was Week 8)
• Ouyang et al., InstructGPT (the three-stage stack)
• Rafailov et al., Direct Preference Optimization
• Dubois et al., AlpacaFarm (best-of-n is a strong baseline)

Discussion

• Best-of-n uses the RM and never trains π_θ. When is that enough for a project, and when is it not?
• If r_w = r_l always, what is the RM loss, and what did the head fail to learn?
• Why is π_ref the SFT model rather than the raw pretrained LM?
• DPO skips PPO. What can a fixed pair file still not invent?