38 — Modern / LLM

RLHF — Alignment#

AlignmentMathematical 1 min read
◆ The PatternMaking language models helpful, harmless, and honest with human feedback

RLHF aligns a pretrained LLM with human preferences in three stages: supervised fine-tuning, reward model training, and PPO optimization. DPO simplifies this to a single training step.

Stage 1: SFT — fine-tune on high-quality demonstrations
Supervised fine-tuning on curated instruction-response pairs
Stage 2: Reward Model — R(x,y) trained on human preferences
Human annotators rank outputs; model learns to predict which response humans prefer
Stage 3: PPO — max E[R(x,y)] − β·KL(π||π_ref)
Optimise policy to maximise reward while staying close to SFT model (prevents reward hacking)
DPO: L = −log σ(β(log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)))
DPO = Direct Preference Optimization — no reward model needed. Simpler, more stable.
// RLHF pipeline — three-stage process
DPO vs RLHF: DPO reformulates RLHF as a simple classification loss on preference pairs — no reward model, no PPO, no RL instability. LLaMA 2, Zephyr, and many modern models use DPO.
Pattern bridge: Training a reward model from human preferences connects to RLHF in LLM alignment and preference testing in statistics. In markets, herd behavior is collective preference shaping price — the market’s reward signal. The one-step alternative has its own topic: DPO.
Worked example

A reward model trained on human comparisons follows the Bradley–Terry model: a reward gap of 1 means the preferred answer wins 73% of the time. DPO skips the reward model and trains on the comparisons directly; with β = 0.1 its loss is 0.693 when the model has no preference, 0.313 at a log-ratio margin of 10 and 0.049 at 30.

Where it misleads
  • Optimising hard against a learned reward model eventually exploits its errors (reward over-optimisation, Gao et al. 2023); the KL penalty is what holds it back.
  • Human preferences are noisy and inconsistent; a 73% preference is a weak signal per comparison.
  • Preference training improves what raters reward — which can include confident tone over correctness.
In code
sigmoid = lambda z: 1 / (1 + np.exp(-z))
p_prefer = sigmoid(1.0)                             # Bradley-Terry: reward gap of 1 -> P(chosen beats rejected)
beta = 0.1
dpo_loss = lambda margin: -np.log(sigmoid(beta * margin))   # margin = difference of log-ratios vs the reference
losses = {m: round(float(dpo_loss(m)), 3) for m in (0, 10, 30)}
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.

Sources
  • P. F. Christiano et al., “Deep Reinforcement Learning from Human Preferences”, NeurIPS, 2017
  • L. Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback”, NeurIPS, 2022
  • R. Rafailov et al., “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model”, NeurIPS, 2023
  • L. Gao, J. Schulman & J. Hilton, “Scaling Laws for Reward Model Overoptimization”, ICML, 2023

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 2
← Previous
LoRA
Open in the full reader, with the topic sidebar