Alignment✓ Mathematical
◆ The PatternMaking language models helpful, harmless, and honest with human feedback
RLHF aligns a pretrained LLM with human preferences in three stages: supervised fine-tuning, reward model training, and PPO optimization. DPO simplifies this to a single training step.
Stage 1: SFT — fine-tune on high-quality demonstrations
Supervised fine-tuning on curated instruction-response pairs
Stage 2: Reward Model — R(x,y) trained on human preferences
Human annotators rank outputs; model learns to predict which response humans prefer
Stage 3: PPO — max E[R(x,y)] − β·KL(π||π_ref)
Optimise policy to maximise reward while staying close to SFT model (prevents reward hacking)
DPO: L = −log σ(β(log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)))
DPO = Direct Preference Optimization — no reward model needed. Simpler, more stable.
// RLHF pipeline — three-stage process
DPO vs RLHF: DPO reformulates RLHF as a simple classification loss on preference pairs — no reward model, no PPO, no RL instability. LLaMA 2, Zephyr, and many modern models use DPO.
Pattern bridge: Training a reward model from human preferences connects to RLHF in LLM alignment and preference testing in statistics. In markets, herd behavior is collective preference shaping price — the market’s reward signal.