Direct Preference Optimization✓ Mathematical
◆ The PatternSkip the reward model — optimize preferences directly
DPO reformulates RLHF as a simple classification problem: given (chosen, rejected) pairs, directly optimize the policy — no reward model, no PPO, no RL. The key insight: the optimal RLHF solution has a closed-form mapping between reward and policy.
L_DPO = −log σ( β · [log π_θ(y_w|x)/π_ref(y_w|x) − log π_θ(y_l|x)/π_ref(y_l|x)] )
Increase probability of chosen, decrease rejected, relative to the reference model.
Implicit reward: r(x,y) = β · log[π_θ(y|x) / π_ref(y|x)] + const
DPO learns the reward implicitly — the policy IS the reward model.
Why DPO took over: (1) simpler pipeline — just SFT then DPO, (2) more stable than PPO, (3) cheaper — no separate reward model forward passes, (4) competitive or better quality. Variants: IPO (no sigmoid), KTO (unpaired), ORPO (combines SFT+DPO).
DPO needs a frozen reference model in memory alongside the training model — essentially doubling memory cost. Efficient implementations use LoRA or offload the reference to CPU.
Interactive — DPO preference optimization
Python — DPO loss#
def dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1):
"""All inputs are log-probabilities of the full sequences"""
chosen_ratio = pi_chosen - ref_chosen
rejected_ratio = pi_rejected - ref_rejected
loss = -F.logsigmoid(beta * (chosen_ratio - rejected_ratio))
return loss.mean()
# In practice: sum log P(token_t | prev) over response tokens
# for both policy (π_θ) and reference (π_ref) modelsPattern bridge: Direct Preference Optimization skips the reward model, optimizing preferences end-to-end. Like simplifying a loss function to remove an intermediate step.