17 — Training

DPO#

Direct Preference OptimizationMathematical 1 min read
◆ The PatternSkip the reward model — optimize preferences directly

DPO reformulates RLHF as a simple classification problem: given (chosen, rejected) pairs, directly optimize the policy — no reward model, no PPO, no RL. The key insight: the optimal RLHF solution has a closed-form mapping between reward and policy.

L_DPO = −log σ( β · [log π_θ(y_w|x)/π_ref(y_w|x) − log π_θ(y_l|x)/π_ref(y_l|x)] )
Increase probability of chosen, decrease rejected, relative to the reference model.
Implicit reward: r(x,y) = β · log[π_θ(y|x) / π_ref(y|x)] + const
DPO learns the reward implicitly — the policy IS the reward model.

Why DPO took over: (1) simpler pipeline — just SFT then DPO, (2) more stable than PPO, (3) cheaper — no separate reward model forward passes, (4) competitive or better quality. Variants: IPO (no sigmoid), KTO (unpaired), ORPO (combines SFT+DPO).

DPO needs a frozen reference model in memory alongside the training model — essentially doubling memory cost. Efficient implementations use LoRA or offload the reference to CPU.
Interactive — DPO preference optimization

Python — DPO loss#

Pattern bridge: Direct Preference Optimization skips the reward model, optimizing preferences end-to-end. Like simplifying a loss function to remove an intermediate step.
Worked example

Relative to the reference model, the chosen answer’s log-probability rises by 2 and the rejected one’s falls by 3: with β = 0.1 the DPO loss is 0.474. If instead the chosen answer falls by 2 while the rejected falls by 6, the loss is 0.513 — still lower than at the start (0.693). DPO rewards the gap, so it can push down the probability of the very answers it prefers.

Where it misleads
  • Falling likelihood of preferred answers is a known failure mode (Pal et al. 2024); monitor the chosen log-probabilities, not just the loss.
  • DPO assumes the Bradley–Terry preference model; with noisy or deterministic preferences it can overfit (Azar et al. 2024).
  • Offline preference data goes stale as the model changes; iterative or online variants address this.
In code
sigmoid = lambda z: 1 / (1 + np.exp(-z))
beta = 0.1
chosen = (-10.0) - (-12.0)                      # log pi(y_w) - log pi_ref(y_w): the chosen answer gained 2 nats
rejected = (-14.0) - (-11.0)                    # the rejected answer lost 3
loss = -np.log(sigmoid(beta * (chosen - rejected)))
both_down = -np.log(sigmoid(beta * ((-2.0) - (-6.0))))   # chosen also falls, by 2; rejected by 6
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
def dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1):
    """All inputs are log-probabilities of the full sequences"""
    chosen_ratio = pi_chosen - ref_chosen
    rejected_ratio = pi_rejected - ref_rejected
    loss = -F.logsigmoid(beta * (chosen_ratio - rejected_ratio))
    return loss.mean()

# In practice: sum log P(token_t | prev) over response tokens
# for both policy (π_θ) and reference (π_ref) models
Sources
  • R. Rafailov et al., “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model”, NeurIPS, 2023
  • M. G. Azar et al., “A General Theoretical Paradigm to Understand Learning from Human Preferences”, AISTATS, 2024
  • A. Pal et al., “Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive”, arXiv:2402.13228, 2024

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 1
← Previous
RLHF
Open in the full reader, with the topic sidebar