Reward Model · PPO✓ Mathematical
◆ The PatternAligning language models with human preferences
RLHF (Reinforcement Learning from Human Feedback) is a 3-stage process: (1) SFT the base model, (2) train a reward model on human comparison data, (3) optimize the SFT model with PPO against the reward model while staying close to the SFT policy.
Stage 2 — Reward: L_RM = −log σ(r(x, y_w) − r(x, y_l))
Bradley-Terry model: chosen response y_w should score higher than rejected y_l.
Stage 3 — PPO: max E[r(x,y)] − β · KL(π_θ || π_ref)
Maximize reward while staying close to the reference policy. β controls the KL penalty.
The KL penalty is crucial — without it, the model "reward hacks": finds adversarial outputs that fool the reward model. Typical β values: 0.01–0.2. RLHF produces noticeably better outputs than SFT alone, but adds significant training complexity.
RLHF was the secret sauce behind ChatGPT's launch. InstructGPT showed that RLHF on a 1.3B model could outperform a 175B SFT model in human evaluations.
Interactive — RLHF pipeline stages
Python — reward model training#
import torch.nn.functional as F
class RewardModel(nn.Module):
def __init__(self, base_model):
super().__init__()
self.model = base_model
self.head = nn.Linear(d_model, 1, bias=False)
def forward(self, input_ids):
hidden = self.model(input_ids).last_hidden_state
reward = self.head(hidden[:, -1, :]) # score from last token
return reward.squeeze(-1)
# Bradley-Terry loss
def reward_loss(chosen_reward, rejected_reward):
return -F.logsigmoid(chosen_reward - rejected_reward).mean()Pattern bridge: Aligning model outputs with human preferences via reward modeling. The math of RLHF is a policy gradient over preference pairs. In markets, herd behavior is collective preference shaping price — the market’s reward signal.