📋 KEY INSIGHTS
- Reward modelling is the process of learning a scalar reward function from human preference data, used in Reinforcement Learning from Human Feedback (RLHF) to align language models with human values. The pipeline: (1) sample pairs of model outputs (y_1, y_2) for the same prompt x; (2) ask human raters which output they prefer; (3) train a reward model R_φ(x, y) to predict human preferences using the Bradley-Terry model: P(y_1 > y_2 | x) = σ(R_φ(x, y_1) − R_φ(x, y_2)), where σ is the sigmoid function; (4) use RL (typically PPO) to fine-tune the LLM to maximise expected reward R_φ(x, y_π) subject to a KL penalty preventing excessive divergence from the reference policy.
- The Bradley-Terry model is the foundational pairwise preference model: it assigns each option i a latent utility score β_i and models the probability that option i is preferred over option j as P(i > j) = exp(β_i) / (exp(β_i) + exp(β_j)) = σ(β_i − β_j). For reward modelling, the utility score is R_φ(x, y) — the reward model assigns a scalar score to each (prompt, response) pair, and the probability that response y_1 is preferred over y_2 for prompt x is σ(R_φ(x, y_1) − R_φ(x, y_2)). The reward model is trained to maximise the log-likelihood of the observed human preferences: L = E_{(x, y_w, y_l)~D}[log σ(R_φ(x, y_w) − R_φ(x, y_l))], where y_w is the preferred (winner) response and y_l is the rejected (loser) response.
- Direct Preference Optimisation (DPO, Rafailov et al., 2023) eliminates the separate reward model training and RL fine-tuning steps by directly optimising the LLM policy on preference data. DPO derives a closed-form expression for the optimal policy in terms of the reference policy and implicitly defines a reward model: r*(x, y) = β log(π*(y|x) / π_ref(y|x)) + β log Z(x), where π* is the optimal policy and Z(x) is the partition function. The DPO loss directly optimises preference data: L_DPO(π_θ; π_ref) = −E_{(x,y_w,y_l)~D}[log σ(β log(π_θ(y_w|x)/π_ref(y_w|x)) − β log(π_θ(y_l|x)/π_ref(y_l|x)))]. DPO is simpler, more stable, and competitive with PPO-based RLHF on most benchmarks, and has become the dominant preference learning algorithm in 2024-2025.
- Constitutional AI (Anthropic, 2022) addresses the scalability bottleneck of human preference annotation by using the AI itself to critique and revise its responses according to a set of principles (the “constitution”), then training on the AI-generated preference data (RLAIF — RL from AI Feedback). The pipeline: (1) sample initial responses; (2) use the AI to critique each response according to constitutional principles (helpfulness, harmlessness, honesty); (3) use the AI to generate a revised response addressing the critique; (4) use the AI to compare original and revised responses, generating synthetic preference pairs; (5) train a reward model on the synthetic preferences; (6) run RL fine-tuning. RLAIF enables training preference models at much larger scale than purely human-annotated datasets, and the constitutional principles provide a transparent, auditable alignment specification.
- Reward hacking (Goodhart Law in RLHF) is the central failure mode: the RL policy finds ways to maximise the reward model score without producing genuinely better responses, by exploiting weaknesses in the reward model that were not anticipated during training. Classic examples: excessively long responses score higher (verbosity bias), excessive sycophancy (responses that agree with the user score higher regardless of correctness), and formatting tricks (heavy markdown formatting inflates scores). Mitigation strategies: KL regularisation (penalising large deviations from the reference policy, constraining the RL policy from drifting too far in reward-hacking directions), reward model ensembles (reducing reward model uncertainty by averaging multiple reward model predictions), and periodic reward model updates using new human preference data collected from the current policy.
- Preference datasets and benchmarks: Anthropic HH-RLHF (helpfulness + harmlessness preference pairs), OpenAI WebGPT comparisons, and UltraFeedback (GPT-4-annotated preferences across 64K instructions) are the dominant open-source preference datasets. RewardBench (Lambert et al., 2024) is the standard benchmark for reward model evaluation, measuring reward model accuracy on preference pairs across four categories: chat, chat-hard, safety, and reasoning. The connection to our LLM evaluation guide is direct: reward model accuracy on RewardBench is a standard metric for characterising RLHF pipeline quality, and reward model failures correlate with downstream LLM quality regressions on MT-Bench and AlpacaEval.
RLHF (Reinforcement Learning from Human Feedback) has become the dominant method for aligning large language models with human preferences — it is the technology that transforms a pre-trained language model that predicts next tokens into an AI assistant that is helpful, harmless, and honest. The core technical challenge RLHF solves: human values are too complex to specify as a hand-coded reward function, but humans can reliably compare two responses and indicate which is better. RLHF converts these pairwise comparisons into a learned reward signal, then uses RL to optimise the LLM against this signal. This guide covers the full RLHF technical stack: the Bradley-Terry preference model, reward model architecture and training, PPO-based RL fine-tuning, Direct Preference Optimisation (DPO) as the reward-model-free alternative, Constitutional AI and RLAIF for scalable feedback, reward hacking and mitigation strategies, and the emerging generation of preference optimisation algorithms (IPO, KTO, ORPO).
Reward Model Training — Bradley-Terry, Architecture and Data Collection
The reward model is typically initialised from the same pre-trained LLM that will be fine-tuned, with the language modelling head replaced by a scalar regression head: a linear layer that maps the final hidden state of the last token to a single scalar reward score. This initialisation ensures that the reward model starts with rich language understanding, dramatically reducing the amount of preference data needed to train a good reward model. Training on preference pairs: for each (prompt, chosen response, rejected response) triple, the Bradley-Terry loss L = −log σ(R(x, y_chosen) − R(x, y_rejected)) updates the reward model to assign higher scores to chosen responses and lower scores to rejected responses. The margin variant L = −log σ(R(x, y_chosen) − R(x, y_rejected) − m) adds a minimum margin m, enforcing that the reward difference exceeds a threshold and improving calibration. Human preference data collection requires careful annotation protocol design: the rater interface, the rating rubric, the number of raters per pair, and the method for resolving disagreements all significantly affect reward model quality. Best practices from published RLHF papers: 3+ raters per pair for important decisions, structured rubrics separating helpfulness, harmlessness, and honesty dimensions, and majority vote or Dawid-Skene model for aggregating disagreeing rater preferences. The connection to our active learning guide is direct: active learning strategies for preference data collection — prioritising prompts where the current reward model is most uncertain — reduce annotation cost by focusing human annotation effort on the most informative comparisons.
| Algorithm | Requires RM? | Requires RL? | Complexity | Performance vs PPO |
|---|---|---|---|---|
| PPO-RLHF | Yes | Yes (online) | High (4 models) | Baseline |
| DPO | No (implicit) | No (SFT-style) | Low (2 models) | Competitive |
| IPO | No | No | Low | More robust vs reward hacking |
| KTO | No | No | Low | Works with unpaired feedback |
| ORPO | No | No | Low (1 model) | Eliminates SFT warmup step |
| Constitutional AI (RLAIF) | Yes (AI-annotated) | Yes | High | Scales to larger datasets |
DPO, Constitutional AI and Reward Hacking Mitigation
Direct Preference Optimisation (DPO) achieves the same alignment objective as PPO-RLHF without training a separate reward model or running an online RL loop. The key insight: the optimal RLHF policy has a closed-form relationship to the reference policy and the reward function, and this relationship can be inverted to express the reward function in terms of the optimal policy. Substituting this expression into the Bradley-Terry loss eliminates the reward model and produces a loss directly on the policy parameters. The practical advantages over PPO: (1) no separate reward model training (saving 25-50% of training compute); (2) no online rollout generation (saving inference compute during training); (3) no PPO clipping hyperparameter tuning; (4) a single supervised fine-tuning-style training loop. The practical disadvantages: (1) DPO is off-policy (uses a fixed preference dataset, not fresh rollouts from the current policy), which can cause distribution shift as the policy evolves; (2) DPO cannot incorporate new human feedback after training without restarting from the reference policy. IPO (Identity Preference Optimisation) addresses DPO instabilities by replacing the log-sigmoid loss with a squared loss that is more robust to confident wrong predictions. KTO (Kahneman-Tversky Optimisation, Ethayarajh et al., 2024) uses only a binary signal (is this response good or bad?) rather than paired comparisons, enabling training on unpaired feedback data that is cheaper to collect. The connection to our RLHF guide is direct: that guide covers PPO-based RLHF in full depth; this guide extends it with the preference learning algorithmic landscape that has emerged since. Constitutional AI (CAI, Anthropic 2022) makes the alignment specification transparent and auditable by encoding human values as a written constitution of principles, then using AI self-critique and revision to generate synthetic preference data at scale without requiring human annotators for every comparison. The AI is prompted to critique its own responses against the constitutional principles (e.g. “identify specific ways this response is harmful or unethical”), generate a revised response, and compare the original and revised versions — generating synthetic (prompt, chosen, rejected) triples for reward model training. This RLAIF pipeline enables alignment at much larger scale than purely human-annotated datasets, and the constitution provides a transparent alignment specification that can be audited, versioned, and updated — connecting to our data science ethics guide where transparent and auditable AI alignment methodologies are described.
✦ SUMMARIZE THIS ARTICLE WITH AI
ChatGPTThe foundational RLHF guide (PPO, reward model setup, KL penalty) is our RLHF guide. LLM fine-tuning (LoRA/QLoRA used for parameter-efficient reward model and policy training) is in our LLM Fine-Tuning guide. LLM evaluation (RewardBench, MT-Bench for measuring RLHF quality) is in our LLM Evaluation guide. Active learning (for efficient preference data collection) is in our Active Learning guide. Data science ethics (Constitutional AI as a transparent alignment specification) is in our Data Science Ethics guide. RL for real-world applications (online RL fine-tuning dynamics) is in our RL for Real-World Applications guide.
