Preference & RL
Preference datasets express relative quality: which answer is better, how good a response is, or which reasoning process is sounder. They train reward models or are used directly via methods like DPO. RL datasets go further: they teach policies that optimize over sequences of decisions, not just single responses.
Preference & Reward Datasets
| Dataset Type | Example Unit | Turn Type | Annotation | Training Use | Behaviors | Weaknesses | Curation Concerns | |
|---|---|---|---|---|---|---|---|---|
| Pairwise preference | Prompt + A/B + winner | Single/multi-turn | A preferred over B | DPO or reward model | Better judgment of quality | Label noise | Clear rubric, rater agreement | |
| Listwise ranking | Prompt + ranked candidates | Single-turn | Full ranking | Richer preference learning | Fine-grained style/quality control | Costly to annotate | Ranking consistency | |
| Scalar scoring | Prompt + response + score | Single/multi-turn | 1-5 or 0-1 rating | Reward regression | Quality estimation | Raters differ a lot | Rater calibration | |
| Rubric-based | Candidates + rubric scores | Single/multi-turn | Scores by dimension | Fine-grained reward signals | Accuracy, clarity, harmlessness | Complex annotation | Rubric design, calibration | |
| Process preference | Problem + reasoning variants | Single-turn | Preferred process | Improve reasoning | Better decomposition, verification | Very expensive | Step segmentation correctness | |
| Constitutional critique/revision | Response + principle + revision | Multi-step | Principle-based critique | Self-improvement alignment | Policy-following, self-critique | Can become rigid | Principle design |
RL Dataset Types
| Dataset Type | Example Unit | Turn Type | Reward Source | What It Trains | Behaviors | Weaknesses | Curation Concerns | |
|---|---|---|---|---|---|---|---|---|
| Logged trajectories | State-action sequences | Multi-step | Human/heuristic reward | Sequential behavior | Planning, recovery, tool use | May contain suboptimal behavior | Reward quality, trajectory length | |
| Online rollouts | Newly sampled trajectories | Multi-step | Reward model/humans | Policy improvement | Adaptation to policy errors | Reward hacking | Prompt mix, reward reliability | |
| Simulated episodes | Agent in environment | Multi-step | Environment score | Goal-directed action | Game play, navigation, search | Simulator mismatch to real users | Realistic environment design | |
| Tool trajectories | Tool call chains | Multi-turn | Task success reward | Reliable action selection | Search, browsing, API use | Long-horizon instability | Accurate simulator, success signal | |
| Outcome-labeled | Prompt + success/fail | Single/multi-step | Success metric | End-task completion | Exactness, execution success | Misses why it failed | High-quality verification |
SFT vs RL
- SFT: Imitate this good answer
- RL: Try actions and get rewarded for what works
Main Risk
If the reward is wrong, the model learns to exploit the reward instead of solving the real task. This is called reward hacking.
Note on RLHF
In practice, RLHF spans both sections above: preference data trains the reward model, then RL optimizes the policy against it. This guide separates them because they use different dataset formats.
Reward hacking occurs when a model finds ways to maximize its reward signal without actually solving the intended task. For example, a model trained with a length-based reward proxy might learn to generate verbose, repetitive responses that score high on the reward model but are unhelpful to users.
Other examples: a model that learns to use confident-sounding language to score well on "helpfulness" even when uncertain, or a coding model that passes specific test cases by hardcoding outputs rather than solving the general problem.
This is one of the core risks of RL-based training and is why reward model calibration, diverse evaluation, and human spot-checks are essential safeguards.
Preference data comes in several forms, each with different tradeoffs:
- Pairwise (A vs B): Cheapest to annotate, but loses magnitude information. You know A is better than B, but not by how much.
- Scalar (rate each 1–5): Captures degree of quality, but suffers from annotator calibration drift: one rater's "4" is another's "3."
- N-way ranking: Richer signal from ordering N responses, but slower and costlier to collect.
DPO uses pairwise data directly. RLHF first trains a reward model (typically on pairwise or ranked preference data) that outputs scalar scores, then optimizes a policy against it. The choice depends on annotation budget, task subjectivity, and whether you need a standalone reward model for other purposes.
Process supervision labels each step of a reasoning chain as correct or incorrect.Outcome supervision only labels the final answer as pass or fail.
Process supervision is far more expensive to annotate but produces more reliable reasoners. The model learns which reasoning steps are valid, not just which final answers happen to be right. Outcome supervision can inadvertently reward models that get the right answer for the wrong reasons.
In practice, outcome supervision is used more widely because it's cheaper (you just need a verifier or test suite), while process supervision is reserved for high-stakes domains like math, code, and formal reasoning where intermediate correctness matters.