DPO (vs RLHF)
Also known as: DPO · direct preference optimization
DPO (Direct Preference Optimization) is a technique that learns from human preference data without training a separate reward model or running a full RL loop.
It’s become a popular alternative to RLHF because it’s:
- Simpler — no reward model to train and stabilize.
- Cheaper — one optimization step, not a multi-stage RL process.
- Equally effective in many setups.
The data (pairs of preferred/dispreferred answers) is the same; only the method differs.
The trend: DPO and its variants are increasingly replacing classic RLHF for alignment, precisely because they're easier to run well.