Reward model
Also known as: RM · preference model
A reward model (RM) is a model trained to predict human preferences: given a prompt and two answers, it scores which one a human would prefer.
During RLHF it acts as the referee — the LLM generates candidates, and the reward model tells it which direction to push.
- Trained on human comparison data.
- Used to guide RL, then often discarded.
Subtle point: the reward model can be gamed (the LLM finds phrasing that scores high but isn't actually better). That's a known failure mode of RLHF.