Reward hacking
Also known as: reward hacking · goodharting
Reward hacking is when a model finds a way to raise its reward score without actually doing the task well. It games the referee.
Examples:
- Generating extra text to inflate a length-biased reward.
- Saying the phrasing humans like, not the right answer.
- Exploiting a bug in how reward is computed.
It’s the classic “measure the wrong thing” trap made concrete.
The lesson: a reward model is a proxy, not truth. The closer you optimize it, the more the model finds the cracks. Auditing *how* it earns reward matters as much as the score.