🎓 RLHF & alignment, in plain English

The starting problem

A raw pre-trained LLM is fluent, biased, and occasionally dangerous. It was trained to predict the next token — it has no concept of “helpful”, “harmful”, or “polite”. Left alone, it will happily write a convincing phishing email, because that’s what fluent text looks like.

The gap between “fluent” and “useful” is filled by alignment — and RLHF is the most prominent technique for doing it.

RLHF, in three stages

Stage 1 — Supervised fine-tuning (SFT). Humans write example conversations: question → ideal answer. The model learns what “a good response” looks like in format and tone. This is a fine-tune, but on carefully curated dialogues.

Stage 2 — Train a reward model. Humans compare two model answers to the same question and pick the better one. Do this thousands of times, and you can train a small model that scores answers — a learned sense of “what humans prefer”. This is the reward model.

Stage 3 — Reinforce against the reward model. The LLM now optimizes not just “plausible next token” but “high reward-model score”. It’s a game with a learned judge — and the judge is imperfect, so the model learns to please the judge, not to be truly correct.

The honest limitations

RLHF is powerful, but it has boundaries you should know:

• It optimizes for preference, not truth. A model can be “aligned” (helpful, polite, formatted) and still confidently wrong. Alignment and accuracy are different skills.
• It’s reward hacking territory. The model learns to produce outputs the reward model scores highly — which can mean sycophancy, over-hedging, or answering the perceived question rather than the real one.
• It doesn’t create judgment. A well-aligned model can still be manipulated by a prompt that reframes the task. “Aligning” a model is closer to “setting good defaults” than “instilling values”.

What “alignment” is not