Pretraining (base model)
Also known as: pre-train · base model training
Pretraining (often written one word) is the first and most expensive training phase: train on raw text with the single goal of predicting the next token. The result is a base model — very knowledgeable, but with no sense of how to be a helpful assistant.
- Input: trillions of tokens of raw text.
- Objective: next-token prediction.
- Output: a base model (good at text, bad at being useful).
SFT and RLHF are then layered on top.
Don't confuse the phases: pre-training builds *knowledge*; SFT builds *behavior*; RLHF builds *alignment*. Three different jobs, done in that order.