Pre-training
Also known as: pretraining · training
Pre-training is the first — and most expensive — phase of building a model.
The recipe is almost comically simple:
- Take a model with random numbers.
- Show it a staggering amount of text (books, code, websites, papers).
- Every step, ask one question: “what comes next?”
- Compare its guess with the real next token, and nudge its parameters to do better.
- Repeat trillions of times.
After this, the model has never been told a single “fact” or “rule” — it has just learned the statistical shape of human language. That’s enough for it to write code, translate, summarize, and almost reason. What it doesn’t have yet is helpfulness: a pre-trained model is a completion engine, not an assistant.
That’s the job of the next phase: fine-tuning.
Cost check: pre-training a frontier model costs tens of millions of dollars in compute. This is why big labs train a handful of models a year, and why most of us fine-tune or rent instead.