Loss function
Also known as: loss · cost function
A loss function turns “how wrong is the model?” into a single number. Training’s entire job is to make that number as small as possible.
- For language models: roughly, how surprising the true next tokens were.
- Lower loss = the model’s predictions match the data better.
You’ll often see “loss” reported while a model trains — it’s the model’s report card, decreasing over time.
Watch for a trap: a *falling* loss doesn't always mean a better model — it can mean the model is memorizing. See [overfitting](/glossary/overfitting/).