Training data
Also known as: training corpus
Training data is the input a model learns from. For an LLM, it’s a massive mix: web pages, books, articles, code repositories, Q&A sites — trillions of tokens.
Key facts:
- Scale: modern LLMs train on datasets so large they’re measured in petabytes.
- Curation matters: what’s included (and excluded) shapes the model’s knowledge and biases.
- The cutoff: models only know what was in the data before their training ended.
Why it shapes answers: a model's "knowledge" is a compression of its training data. Anything not in that data is, fundamentally, unknown to it.