Corpus
Also known as: corpora
A corpus (plural: corpora) is a large collection of text. It’s the raw text pool that gets cleaned, tokenized, and turned into training data.
Examples: the Common Crawl web snapshots, Wikipedia dumps, project Gutenberg.
A corpus is potential training data; a dataset is curated training data.
Why you'd hear it: "trained on a 2-trillion-token corpus" = the model learned from 2 trillion chunks of text drawn from these pools.