Sampling
Also known as: token sampling
Sampling is the step where the model turns its probabilities over the vocabulary into an actual next token. Instead of always taking the top choice, it samples from the distribution.
That’s where the “creativity” knobs live:
- Temperature — how spread out the probabilities are.
- Top-p (nucleus) — sample only from the smallest set that covers probability p.
- Top-k — sample only from the top k tokens.
The mental model: low sampling settings = predictable, focused; high = diverse, surprising. It's the same dial for "safe" vs. "creative".