Token
Also known as: tokens · tokenization
A token is a chunk of text that a model works with. Models don’t read words or sentences — they read long sequences of tokens, and they write them one at a time.
A typical English word is 1–2 tokens, but long or uncommon words split into more:
| You type | Model sees (roughly) |
|---|---|
cat | cat — 1 token |
unbelievable | un + believ + able — 3 tokens |
🚀 | 🚀 — 1 token (usually) |
Why tokens matter for you:
- Cost — most AI services charge per token, both what you send and what it answers.
- Limits — a model can only “see” a limited number of tokens at once (its context window).
- Speed — generation is one token at a time, which is why answers appear in a stream.
Rule of thumb: 1 token ≈ ¾ of an English word. So 1,000 tokens ≈ 750 words.