Tokenization
Also known as: tokenizer · chunking
Tokenization is the process of slicing raw text into tokens — the units a model reads and writes. Models never see whole words; they see a sequence of tokens.
A tokenizer is a small lookup system that decides the split. Common words often get one token; long or rare words get chopped into several.
| Input | Tokens (typical) |
|---|---|
hello | hello |
unhinged | un + hing + ed |
émoji | é + mo + ji (varies) |
Watch out: tokenization differs per model. The same sentence is a different number of tokens in GPT-4 vs Claude vs a local Llama — which is why "X tokens" only makes sense *for a specific model*.