Latency
Also known as: TTFT · time to first token
Latency is how long the user waits. It has two parts:
- TTFT (time to first token): prompt → first word. Feels like “is it thinking?”.
- Throughput / inter-token time: how fast the rest streams in.
Both matter — a fast first token that then stalls feels broken.
Optimize both: TTFT for the "is it alive?" feeling, and streaming throughput for the "it's flowing" feeling. Thinking models have high TTFT by design — they're reasoning first.