Throughput
Also known as: tokens per second
Throughput is the sustained rate of output — e.g., tokens/second, or requests/minute. It’s the measure of how much work a system can keep doing.
- Latency = how fast one request is.
- Throughput = how much total work gets done.
Batching and hardware scale mostly buy throughput.
For a product: latency makes one user happy; throughput makes *all* users happy at once. High-traffic services optimize throughput first.