Batching
Also known as: batch · continuous batching
Batching is grouping multiple inference requests and processing them together so the GPU stays fully utilized.
- Static batching: a fixed set of requests per step.
- Continuous batching: dynamically add/finish requests each step (the modern default).
It’s a primary driver of cost and throughput at scale.
Why it's everywhere: a GPU is expensive and fast in parallel. Batching is how providers squeeze many users out of one GPU — and why your per-token price is what it is.