ExLlamaV2
Also known as: exllama v2 · EXL2
ExLlamaV2 is a high-performance inference engine for CUDA GPUs, with its own EXL2 quantization format.
- Very fast token generation on a single GPU.
- EXL2: a compact, high-quality quant.
- Popular for speed-focused local serving.
When it wins: if raw tokens-per-second on one CUDA GPU is the goal, ExLlamaV2 is a top contender alongside vLLM and llama.cpp.