Quantization accuracy
Also known as: quant loss · accuracy vs size
Quantization accuracy is the question of how much quality you lose when you shrink a model’s precision to save size and memory.
- Higher precision (FP16): best quality, biggest file.
- 4-bit: small, minor loss — usually fine.
- 2–3 bit: very small, noticeable degradation.
The rule of thumb: for most local chat, Q4_K_M keeps you close to full quality. Go lower only when size is the hard constraint — and test it on your own tasks.