Quantization
Also known as: quant · 4-bit · GGUF
Quantization is compression for models.
A model is just a big pile of parameters — each one a floating-point number. Originally each number uses 16 bits (FP16). Quantization says: “these numbers don’t need that much precision” — and stores each one in 8, 6, 5, or 4 bits (often called INT8, Q4…).
What you get:
| FP16 (full) | Q4 (4-bit) | |
|---|---|---|
| A 7B model on disk | ~14 GB | ~4 GB |
| RAM needed | ~16 GB | ~6–8 GB |
| Quality | reference | ~95–99% of it (usually) |
So quantization is what makes “run a decent model on a laptop” a real thing instead of a data-center one. The trade is small but real: the rougher the numbers, the slightly dumber the model — for everyday use, a good Q4 build is usually indistinguishable from the full version.
You'll see the format names: GGUF (llama.cpp / Ollama) and AWQ / GPTQ (other tooling). Same idea — squeezing numbers to fit your hardware.