Quantization & local inference
How a 14 GB model becomes a 4 GB one — what quantization does, what it costs, and how to know if your machine can run it.
The core idea
A model’s “brain” is billions of numbers — its weights. Each weight is normally stored as a 16-bit floating point number (FP16). For a 7-billion-parameter model, that’s ~14 GB of pure numbers before a single word of your prompt is processed.
Quantization is the art of storing each weight with fewer bits:
| Precision | Bits/weight | 7B model size | Runs on |
|---|---|---|---|
| FP16 | 16 | ~14 GB | Workstation / server |
| INT8 | 8 | ~7 GB | Good GPU (8 GB VRAM) |
| INT4 (Q4_K_M) | ~4.5 | ~4 GB | Most modern laptops |
| INT3 / Q3 | ~3 | ~3 GB | Older hardware, at a cost |
The trick that makes this work: the model is surprisingly tolerant of imprecise weights. Dropping from 16 bits to 4 loses information, but the model’s behavior degrades much less than the size reduction suggests — usually a small quality drop, a big capability unlock.
What you actually lose
Quantization isn’t free. The typical trade-offs, in order of how you’ll notice them:
- Edge cases get dumber. Common tasks (summarize, translate, chat) barely change. Rare or complex reasoning (multi-step math, subtle code) degrades more.
- Consistency drops slightly. A quantized model is a bit more “loose” in its output distribution — more variation, occasional weird choices.
- The floor drops faster than the ceiling. A Q4 model is usually 90–95% as good as FP16 for everyday use, but its worst answers are noticeably worse.
The practical rule of thumb
Your RAM (or VRAM, if using a GPU) should be at least 1.5× the model file size.
A 4 GB Q4 model needs ~6 GB of free RAM to run comfortably. An 8 GB laptop can run a 7B Q4 model. A 16 GB machine can run a 13–14B Q4 model. Beyond that, you need a dedicated GPU or a server.
GGUF: the local model format
You’ll see “GGUF” everywhere in local AI. It’s a file format (originally from the llama.cpp project) that stores a quantized model with all its metadata in a single file. When someone says “download the GGUF”, they mean: here’s the model, ready to run locally, no server needed.
The quantization level is usually in the filename: model-7b-Q4_K_M.gguf = 7B parameters, 4-bit, “K” quantization method, “M” = medium.