llama.cpp: the engine that runs local AI

By the end of this page, you'll be running the engine that powers most of local AI — including Ollama — with full control: your own model file, your own server, your own numbers. No installer, no daemon, one zip file.

What llama.cpp actually is

A quantization pipeline plus an inference engine, written in C/C++. It reads a single file — a GGUF — and runs the model on CPU (or GPU). Three things to know before you start:

It’s the engine, not the product. Ollama wraps it. LM Studio wraps it. You get the raw machine.
One file = one model. The whole thing is a .gguf you can copy, rename, put on a USB stick.
CPU-friendly by design. That’s the whole point of the project.

Step 1 — Get the build

Official Windows release (CPU build, ~25 MB, no installer):

https://github.com/ggml-org/llama.cpp/releases/download/b12223/llama-b12223-bin-win-cpu-x64.zip

Unzip it — you get a folder of .exe files, the important ones being llama-server.exe (the API server) and llama-cli.exe (the chat). Confirm:

$ llama-server.exe --version
version: 0.9.0-dev (build 12223, commit 0c6dca2fa)
built with Clang 20.1.8 for Windows x86_64

Step 2 — Get a model (GGUF)

Models are distributed as GGUF on Hugging Face. A good starter — 4B parameters, Q4_K_M quantization, ~2.3 GB:

https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf

Save it somewhere, e.g. models/Qwen3-4B-Q4_K_M.gguf. That’s the entire model database.

Step 3 — Serve it (the useful part)

llama-server.exe -m models/Qwen3-4B-Q4_K_M.gguf --port 8090 --host 127.0.0.1 -c 4096

One line, and you have an OpenAI-compatible API on port 8090:

$ curl http://127.0.0.1:8090/v1/chat/completions -d @payload.json

Real response from a test run (payload asks the model to reply API_OK and nothing else):

{"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"API_OK", ...

Note the content is exactly API_OK. Qwen3-family models “think” first — llama.cpp puts that reasoning in a separate reasoning_content field, so your actual answer stays clean. Same endpoint shape as Ollama’s, so any app you pointed at :11434/v1 can point at :8090/v1 instead.

Step 4 — Chat from the terminal

llama-cli.exe -m models/Qwen3-4B-Q4_K_M.gguf -p "Hello, how are you?" -n 200

It prints the answer and exits. Add -cnv-style flags (or just use the server) for real conversations.

What to expect on CPU

Measured with llama-bench.exe on the 4B Q4_K_M model, 4 CPU threads:

Workload	Speed
Prompt processing (64 tokens)	45.8 t/s
Generation (128 tokens)	12.8 t/s

Rough rule: 4B ≈ 12–13 t/s, 9B ≈ 6 t/s on a mid-range CPU. Enough to read, slow for novels. If you have a GPU, the CUDA/Metal builds of the same release will run the same model 5–20× faster.

Beware the thinking models. Qwen3 and Qwen3.5 "think" before answering, and small ones (4B) can burn your entire `-n` token budget *thinking* without ever answering. If the output is only reasoning, raise `-n` (e.g. 600) or use a model with thinking off.

Shell + accents = broken JSON. If your prompt has `é`, `à`, `ñ` and you paste it inline in `curl -d`, the shell may mangle the UTF-8 and the API returns a 400 `ill-formed UTF-8 byte`. Put the JSON in a file and use `curl -d @payload.json` instead. (We hit this during testing.)

Where this leaves you

You can now run any GGUF model with a single file and a single binary. Ollama added the convenience layer (auto-download, model registry, auto-unload); llama.cpp is the layer underneath it.

Next tutorial: [Open WebUI — a ChatGPT-style interface, local →](/tutorials/local/open-webui/)