Context offloading
Also known as: offloading · CPU offload
Context offloading (layer/KV offload) moves part of the model from GPU memory to system RAM when it doesn’t all fit in VRAM.
- Lets you run a model bigger than your VRAM.
- The offloaded part is slower (RAM bandwidth).
- A partial offload is a reasonable compromise.
The escape hatch: "it doesn't fit in my GPU" doesn't have to be the end. Offload a few layers and the model still runs — just a bit slower.