TurboQuant: More Context, Less Memory

Running large language models locally has always hit the same wall. The KV cache grows with every token you process. A 13B model with a 32K context window needs about 16 gigabytes just for the cache. Add another thousand tokens and you’re looking at half a gig more. If your GPU has less VRAM than that, you’re stuck with shorter contexts or a bigger graphics card.

TurboQuant tackles this by quantizing the KV cache itself. Instead of storing values as 16-bit floats, it compresses them to 4-bit integers. The math is straightforward. Four times less memory for the same context length. The quality impact is small. About one to two percent degradation in perplexity, which is barely noticeable in practice.

I tested it with llama.cpp. The build process is simple. Add a flag to cmake, compile, and you’re done. The quantized cache activates automatically when you run a model. No extra parameters to configure. It just uses less memory and the output quality stays roughly the same.

The practical effect is that a 13B model with 32K context fits on a 12GB GPU. It was impossible before. A 7B model with 64K context fits on an 8GB card. Again, not possible without TurboQuant. This isn’t a marginal improvement. It changes what’s actually runnable on consumer hardware.

Apple Silicon benefits differently. The unified memory architecture means the GPU and CPU share the same pool. TurboQuant frees up memory that both can use. Thermal throttling also improves because there’s less data moving around. The model runs cooler and more consistently.

Flash Attention and TurboQuant can work together. Flash Attention optimizes how attention is computed. TurboQuant reduces how much memory the cache uses. Using both gives you faster inference plus more context. The quality stays at about 98% of full precision with both active.

The trade-off is real but small. One to two percent quality loss isn’t nothing. For tasks that need maximum precision, you might want full precision. But for most everyday use, the difference is imperceptible. You get four times more context for the same hardware cost.

Ollama supports it through an environment variable. Set OLLAMA_KV_QUANT to 4 and run your model. That’s the entire setup. No config files, no compilation, no special build.

I’ve been using it for a few days. The biggest change is that I stopped worrying about context limits. Long documents, extended conversations, complex codebases. The model can handle more without running out of memory. It’s a small change in how I use the tool, but it removes a constraint I didn’t realize was limiting me.

TurboQuant doesn’t make your model smarter. It just lets it remember more. For local AI, that’s often the difference between usable and frustrating.