I am trying to run large models on hardware with limited VRAM. Does vLLM support 4-bit quantization methods like AWQ? I want to know if I can maintain the high throughput benefits while significantly reducing the model's memory footprint to fit on a single RTX 3090 or 4090 card for a local community project.
3 answers
Yes, it has excellent native support for AWQ quantization. In fact, using AWQ with this framework is one of the most efficient ways to run models on consumer hardware. When you use the 4-bit weights, the memory overhead for the model itself drops by nearly 70%, leaving much more room for the KV cache. This means you can still handle a decent number of concurrent users even on a single 24GB card. The performance loss compared to FP16 is negligible for most chatbot applications, making it an ideal choice for developers who are trying to balance cost and speed.
Does the installation process for the AWQ kernels require a specific version of CUDA, or can we run it on the standard 11.8 environment?
It works perfectly for 4-bit models. We are currently running a Mixtral instance on two 3090s using this exact setup with great success.
That's impressive for Mixtral! It really shows how much of a difference the quantization combined with smart memory management can make.
It generally prefers CUDA 12.1 for the latest kernel optimizations, but you can build it for 11.8 if needed. I highly recommend moving to 12.1 though, as the kernels are significantly faster and more stable for the PagedAttention operations.