I am working with the 128k context version of Llama 3, and I’m seeing huge memory waste. Does the PagedAttention in vLLM completely solve the fragmentation issue, or do I still need to manually tune the block size to accommodate these extremely long sequences?
3 answers
The core innovation of vLLM is that it treats GPU memory like virtual memory in an OS. This means it allocates "pages" of KV cache only when needed. For 128k context, you don't necessarily need to change the block size (default is 16), but you do need to be careful with your --gpu-memory-utilization setting. If you set it too high, you won't leave enough room for the dynamic allocation of new pages as the conversation grows. I recommend starting at 0.85 and using FP8 quantization for the KV cache itself, which was a feature introduced in the mid-2024 releases to save 50% of cache memory.
Are you seeing "Out of Memory" errors during the prefill stage or during the actual token generation phase?
Try reducing the --max-model-len to only what you actually need. Just because the model supports 128k doesn't mean your app should allow it.
Smart advice, Brandon. Limiting the context window at the vLLM server level is the most effective way to prevent runaway memory usage while you are still tuning your production hardware.
It usually happens during generation, Kevin. It seems like vLLM handles the initial prompt fine, but as the response gets longer, the system runs out of physical blocks to assign to the active sequences, causing a crash or a slow-down.