I’m running a Llama-3 based service, and as my context windows grow toward 32k and 64k tokens, the self-attention mechanism is eating all my VRAM and causing massive latency spikes. I’ve heard FlashAttention-2 is the answer for "IO-aware" attention, but I’m confused about the implementation.
Do I need to rewrite my attention kernels in CUDA, or is there a high-level way to toggle this in PyTorch? Also, does it play well with KV-cache management and PagedAttention? I’m specifically using NVIDIA A100s and H100s—are there extra optimizations for the Hopper architecture I should be aware of?
3 answers
If you're using the transformers library, enabling it is literally a one-liner. You just pass attn_implementation="flash_attention_2" when loading your model.
In 2026, you definitely don't need to write custom CUDA kernels unless you're doing extreme research. FlashAttention-2 is now "batteries included" in most major frameworks. Its primary role is to eliminate the $O(N^2)$ memory bottleneck by using Tiling and Recomputation.
Instead of writing the huge $N \times N$ attention matrix to the slow GPU memory (HBM), FlashAttention-2 breaks the matrix into small blocks that fit into the super-fast on-chip SRAM. It computes the attention for that block, updates the output, and moves to the next. For your H100s, FlashAttention-2 is a must because it’s optimized for Ampere and Hopper architectures, delivering up to 225 TFLOPs/s on an A100.
For your long-context Llama-3 project, the interaction with the KV-cache is where you'll see the biggest win. FlashAttention-2 reduces the memory reads/writes for the "Outer Loop" of the attention calculation.
Exactly, Steven. For those on H100s, look into FlashAttention-3 (released recently), which adds support for FP8 quantization and "Warp Specialization." It splits the GPU warps into "producers" (loading data) and "consumers" (doing the math), which hides memory latency even better than version 2. If you combine FlashAttention-2 with PagedAttention (like in vLLM), you can effectively serve 10x more users on the same hardware by preventing KV-cache fragmentation.
One thing to remember, Richard: FlashAttention-2 requires FP16 or BF16 precision. It won't work with FP32 because the whole point is to fit data into SRAM, which is very limited. If you’re using PyTorch 2.2+, you can also use torch.nn.functional.scaled_dot_product_attention (SDPA), which automatically selects the FlashAttention-2 kernel if your hardware (Ampere+) and data types are compatible. This is often better than the manual flash-attn package because it handles "causal" masking and dropout automatically without you needing to manage the low-level logic.