Install flash-attn on a GPU server, and when you do not need it

Most GPU servers no longer need the flash-attn package: PyTorch's built-in scaled_dot_product_attention already runs a FlashAttention-2 kernel, transformers uses it by default, and vLLM and SGLang ship their own attention kernels. You need flash-attn when your code imports flash_attn or loads a model with attn_implementation="flash_attention_2". The catch is the release. flash-attn 2.8.3.post2, published on 2026-10-13, has prebuilt wheels only for PyTorch 2.4 to 2.8 on CUDA 12, and its source no longer compiles against PyTorch 2.13 or newer. On current PyTorch that leaves three ways in: pin torch 2.8 and take the wheel, build from source, or let transformers load a prebuilt kernel from the Hugging Face Hub.

What FlashAttention is#

FlashAttention computes ordinary, exact attention in a different order. Standard attention writes a score for every pair of tokens to the GPU's main memory, a matrix that grows with the square of the sequence length, and then reads it back. FlashAttention works in tiles that stay in the GPU's small, fast on-chip memory, so it never stores the full matrix and moves far less data. The 2022 paper calls it IO-aware exact attention: the result is the same, only the memory traffic changes.

Each version targets newer hardware. FlashAttention-2 (2023) runs about twice as fast as the first version and reaches 50 to 73% of an A100's theoretical throughput, with memory that grows linearly with sequence length instead of quadratically. FlashAttention-3 (2024) is written for Hopper and reports 1.5 to 2 times FlashAttention-2 on an H100. FlashAttention-4, written in NVIDIA's CuTe DSL, targets Hopper and Blackwell and is still in beta.

Do you still need the package#

The software you probably run already has FlashAttention inside it:

You can check that PyTorch's own FlashAttention kernel works on your GPU without installing anything. Allowing only that backend makes the call use it:

import torch
import torch.nn.functional as F
from torch.nn.attention import SDPBackend, sdpa_kernel

q, k, v = (torch.randn(2, 16, 4096, 128, device="cuda", dtype=torch.bfloat16) for _ in range(3))
with sdpa_kernel(SDPBackend.FLASH_ATTENTION):
    out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print(out.shape)

If the inputs do not qualify for the fused kernel, PyTorch warns with the reason. The package is still the right tool for training code that calls flash_attn functions directly, for its variable-length and KV cache functions, and for repositories that list flash-attn in their requirements.

Install the prebuilt wheel#

The prebuilt wheels live on the project's GitHub release page, not on PyPI, which holds only the 8.5 MB source archive. Each file name encodes four things your environment has to match:

Release 2.8.3.post2 has 48 wheels, about 256 MB each: CUDA 12 only, PyTorch 2.4, 2.5, 2.6, 2.7 and 2.8, Python 3.9 to 3.13, both ABIs, Linux x86_64. There is no CUDA 13 wheel and none for PyTorch 2.9 or newer. So the wheel route means a fresh environment with PyTorch 2.8. On a QuantaCloud Bare Metal VM (Ubuntu 22.04), with uv as in the vLLM install guide:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
mkdir -p ~/fa && cd ~/fa
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128

The cu128 build of PyTorch needs an R570 or newer driver, and checking your driver and CUDA version shows which build yours supports.

Then build the file name the same way flash-attn's own setup script does, and install that wheel by its URL:

WHEEL=$(python - <<'EOF'
import sys, torch
torch_mm = ".".join(torch.__version__.split("+")[0].split(".")[:2])
cuda = torch.version.cuda.split(".")[0]
abi = str(torch._C._GLIBCXX_USE_CXX11_ABI).upper()
py = f"cp{sys.version_info.major}{sys.version_info.minor}"
print(f"flash_attn-2.8.3.post2+cu{cuda}torch{torch_mm}cxx11abi{abi}-{py}-{py}-linux_x86_64.whl")
EOF
)
echo "$WHEEL"
pip install "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3.post2/$WHEEL"

With PyTorch 2.8 on Python 3.12 this prints flash_attn-2.8.3.post2+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl, because PyTorch's Linux wheels use the C++11 ABI since version 2.7. A 404 from pip means no wheel exists for your combination.

pip install flash-attn==2.8.3.post2 --no-build-isolation performs the same lookup, but it runs the setup script first, and that script stops with CUDA_HOME environment variable is not set on a VM without the CUDA toolkit. The direct URL needs no toolkit at all.

Build from source#

You build from source when no wheel matches, which today means any PyTorch from 2.9 on. Which source to build depends on your PyTorch. Release 2.8.3.post2 still compiles with C++17, and the headers of PyTorch 2.13 and newer need C++20, so the build fails. PyTorch 2.14 and 2.15 stop it with #error C++20 or later compatible compiler is required to use ATen. The fix is merged on the main branch, which reports itself as version 2.8.4, but no release carries it yet. On current PyTorch, build main at a pinned commit.

The build needs four things. A CUDA toolkit whose major version matches torch.version.cuda, because PyTorch refuses to compile against a different major version. A C++ compiler. The Python packages ninja, packaging, psutil, setuptools and wheel. And PyTorch installed first, with --no-build-isolation so the build can see it. For PyTorch 2.15 from PyPI, which is a CUDA 13.2 build:

python -c "import torch; print(torch.__version__, torch.version.cuda)"

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y build-essential git cuda-toolkit-13-2
export CUDA_HOME=/usr/local/cuda-13.2 PATH=/usr/local/cuda-13.2/bin:$PATH

pip install ninja packaging psutil setuptools wheel
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout f63c472c4dc19237c0635d6a629df065ff64796e
FLASH_ATTENTION_FORCE_BUILD=TRUE FLASH_ATTN_CUDA_ARCHS=80 \
  pip wheel . --no-build-isolation --no-deps -w ~/wheels
pip install ~/wheels/flash_attn-*.whl

Use the toolkit version your PyTorch prints, cuda-toolkit-12-8 for a cu128 build for example. The toolkit packages install no driver, so they leave the VM's driver alone. FLASH_ATTENTION_FORCE_BUILD=TRUE skips the wheel lookup. The setup script fetches its one submodule, CUTLASS, from the clone by itself.

FLASH_ATTN_CUDA_ARCHS is the setting that matters most for build time. By default the script compiles every kernel for each of five GPU generations, and setting it to the one you have compiles each kernel once. Kernels built for 8.0 also run on the 8.6 and 8.9 cards, because a binary runs on any GPU with the same major compute capability and an equal or higher minor one:

How long it takes depends on cores and memory. flash-attn's README gives 3 to 5 minutes on a 64-core machine with ninja, and about 2 hours without ninja, because the build then runs one compile at a time. Left unset, MAX_JOBS is chosen by the setup script from the free memory and CPU cores, and the README warns that machines with less than 96 GB of RAM can run out with too many parallel jobs. If the build is killed, set MAX_JOBS=1 and run it again.

Building to ~/wheels instead of installing directly keeps the result. Stopping a QuantaCloud instance terminates it and deletes its disk, so copy the wheel to your laptop before you stop (transferring files) and install it on the next instance in seconds instead of compiling again. It only fits the same PyTorch, CUDA and Python.

The errors people hit#

Most failures come from a mismatch between flash-attn and the PyTorch it builds against or loads into:

flash-attn's own wheel build files document the undefined symbol error for a C++ ABI mismatch, which is why the ABI is part of the wheel name. The CUDA out of memory guide covers memory errors at run time.

Use it from transformers#

With the package installed, transformers takes it by name. FlashAttention needs half precision, so set the dtype:

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B", dtype=torch.bfloat16, attn_implementation="flash_attention_2"
).to("cuda")
print(model.config._attn_implementation)

Without the package, transformers 5.22.2 can download a prebuilt FlashAttention-2 kernel from the Hugging Face Hub instead, which avoids the build entirely. Install the kernels extra, which brings a version transformers accepts:

pip install "transformers[kernels]==5.22.2"

The same code then logs a warning that flash_attn is not installed and that it is using kernels-community/flash-attn2 from the kernels library instead. transformers 5.22.2 accepts kernels from 0.17.2 up to but not including 0.18, so do not upgrade kernels separately to 0.18.2. A Hub kernel is code that runs on your machine, and transformers trusts only the kernels-community organization unless you pass allow_all_kernels=True.

FlashAttention-3 and 4 on Hopper and Blackwell#

FlashAttention-3 is a beta for Hopper GPUs. The repository lists H100 and H800 and CUDA 12.3 or newer, with 12.8 recommended, and covers FP16 and BF16 training plus an FP8 forward pass. It installs from source, from the repository's hopper directory with python setup.py install. The flash-attn-3 name on PyPI holds only a placeholder. In transformers, attn_implementation="flash_attention_3" falls back to a Hub kernel when the package is missing and the kernels extra is installed. On QuantaCloud the Hopper GPUs are the H100 PCIe and H200 NVL, when the catalog lists them.

FlashAttention-4 is the one that installs without a build. It is written in NVIDIA's CuTe DSL, ships as a 0.56 MB pure-Python wheel, and its README describes it as optimized for Hopper and Blackwell GPUs such as the H100 and B200. It is still a beta with a new release every week, so pin one:

pip install "flash-attn-4[cu13]==4.0.0b39"

The README imports it as from flash_attn.cute import flash_attn_func. Leave out [cu13] on a CUDA 12 PyTorch. transformers' attn_implementation="flash_attention_4" requires compute capability 9.0 or newer, and SGLang 0.5.23 installs FlashAttention-4 as one of its own dependencies. FlashAttention-4 installs into the same flash_attn package directory, and the 2.8.3.post2 package ships an older flash_attn/cute folder of its own, so I keep the two in separate environments.

The README's FlashAttention-2 support list still stops at Hopper. The 2.8.3.post2 wheels are compiled for compute capability 8.0, 9.0, 10.0 and 12.0, which includes the RTX PRO 6000 Blackwell, but the project does not list Blackwell as supported. B200 and B300 servers are built to order: Send a capacity brief.

Check that it works#

This runs flash-attn's kernel and compares it with PyTorch's own attention on the same inputs:

import torch
import torch.nn.functional as F
import flash_attn
from flash_attn import flash_attn_func

q, k, v = (torch.randn(2, 4096, 16, 128, device="cuda", dtype=torch.bfloat16) for _ in range(3))
out = flash_attn_func(q, k, v, causal=True)
ref = F.scaled_dot_product_attention(
    q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=True
).transpose(1, 2)
print(flash_attn.__version__, tuple(out.shape), (out - ref).abs().max().item())

flash-attn takes tensors shaped batch, sequence, heads, head size, and PyTorch takes batch, heads, sequence, head size, hence the transposes. If it prints the version, the shape and a difference at the level of BF16 rounding, the extension loaded, ran on the GPU and agrees with PyTorch. An import error here is one of the rows in the errors table.

flash-attn FAQ#

Is flash-attn the same as FlashAttention in PyTorch?

They share the algorithm, not the package. PyTorch's scaled_dot_product_attention includes a FlashAttention-2 kernel and picks it automatically when the inputs qualify. The flash-attn package is the original implementation, with its own functions and more options.

Which flash-attn wheel do I need?

The one whose name matches your PyTorch's CUDA major version, its major and minor version, its C++ ABI and your Python, from the GitHub release page. The snippet in the wheel section prints the exact name. Release 2.8.3.post2 has wheels for PyTorch 2.4 to 2.8 on CUDA 12 only.

Does flash-attn work with PyTorch 2.14 or 2.15?

Not from a release. The newest release has no wheels for them, and its source does not compile against PyTorch 2.13 or newer. Build the main branch at a pinned commit, or let transformers load the kernels-community Hub kernel.

Does vLLM need flash-attn installed?

No. vLLM 0.33.0 compiles its own build of FlashAttention, vllm-flash-attn, into its wheel, so a separate flash-attn package adds nothing vLLM uses. Keep vLLM in its own environment, as the vLLM install guide recommends.

My rule: start with PyTorch's SDPA, which transformers already uses, and install flash-attn only when code asks for it by name. When it does, match PyTorch 2.8 to the prebuilt wheel if you can choose your PyTorch, use the Hub kernel through transformers if you cannot, and build main from source only for code that imports flash_attn on PyTorch 2.13 or newer. For training workloads, the fine-tuning VRAM guide helps you pick the GPU, and PyTorch DDP and FSDP covers more than one.