Flash-Attention Prebuilt Wheels for Windows + Blackwell (SM 12.0)

🎯 Overview

Prebuilt flash-attn wheels for Windows, targeting NVIDIA RTX 50 series (Blackwell / SM 12.0) GPUs, compiled with CUDA 13.0.

🇨🇳 国内镜像 / Also on ModelScope: https://www.modelscope.cn/models/wxd11011/flash-attn-windows-blackwell

This repository fills that gap by compiling from source, making it ideal for ComfyUI / Stable Diffusion / Flux / LLM inference users who want to download and use directly without setting up a build environment.

Two versions are provided — choose based on your installed PyTorch and Python:

Version	flash-attn	Python	PyTorch	PTX	Filename	SHA256
🅰️	2.8.4	3.13	2.13.0+cu130	✅	`flash_attn-2.8.4+...cp313...whl`	5ed057fd53fbd870fedb7f309b0f1624196e47ce237311a05c46b112bf089744
🅱️	2.8.3	3.12	2.9.1+cu130	❌	`flash_attn-2.8.3+...cp312...whl`	c3e10b874565b64e18919c90e2ab9af5f7bc9d97296fc052991c2686533487a5

⚠️ How to Choose

Wheels are version-sensitive. Check your environment first:

python -c "import torch, sys; print('Python:', sys.version); print('PyTorch:', torch.__version__, '| CUDA:', torch.version.cuda)"

Python 3.13 + PyTorch 2.13.0+cu130 → Version 🅰️ (flash-attn 2.8.4)
Python 3.12 + PyTorch 2.9.1+cu130 → Version 🅱️ (flash-attn 2.8.3)

📦 Download & Install

Method 1: huggingface-cli

# Version 🅰️
huggingface-cli download YOUR_USERNAME/flash-attn-windows-blackwell \
  flash_attn-2.8.4+cu13torch2.13cxx11abiTRUE-cp313-cp313-win_amd64.whl \
  --local-dir ./

# Version 🅱️
huggingface-cli download YOUR_USERNAME/flash-attn-windows-blackwell \
  flash_attn-2.8.3+cu13torch2.9.1cxx11abiTRUE-cp312-cp312-win_amd64.whl \
  --local-dir ./

Method 2: Python API

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="YOUR_USERNAME/flash-attn-windows-blackwell",
    filename="flash_attn-2.8.4+cu13torch2.13cxx11abiTRUE-cp313-cp313-win_amd64.whl"
)
print(path)

Method 3: Direct Download

Click the "Files and versions" tab above, then click the download icon next to the .whl file.

Install

pip install flash_attn-2.8.4+cu13torch2.13cxx11abiTRUE-cp313-cp313-win_amd64.whl

ComfyUI portable users, use the embedded Python path:

d:\your_path\ComfyUI_windows_portable_nvidia\python_embeded\python.exe -m pip install the_wheel_file.whl

✅ Verify Installation

import torch
from flash_attn import flash_attn_func
import flash_attn

def test_flash_attn():
    print(f"flash_attn version: {flash_attn.__version__}")

    if not torch.cuda.is_available():
        print("❌ CUDA is not available. Flash Attention requires GPU support.")
        return False

    # Small dimensions to avoid OOM
    batch_size, seq_len, n_heads, head_dim = 2, 8, 4, 32
    dtype = torch.float16
    device = "cuda"

    # Random q, k, v with gradients
    q = torch.randn((batch_size, seq_len, n_heads, head_dim), dtype=dtype, device=device, requires_grad=True)
    k = torch.randn((batch_size, seq_len, n_heads, head_dim), dtype=dtype, device=device, requires_grad=True)
    v = torch.randn((batch_size, seq_len, n_heads, head_dim), dtype=dtype, device=device, requires_grad=True)

    # Forward pass
    out = flash_attn_func(q, k, v, causal=False)

    # Check output shape
    expected = (batch_size, seq_len, n_heads, head_dim)
    if out.shape != expected:
        print(f"❌ Output shape mismatch: {out.shape} != {expected}")
        return False

    # Backward pass (test gradients)
    loss = out.sum()
    loss.backward()

    if q.grad is None or k.grad is None or v.grad is None:
        print("❌ Gradient computation failed")
        return False

    print("✅ Flash Attention test passed!")
    return True

if __name__ == "__main__":
    test_flash_attn()

🎮 Supported GPU Architectures

Version 🅰️ (flash-attn 2.8.4)

5 architectures × 96 kernels (480 cubins) + 96 sm_120 PTX:

Arch	GPU	Code Type
sm_80	A100, RTX 3090	SASS
sm_90	H100, H200	SASS
sm_100	B100, B200 (datacenter)	SASS
sm_110	Blackwell variant	SASS
sm_120	RTX 5060/5070/5080/5090	SASS + PTX ✅

Version 🅱️ (flash-attn 2.8.3)

4 architectures × 72 kernels (288 cubins), no PTX:

Arch	GPU	Code Type
sm_80	A100, RTX 3090	SASS
sm_90	H100, H200	SASS
sm_100	B100, B200 (datacenter)	SASS
sm_120	RTX 5060/5070/5080/5090	SASS

❌ NOT Supported

sm_86: RTX 3060 / 3070 / 3080 / 3080 Ti / A10 / A40
sm_89: RTX 4060 / 4070 / 4080 / 4090 / L40 / L40S

📌 Use Cases

✅ ComfyUI custom nodes requiring flash-attn
✅ Stable Diffusion / Flux inference acceleration
✅ LLM inference (vLLM / Transformers)
✅ Any project needing Flash Attention on Windows + Blackwell GPU

⚠️ Notes

Versions must match exactly, especially PyTorch and Python versions
This is a community-compiled, unofficial build — verify file integrity before use
If you get DLL load failed, ensure CUDA 13.0 runtime and VC++ 2022 redistributable are installed
PyTorch distributed on Windows requires gloo backend (NCCL not supported)

🙏 Acknowledgments

Flash-Attention: https://github.com/Dao-AILab/flash-attention
PyTorch: https://pytorch.org
CUDA Toolkit: https://developer.nvidia.com/cuda-toolkit
Compilation reference: https://blog.csdn.net/u014451778/article/details/156356011

📄 License

This is a compiled artifact of flash-attention, licensed under the BSD 3-Clause "New" or "Revised" License.

License text: https://github.com/Dao-AILab/flash-attention/blob/main/LICENSE

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

Other

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support