HuggingFaceBio

/

Carbon-A-1.2B

Like 9

Follow

Hugging Face Biology Research 163

Fixes the FlashAttention install. As documented, pip install --no-build-isolation flash-attn fails:

1. psutil is missing (ModuleNotFoundError within seconds).
2. With psutil, it compiles from source, because torch>=2.7,<3 resolves to the latest PyTorch (2.14 + cu130), for which FlashAttention has no wheel. The compile then fails: PyTorch 2.14 headers need C++20 (std::strong_ordering), and flash-attn builds with C++17.

Changes:

• requirements.txt: torch>=2.7,<2.9 and psutil. flash-attn 2.8.3 has prebuilt wheels for PyTorch 2.7 and 2.8 on Python 3.9–3.13; for 2.9, only on Python 3.12.
• Install command: pip install --no-build-isolation --no-cache-dir "flash-attn==2.8.3". Pinning keeps pip from picking 2.8.3.post1, which has almost no wheels. --no-cache-dir avoids a flash-attn installer error (Invalid cross-device link) when /tmp and ~/.cache are on different filesystems.
• Both READMEs explain the cap, note that the default CUDA 12.8 build of PyTorch runs with CUDA 13 drivers, and give --attn-implementation sdpa for newer PyTorch, with its measured effect.

Tested on an RTX 3090 Ti (driver CUDA 13.4), following the README in a fresh environment:

• Python 3.11 → torch 2.8.0+cu128, transformers 4.57.6; flash-attn 2.8.3 installed from a prebuilt wheel in 18 s.
• Yeast benchmark: nucleotide/exon/gene F1 0.9984 / 0.9429 / 0.9531, matching the card. Raw per-base probabilities are bit-identical to a torch 2.7 + flash-attn 2.8.3 run, and also to torch 2.9.1 cu128/cu130 runs.
• For comparison: PyTorch 2.14 + SDPA gave 0.9984 / 0.9428 / 0.9525, and the kernels-community/flash-attn2 Hub kernel (torch 2.12, Transformers 5) gave 0.9984 / 0.9425 / 0.9527. Neither reproduces the reported probabilities.

🤖 Generated with Claude Code

Closing: we're keeping the current requirements and FlashAttention setup unchanged.