Commit ·
f556a68
1
Parent(s): d3aaf4c
Document NVFP4 and experimental INT4RHT tiers
Browse files
README.md
CHANGED
|
@@ -6,10 +6,38 @@ Apache-2.0 SageAttention3 implementation in `thu-ml/SageAttention`.
|
|
| 6 |
Use this package as an explicitly gated lower-precision tier for long video or
|
| 7 |
audio sequences. Its preferred `sage3_prefill_fp4_bf16` entry accepts raw BF16
|
| 8 |
NHD Q/K/V and owns Sage3 centering, padding, FP4 quantization, and correction
|
| 9 |
-
preparation behind a caller-owned CUDA Graph-safe workspace. Head
|
| 10 |
-
|
|
|
|
|
|
|
| 11 |
the higher-fidelity
|
| 12 |
`flashrt/sageattention2-blackwell` default.
|
| 13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
See `README.md` for preprocessing, API usage, accuracy expectations, and the
|
| 15 |
-
complete capability contract.
|
|
|
|
| 6 |
Use this package as an explicitly gated lower-precision tier for long video or
|
| 7 |
audio sequences. Its preferred `sage3_prefill_fp4_bf16` entry accepts raw BF16
|
| 8 |
NHD Q/K/V and owns Sage3 centering, padding, FP4 quantization, and correction
|
| 9 |
+
preparation behind a caller-owned CUDA Graph-safe workspace. Head dimension 64
|
| 10 |
+
is available in CUDA 12.8+ artifacts; head dimension 128 requires a CUDA 13.0+
|
| 11 |
+
artifact. Query `capabilities()` rather than assuming a head-dimension set. It
|
| 12 |
+
does not support GQA/MQA in v1 and does not replace
|
| 13 |
the higher-fidelity
|
| 14 |
`flashrt/sageattention2-blackwell` default.
|
| 15 |
|
| 16 |
+
## Published and experimental variants
|
| 17 |
+
|
| 18 |
+
The v1 artifacts currently publish one executable quantization tier:
|
| 19 |
+
|
| 20 |
+
- **Sage3 NVFP4/E2M1**: FP4 Q/K/V preparation plus block-scaled FP4
|
| 21 |
+
attention, exposed through `sage3_prefill_fp4_bf16` and the lower-level
|
| 22 |
+
`prepare_qkv_fp4_nhd` / `blockscaled_fp4_attention_static` APIs.
|
| 23 |
+
|
| 24 |
+
The **INT4RHT** tier is not present in the v1 binaries and is not callable yet.
|
| 25 |
+
It is a design-stage fallback for deployments that need an accuracy point
|
| 26 |
+
between SageAttention2 and Sage3 NVFP4. Its proposed contract is symmetric
|
| 27 |
+
INT4 Q/K with per-16-element floating-point scales, a shared Hadamard-128
|
| 28 |
+
rotation after RoPE and centering, and the SageAttention2 FP8 P/V path:
|
| 29 |
+
|
| 30 |
+
```python
|
| 31 |
+
out = sage3.sage3_prefill_int4rht_bf16(
|
| 32 |
+
q, k, v, out=out_buffer, workspace=workspace,
|
| 33 |
+
)
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
That symbol will only be published after it passes all three gates: cosine at
|
| 37 |
+
least 0.9999 against the INT4RHT reference on random and real activations,
|
| 38 |
+
model-capture cosine at least 0.997, and all-in latency below SageAttention2
|
| 39 |
+
FP8V. Until then, callers must not probe or advertise it as a supported
|
| 40 |
+
variant.
|
| 41 |
+
|
| 42 |
See `README.md` for preprocessing, API usage, accuracy expectations, and the
|
| 43 |
+
complete capability contract.
|