Hugging Face's logo

kernels-community
/
vllm-flash-attn3

Kernels
Model card Files
xet
 Community
9

attention sinks & backward

#3
by acforvs - opened
Discussion
acforvs

Hi, thanks for all the amazing work!

I noticed that a new s_aux parameter has been added to the fwd pass to support attention sinks. However, I wasn't able to find any related changes in the backward pass.

Does the current implementation support training as well? if not, are there any plans to add support for attention sinks to the bwd pass?

Many thanks,
Vlad
pcuenq

Good question! Are there any plans for it @danieldk ?
liujingcs
•
edited Aug 7, 2025

Same question here! Is support for this planned?
KirillShmilovich

@pcuenq @acforvs any updates here? Also just to follow up, does this mean any training using this kernel (SFT, GRPO) should be avoided?

yinzhangyue

Hi~ Any support for the backward now? Or still aviod any training?

Sign up or log in to comment