Blueprint GQA description mistake?
#5
by
erclee2 - opened
Hello,
I noticed that the description of GQA and the diagram in the blueprint doesn't match the implementation in transformers. In the blueprint query heads are repeated, but in the implementation KV heads are repeated.
Just wanted to give a heads up!
Whoops, I see this is covered already here: https://huggingface.co/HuggingFaceTB/SmolLM3-3B/discussions/8
erclee2 changed discussion status to
closed