AWQ INT4?

#11
by garrettkolson - opened

Thanks for this - excited to try it out! You mentioned that there would be an official AWQ quant for this, do you know when that will be up?

SenseNova org

Thanks for your interest! We already have an official 4-bit AWQ version available here: https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ

Thank you! I see that the vLLM PR only supports the BF16 weights - will there be quantization support added eventually?

Sign up or log in to comment