Is there a way to quantize this model
#2
by ParisNeo - opened
Hi there.
Your model is really cool. I have added it to lollms so that people can use it to interact with images.
But I don't have a big GPU and it is making my GPU suffer.
I wonder if it is in anysense possible to quantize the model to 4bits using GPTQ? I know it is possible with llama models.
Best regards