Quantization support
Would there be any awq or bitsandbytes quantization support for this model?
Most likely, in a month and a half they will add GPTQ-Int8 and int4
When can I expect the quantized model
When can I expect the quantized model
I have successfully applied Hunyuan’s new SiNQ quantization technique to the 9B and 2B models, but only to the LLM component. To make inference feasible on my hardware, I built a custom “franken-inference” pipeline. The quantized LLM runs on my RTX 4070 Ti, while the visual encoder is offloaded to a secondary RTX 2070. Without this split, the full model would not fit in VRAM even at 4-bit quantization.
Right now, the implementation isn’t heavily optimized and could use some cleanup, but it’s fully functional. During inference, the 9B quantized LLM uses under 9GB of VRAM on the 4070 Ti. The model weights are around 7GB, and the KV cache adds another 2GB. Meanwhile, the visual encoder runs smoothly on the 2070, consuming under 5GB of VRAM.
The current quantization level is approximately 4-bit, but I have headroom to go higher, potentially to 5-bit or even 6-bit, given my 12GB VRAM budget. I’m also exploring adding FlashAttention support to further reduce memory pressure and improve inference speed.
The visual encoder can even be offloaded to the CPU if needed. It’s still surprisingly fast, and this would free up more GPU memory for longer sequences. But it can also be run on the same GPU as the LLM part.
If you’re interested in the code, setup details, or want to collaborate on optimizing it further, I’d love to hear from you!
There is on big issue in my code and its system ram usage, that one spikes a LOT so below 64gb might not be enough atm, im working on a fix though (;
EDIT: Found a massive bug that makes inference speed crazy slow /:
When can I expect the quantized model
I have successfully applied Hunyuan’s new SiNQ quantization technique to the 9B and 2B models, but only to the LLM component. To make inference feasible on my hardware, I built a custom “franken-inference” pipeline. The quantized LLM runs on my RTX 4070 Ti, while the visual encoder is offloaded to a secondary RTX 2070. Without this split, the full model would not fit in VRAM even at 4-bit quantization.
Right now, the implementation isn’t heavily optimized and could use some cleanup, but it’s fully functional. During inference, the 9B quantized LLM uses under 9GB of VRAM on the 4070 Ti. The model weights are around 7GB, and the KV cache adds another 2GB. Meanwhile, the visual encoder runs smoothly on the 2070, consuming under 5GB of VRAM.
The current quantization level is approximately 4-bit, but I have headroom to go higher, potentially to 5-bit or even 6-bit, given my 12GB VRAM budget. I’m also exploring adding FlashAttention support to further reduce memory pressure and improve inference speed.
The visual encoder can even be offloaded to the CPU if needed. It’s still surprisingly fast, and this would free up more GPU memory for longer sequences. But it can also be run on the same GPU as the LLM part.
If you’re interested in the code, setup details, or want to collaborate on optimizing it further, I’d love to hear from you!
There is on big issue in my code and its system ram usage, that one spikes a LOT so below 64gb might not be enough atm, im working on a fix though (;
EDIT: Found a massive bug that makes inference speed crazy slow /:
okay i fix the main bug, now everything works, though it takes a lot of ram usage still but inference works on my 12gb vram card (;
When can I expect the quantized model
If you cant wait you can use this one, its a bit messy and stuff but it should be possible to get it running (;
https://huggingface.co/wsbagnsv1/Ovis2.5-9B-sinq-4bit-experimental