Can this fit into 12gb vram?
#2
by telvenes - opened
Can this fit into 12gb vram?
Yes, but for 12GB VRAM, using https://huggingface.co/baicai1145/s2-pro-w4a16-late7attn-ffnmix gives better results.
With vllm? How do yoy serve it.?
https://github.com/baicai-1145/sglang-omni I submitted a PR to sglang-omni: https://github.com/sgl-project/sglang-omni/pull/150, but it has not been merged for the time being.