How could I use vllm to serve this gte model？
#25
by qiujiaji - opened
I try to use vllm to serve this model but found this 'NewModel' is not supported. I try to use vllm because I found it's a bit slow when I deploy gte with litserver: it reaches 1300 qps with 8 H20, which is only a bit faster than Qwen-0.5B. I think it should be a lot faster since gte only infers a embedding while Qwen-0.5B infers multiple tokens in my case.
qiujiaji changed discussion title from How to I use vllm to serve this gte model？ to How could I use vllm to serve this gte model？