Endless Repetition? Anyone encountered?
#48
by
evilperson068
- opened
My model repeats for ever the same text block, even for bf16. I am using ollama with open-webui.
You should set the temperature to 1. Also, since you can use BF16, you might want to try sglang first?
GOODE
You should set the temperature to 1. Also, since you can use BF16, you might want to try sglang first?
Thank you for your reply.
However, neither sglang nor vllm worked for me, even followed the exact steps to install dependencies. And ollama TTFT is very slow and easily eats up all resources.
I think something didn't work well, either software-support or dependencies.