Garbage output?
#1
by
droussis - opened
Only getting garbage output by this quant, no matter what configurations I try.
Anybody got this to work?
Same happened to me
Same. I can get it to load but nothing valid output. It's trying based on load, and vLLM log claims tokens are produced, but none are valid.
Hi，@droussis @ortegaalfredo @JDWarner
Thanks for raising this issue. We will delete this model and please try the new one here: https://huggingface.co/Intel/Step-3.5-Flash-int4-mixed-AutoRound