Hugging Face's logo

TheBloke
/
Phind-CodeLlama-34B-v2-AWQ

Text Generation
Transformers
Safetensors
llama
code llama
Eval Results (legacy)
text-generation-inference
4-bit precision
awq
Model card Files
xet
Community
3

I get repeated gibberish output all the time.

#3
by pootow - opened
Discussion
pootow

I am using text-generation-webui to test this model. whatever the temperature, top_p, etc. etc. is tuned, the model just gives repeated gibberish output after few "normal" words.

why would it be? any suggestions?
deleted

Try a different prompt? I have found a lot of models dont work right with the 'suggested' prompts. I have the best luck with VIcuna1.1 even when it says use something like llama .

pootow

@Nurb432 thanks for your suggestion. It is not the problem of the prompt, I downloaded the GPTQ version, and all things are back to normal and even the inference speed is getting up.
pootow

I find the problem, I should not load the model using AutoAWQ loader, but use huggingface transformer loader, evething is just fine.

pootow changed discussion status to closed

Sign up or log in to comment