Effects of quantization in Bloom models
#8
by elsatch - opened
Hi!
I have been trying the Flor model, following the instructions on the model card and performs ok. I have also tried quantizing it to Q4_0 and the output goes wild. Compared to other models like Mistral, the quantized results don't seem to be on par with the unquantized version.
Have you researched about the quantization effect in some models like BLOOM?
Thanks in advance!