Are the nf4 weights needed?
#1
by sijeri4435 - opened
Hi, thank you for this upload, as I have a 5090 and therefore can't run the full model myself. I've downloaded your nf4 quants and the inference code provided, but I noticed that the nf4 models don't seem to be used. Correct me if I'm wrong, but it seems that the code quantizes the original models on-the-fly instead of using the pre-quantized models. If this is the case, are the weights necessary? Thank you for responding back.
hey just noticed this, i apologize because i was just using the repo to store individual pieces for some stuff i am doing locally. i've uploaded all the quantized weights though, you should be able to just clone the repo and run the generate prequant script now.
cahlen changed discussion status to closed