Spaces:
Running
Offering GGUF quantizations to save conversion hassle
Hi @Zeldeo ,
I recently performed a quantization on the Qwen 2B model to make it much easier to deploy and avoid compatibility issues with 16-bit files (such as qi_k_m.gguf formats).
I wanted to check with you: would you prefer that I send the quantized files over for you to host directly under your repository/organization, or would you rather I upload them on my end as community contributions and tag/link your original repository?
Either option works completely fine for me! I just want to make the quantized versions readily available to save people the hassle of converting the 16-bit files themselves.
Let me know what you prefer. Thanks!
Hi @aviel052 !
We do not currently offer GGUF versions.
We plan to look into this (particularly to determine whether, for example, it’s better to trim first and then convert to GGUF, or to convert to GGUF first and then trim).
When the time comes, we’ll likely offer this option for new models we release rather than for older ones already online on the organization.
This could change, but as things stand, I think it’s best if you host your GGUF checkpoint under your own name. People will be able to find it via the “Quantized” section of the Model Tree.
In any case, I’m glad to hear that our work is helpful to you :)