GGUF Version of the best 7B LLM!
First of all: Thank you for this awesome model! It seems to perform really well. Small models are great, as one can run them locally. =D
@TheBloke Could you create a gguf Version of this model?
(By the way, I have access to a computer with two rtx 3090 - I am not quite sure how to create gguf versions, but if its doable, I could perhaps help.)
I could also use the old server of a company of a friend. It is equipped with 3 M40 GPUs. While beeing a bit old, they still got some Vram. Don't know if this is usefull.
Thanks @SimSim93 ! I'm currently evaluating a DPO version of this model, it should be even better.
If you want to make GGUF versions of a 7B model, you don't need any big hardware. I created this notebook to automate this process (T4 GPU): https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu#scrollTo=fD24jJxq7t3k
I'm currently evaluating a DPO version of this model, it should be even better.
Not much Difference in Scores.