GGUF / Ollama version?
Hey, this looks really awesome! I tried it out and its been very helpful so far.
is there any chance we could get a GGUF version, or get a release for ollama? I'd like to be able to use it in openwebui, and ollama is the easiest way. But if there was a GGUF version I could directly import it that way also.
Thanks!
I'll pitch in for you. I'll post on GGUF-A-Lot org with a nice set of them. I would need better hardware to do an imatrix but I might just see about it. I will do --leave-output-tensors, with f16, Q8_0, Q6_k, Q4_KM but I don't suggest using a Q4 personally.
https://huggingface.co/GGUF-A-Lot
Hey, this looks really awesome! I tried it out and its been very helpful so far.
is there any chance we could get a GGUF version, or get a release for ollama? I'd like to be able to use it in openwebui, and ollama is the easiest way. But if there was a GGUF version I could directly import it that way also.Thanks!