Getting this to run in Ooba? Anyone know what settings I have to choose?
It seems Ooba got some updates talking about Minotaur 8K? But I can't get either the GPTQ or GGML versions of this model to run in Ooba even after update? I can get up to 65B models to run so it's not a VRAM or RAM issue? Any ideas? Or is this model still not supported? Or special setup steps?
Minotaur is a non-Llama model. text-generation-webui doesn't currently support non-Llama models in GGML format. So this model just won't work in text-generation-webui.
The updates you're describing would apply to the GPTQ version of Minotaur, which text-generation-webui does support. It supports all model types in GPTQ format, but only Llama models in GGML.
Thanks yes I tried getting it working in GPTQ and got weird errors. Like I downloaded it in both GPTQ and GGML. Thanks I will check into it more later. Do you think it's likely or possible for Ooba to get GGML support for models like this? Or do you really have no opinion or idea? Thanks again.
It's certainly possible for text-generation-webui to support it. He would need to add support for another backend, one that supports all GGMLs. ctransformers would be a good choice for that: https://github.com/marella/ctransformers
If someone did a PR to implement that he'd probably include it. But I don't have time to do that myself at the moment.
That's all the questions I have and I would never ask you to go submit code to Ooba when your as busy as you are. If you do it one day because you want to so be it. But I imagine or at least hope that someone will get around to it. lol Thanks for answering my questions TheBloke. Your putting out amazing work quantizing all these models that other people are making. I don't know if that is the correct term. Nor do I want to take up any more of your time so just take this as a thanks you don't need to reply.
Yup that's the correct term! You're welcome, glad you're finding them useful.