How to use ctransfomers with GGML files?
Tom, I have not been able to get GGML models to work with ctransformers or llama-cpp-python. They do work when I use LM Studio. I've tried several different ones and always get a nondescript error: "Failed to create LLM 'llama' from 'stablebeluga-7b.ggmlv3.q5_1.bin'." The .bin file is in the same directory and it shows up on a listdir(). What step am I missing?
code:
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained('stablebeluga-7b.ggmlv3.q5_1.bin', model_type='llama')
Tom, I have not been able to get GGML models to work with ctransformers or llama-cpp-python. They do work when I use LM Studio. I've tried several different ones and always get a nondescript error: "Failed to create LLM 'llama' from 'stablebeluga-7b.ggmlv3.q5_1.bin'." The .bin file is in the same directory and it shows up on a listdir(). What step am I missing?
code:
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained('stablebeluga-7b.ggmlv3.q5_1.bin', model_type='llama')
Transformers load Transformers models, GGML models are loaded my llama.cpp and its variants.