torch and llama.cpp integration
#1
by TobDeBer - opened
I just tried with latest torch/transformers and llama.cpp and inference failed.
Are there upstream branches I can use?
ValueError: The checkpoint you are trying to load has model type granitemoe but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
@TobDeBer you need this specific PR: https://github.com/huggingface/transformers/pull/33207/
the code for the MoE model is not merged into HF transformers main branch yet
Thanks!
It has been merged last night.
TobDeBer changed discussion status to closed
awesome