TheBloke

/

phi-2-GPTQ

like 30

I have tried the Q3 and Q4 models. They fail to load with llama.cpp in the text gen web ui. I am on Linux.
In installed the text gen web ui yesterday so it should be up to date. Other models work fine.

This is the error from the phi-2.Q3_K_M.gguf model:
The error on the command line where server.py is running says: 2023-12-19 15:38:43 ERROR:Failed to load the model.
Traceback (most recent call last):
File "/home/somename/text-generation-webui/modules/ui_model_menu.py", line 210, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxx/text-generation-webui/modules/models.py", line 89, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/somename/text-generation-webui/modules/models.py", line 259, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/somename/text-generation-webui/modules/llamacpp_model.py", line 91, in from_pretrained
result.model = Llama(**params)
^^^^^^^^^^^^^^^
File "/home/somename/miniconda3/envs/textgen/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 957, in init
self._n_vocab = self.n_vocab()
^^^^^^^^^^^^^^
File "/home/somename/miniconda3/envs/textgen/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 2264, in n_vocab
return self._model.n_vocab()
^^^^^^^^^^^^^^^^^^^^^
File "/home/somename/miniconda3/envs/textgen/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 252, in n_vocab
assert self.model is not None
^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Exception ignored in: <function LlamaCppModel.__del__ at 0x7fd3c3130180>
Traceback (most recent call last):
File "/home/somename/text-generation-webui/modules/llamacpp_model.py", line 49, in del
del self.model
^^^^^^^^^^
AttributeError: 'LlamaCppModel' object has no attribute 'model'

========================

========================

========================

========================

The error in the web console says:
Traceback (most recent call last):

File "/home/somename/text-generation-webui/modules/ui_model_menu.py", line 210, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)

                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/somename/text-generation-webui/modules/models.py", line 89, in load_model

output = load_func_maploader

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/somename/text-generation-webui/modules/models.py", line 259, in llamacpp_loader

model, tokenizer = LlamaCppModel.from_pretrained(model_file)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/somename/text-generation-webui/modules/llamacpp_model.py", line 91, in from_pretrained

result.model = Llama(**params)

           ^^^^^^^^^^^^^^^

File "/home/somename/miniconda3/envs/textgen/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 957, in init

self._n_vocab = self.n_vocab()

            ^^^^^^^^^^^^^^

File "/home/somename/miniconda3/envs/textgen/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 2264, in n_vocab

return self._model.n_vocab()

   ^^^^^^^^^^^^^^^^^^^^^

File "/home/somename/miniconda3/envs/textgen/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 252, in n_vocab

assert self.model is not None

   ^^^^^^^^^^^^^^^^^^^^^^

AssertionError