dicta-il

/

dictalm2.0-instruct-AWQ

like 0

Follow

DICTA: The Israel Center for Text Analysis 149

[Using Ubuntu]
The quantized dicta-il/dictalm2.0-instruct-AWQ model doesn't run on CPU without GPU.
I get the following error:

File "/home/dev/.local/share/virtualenvs/new_final-QzMsWVq1/lib/python3.11/site-packages/awq/modules/linear/gemm_ipex.py", line 18, in __init__
    assert IPEX_INSTALLED, \
           ^^^^^^^^^^^^^^
AssertionError: Please install IPEX package with `pip install intel_extension_for_pytorch`.

Installing the intel_extension_for_pytorch doesn't make a difference and the error persists.
Let alone the fact that I'm using an AMD CPU.

Also tried using various autoawq versions including 0.27 and 0.28

Any chance you guys can fix it?

P.S. -- The full Dicta 7B dicta-il/dictalm2.0-instruct runs well on machines with CPU only (no GPU).