Hugging Face's logo

chainyo
/
alpaca-lora-7b

PEFT
PyTorch
Safetensors
English
llama
alpaca
lora
Model card Files
xet
 Community
10

How to run on CPU?

#2
by MariaM07 - opened
Discussion
MariaM07

Hi!

I'm on Windows, when I tried to run it on GPU I got the error: "CUDA out of memory" (GPU memory = 18 GB).

I tried running on CPU doing:

device = "cpu"
input_ids = input_ids.to(device)
model = model.to(device)

Then I get the error: Exception has occurred: NotImplementedError
Cannot copy out of meta tensor; no data!
File "C:\generate_alpaca7B_cpu.py", line 54, in
model = model.to(device)
NotImplementedError: Cannot copy out of meta tensor; no data!

Any solution, please?
chainyo
 Owner

Are you using WSL2 on Windows?
MariaM07

I'm not using it, I'll try it, thanks!
chainyo
 Owner

Hi @MariaM07 , did you solve your issue running this model on Windows?
MariaM07

Hi @chainyo , I couldn't run it on Windows because I had problemsn with the bitsandbytes library. I think I don't have GPU memory anyway, actually I have 6GB memory GPU.
chainyo
 Owner

Ok, I understand, Windows is not so great to experiment with bitsandbytes and GPU in general (outside gaming). But you should be able to run it on CPU without bitsandbytes if at least you have enough RAM.

Sign up or log in to comment