Can it run on CPU?

by BastiNi - opened Oct 28, 2022

•

I tried to run the model on my local laptop, but I keep getting the following error on inference:

RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

I'm loading the model using

model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", revision="float16", torch_dtype=torch.float16, low_cpu_mem_usage=True)

I tried several PyTorch and hugging face versions, including the ones from this repo (torch==1.9.1, transformers==4.12.3).

Maybe the model is not supporting CPU inference?

EleutherAI org Oct 28, 2022

Try using the float32 version if you want to do CPU inference

stellaathena changed discussion status to closed Dec 8, 2022

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment