ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models
Hello, I have been using Google Colab for a while, and it worked well until recently when I encountered this error while running it. Do you know why this might be happening?
Traceback (most recent call last):
File "/content/drive/MyDrive/joy-caption-pre-alpha/app.py", line 179, in
main()
File "/content/drive/MyDrive/joy-caption-pre-alpha/app.py", line 149, in main
models = load_models()
File "/content/drive/MyDrive/joy-caption-pre-alpha/app.py", line 45, in load_models
text_model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, device_map="auto", torch_dtype=torch.bfloat16).eval()
File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
return model_class.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3990, in from_pretrained
dispatch_model(model, **device_map_kwargs)
File "/usr/local/lib/python3.10/dist-packages/accelerate/big_modeling.py", line 498, in dispatch_model
model.to(device)
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2839, in to
raise ValueError(
ValueError: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype.
any solution to this?