Frequent Error
I am getting this error :
TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=str, device=NoneType), but expected one of:
- (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
- (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
Tried on both collab instance : T4 GPU and v5e-1 TPU
Thanks for your interest in the model!
This error is caused by a known regression in recent versions of the transformers library (specifically versions 4.49.0+ and some late 4.48.x builds).
It's because model configuration file stores the data type as a string but the updated library code is passing this string directly to torch.empty().
The easiest would be do downgrade your transformers package until the fix (which already exists) is released or you can build the latest changes by building directly from github.
Let me know if that helps :)
HI. Are you planning to convert it to GGUF? THanks !
Hello, is it possible to run this on google colab T4? :)
import torch
import os
from PIL import Image
from transformers import AutoModelForCausalLM
os.environ["TORCH_USE_CUDA_DSA"] = "1"
moondream = AutoModelForCausalLM.from_pretrained(
"alecccdd/moondream3-preview-4bit",
trust_remote_code=True,
dtype=torch.float16, # Change from bfloat16
device_map={"": "cuda"},
)
moondream.compile() # Critical for fast inference
# Load an image
image = Image.open("smple.jpg")
# Ask a question
result = moondream.query(image=image, question="What's in this image?")
print(result["answer"])
Loading quantized weights from /root/.cache/huggingface/hub/models--alecccdd--moondream3-preview-4bit/snapshots/dab37eb440fbf6386be83ab47fac10f49a631729...
---------------------------------------------------------------------------
AcceleratorError Traceback (most recent call last)
/tmp/ipykernel_2215/2925464998.py in <cell line: 0>()
12 device_map={"": "cuda"},
13 )
---> 14 moondream.compile() # Critical for fast inference
15
16 # Load an image
9 frames
/usr/local/lib/python3.12/dist-packages/torchao/quantization/utils.py in pack_tinygemm_scales_and_zeros(scales, zeros, dtype)
356 guard_dtype_size(zeros, "zeros", dtype=dtype)
357 return (
--> 358 torch.cat(
359 [
360 scales.reshape(scales.size(0), scales.size(1), 1),
AcceleratorError: CUDA error: named symbol not found
Search for `cudaErrorSymbolNotFound' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.