Instructions to use google/gemma-3n-E4B-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/gemma-3n-E4B-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/gemma-3n-E4B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("google/gemma-3n-E4B-it")
model = AutoModelForMultimodalLM.from_pretrained("google/gemma-3n-E4B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/gemma-3n-E4B-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/gemma-3n-E4B-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-3n-E4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/google/gemma-3n-E4B-it

SGLang

How to use google/gemma-3n-E4B-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/gemma-3n-E4B-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-3n-E4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/gemma-3n-E4B-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-3n-E4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use google/gemma-3n-E4B-it with Docker Model Runner:
```
docker model run hf.co/google/gemma-3n-E4B-it
```

Gemma3n not working on H20 with bfloat16 data type.

#30

by NOWSHAD - opened Jul 16, 2025

Discussion

NOWSHAD

Jul 16, 2025

•

edited Jul 16, 2025

I tried running Gemma3n model on H20 card with bfloat16 data type and it throws floating point exception and fails. Similarly I tried float32 data type and it works.
The example I'm trying is the same one present in the sample notebook.
On the other hand if I use the same example from the model card, bfloat16 fails with floating point exception (same as earlier) but flaot32 fails with below error

Unsupported: call_method GetAttrVariable(UserDefinedObjectVariable(AttentionInterface), _global_mapping) __getitem__ (ConstantVariable(),) {}

I tried the same in H100 and it's working as expected.

Anyone else faced this issue?

BalakrishnaCh

Google org Jul 24, 2025

Hi @NOWSHAD ,

Welcome to Google Gemma family of open source models, the above issues might be because of the compatibility of floating point operation on a particular hardware. Please find the following suggestion to avoid such kind of issues:

Update Everything: Ensure your NVIDIA drivers, CUDA toolkit, cuDNN, PyTorch, transformers, and bitsandbytes libraries are all at their absolute latest stable versions. This is the most common fix.
Check Compatibility Matrix: Verify if your specific driver and CUDA versions are officially recommended for PyTorch on Hopper GPUs.

How the model is loaded (from_pretrained arguments), model.generate() is called (e.g., do_sample=False vs. True, specific generation parameters).

Thanks.

CHNtentes

Jul 25, 2025

I tried running Gemma3n model on H20 card with bfloat16 data type and it throws floating point exception and fails. Similarly I tried float32 data type and it works.
The example I'm trying is the same one present in the sample notebook.
On the other hand if I use the same example from the model card, bfloat16 fails with floating point exception (same as earlier) but flaot32 fails with below error
Unsupported: call_method GetAttrVariable(UserDefinedObjectVariable(AttentionInterface), _global_mapping) __getitem__ (ConstantVariable(),) {}
I tried the same in H100 and it's working as expected.

Anyone else faced this issue?

For core dump with H20, check this: https://github.com/vllm-project/vllm/issues/4392#issuecomment-2227935528

NOWSHAD

Jul 25, 2025

Thank you @BalakrishnaCh @CHNtentes for your inputs. Yes, after playing around with the generation config and updating the packages resolved the issue.

NOWSHAD changed discussion status to closed Jul 25, 2025

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment