Instructions to use google/gemma-4-31B-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/gemma-4-31B-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/gemma-4-31B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("google/gemma-4-31B-it")
model = AutoModelForImageTextToText.from_pretrained("google/gemma-4-31B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
AMD Developer Cloud
Local Apps Settings

vLLM

How to use google/gemma-4-31B-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/gemma-4-31B-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-4-31B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/google/gemma-4-31B-it

SGLang

How to use google/gemma-4-31B-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/gemma-4-31B-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-4-31B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/gemma-4-31B-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-4-31B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use google/gemma-4-31B-it with Docker Model Runner:
```
docker model run hf.co/google/gemma-4-31B-it
```

fix: chat template — null handling, reasoning preservation, turn-tag balance, input validation

#118

by lucianommartins - opened about 7 hours ago

base: refs/heads/main

←

from: refs/pr/118

Discussion Files changed

+65

-41

lucianommartins

Google org about 7 hours ago

Summary

Improves Gemma4 chat template:

Bug fixes

None values now render as null instead of Python's None
String-typed tool_calls[].function.arguments now raises a clear error instead of silently producing malformed DSL
Prior-turn reasoning/thinking is preserved across multi-turn tool-call chains (preserve_thinking flag, default=true)
Consecutive assistant messages now produce balanced <|turn>model/<turn|> tags via forward-scan continuation detection

Improvements

enable_thinking normalized once with | default(false), eliminating repetitive is defined and checks
image_url and input_audio content types now map to <|image|> and <|audio|> (OpenAI compatibility)
Empty messages=[] handled gracefully instead of crashing
Unmatched tool_call_id in tool responses falls back to 'unknown' instead of crashing
Consistent .get() access prevents StrictUndefined errors for optional message keys
O(1) backward scan for model-turn continuation (was O(n) per message)

fix: chat template — null handling, reasoning preservation, turn-tag balance, input validation03cf5a16

tapu1125

about 6 hours ago

•

edited about 6 hours ago

Thanks for the commit, I hope this template changes fix premature ending problems. First in the morning to apply these new changes!
I am convincing teams that we can fix the issues, we should not give up and transfer to Qwen models lol. Cannot let gemma4 go.

jscott3201

about 5 hours ago

I've been sharing this publicly which it seems you fix a few things here as well: https://gist.github.com/jscott3201/ad69c4ffbd79f18b11a0f6a94c94fadf

I can move this to a repo with MIT/Apache2 if you want to pull anything off this template for the official template.

seamon67

about 5 hours ago

I've been sharing this publicly which it seems you fix a few things here as well: https://gist.github.com/jscott3201/ad69c4ffbd79f18b11a0f6a94c94fadf

I can move this to a repo with MIT/Apache2 if you want to pull anything off this template for the official template.

I have been running your template but can you now comment on which is better - Google's new one or yours and will you be coming up with a new one incorporating google's changes?

jscott3201

about 5 hours ago

•

edited about 5 hours ago

I would have to diff them but they designed the LLM so I'll lean towards theirs being correct! Their 4 noted issues align with my callouts in the header.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment