Instructions to use google/gemma-4-31B-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/gemma-4-31B-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/gemma-4-31B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("google/gemma-4-31B-it")
model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-31B-it", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
AMD Developer Cloud
Local Apps Settings

vLLM

How to use google/gemma-4-31B-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/gemma-4-31B-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-4-31B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/google/gemma-4-31B-it

SGLang

How to use google/gemma-4-31B-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/gemma-4-31B-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-4-31B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/gemma-4-31B-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-4-31B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use google/gemma-4-31B-it with Docker Model Runner:
```
docker model run hf.co/google/gemma-4-31B-it
```

When Gemma doesn't have the right tool schema it hallucinates it, create outputs of the fake-calls and use them as memory (WIP)

#136

by luislozanog86 - opened 1 day ago

Discussion

luislozanog86

1 day ago

During the Gemma X Cerebras hackathon I found out a Gemma 4 31B execution-state bug/failure. Then I tested it across diff providers (openrouter, cerebras, an running this version directly from aws gpus). When tools are required but no schemas exist, the model can emit tool-shaped text and later claim the unexecuted action succeeded: including invented command output. Reproducer: https://github.com/luislozanogmia/inference_experiments/tree/master/gemma4_tool_call_error.

Important: valid structured tool calling works when good and usable schemas are supplied

I am going to try to fine-tune this little cute AI but I need to test whether the cause is template/token history, especially the tool-response boundary or a localized internal circuit instead of burning GPUs on nothing.

If anyone is interested in joint-venture to solve this ping me, I believe Gemma 4 31-B is extremely good for open source but we need this fix to be more popular (and used in production without worries).

lgusm

Google org 1 day ago

Hi @luislozanog86 , thanks for raising this and providing the examples! That really helps!!

We’ve reviewed the code in your inference_experiments GitHub repository and ran identical tests natively locally. Based on those tests, we found that this is caused by an API integration mismatch rather than a model failure.

In the reproduction scripts provided, the raw <|tool_call> tags leak because the model is forcefully prompted to act as an execution agent (e.g., "You MUST use tools"), but the underlying API client or routing API (like OpenRouter) fails to actually attach the JSON tools schema array to the request payload. The model tries to obey your strict prompt, but without the native tooling schemas loaded in the backend, it hallucinates the internal tags.

How to resolve this:

1- Verify Tool Injection: Ensure your client/API wrapper is legitimately passing the tools payload array. We confirmed that if the tools are properly attached, even with a gentle or completely normal conversational prompt, the model natively recognizes its capabilities and seamlessly uses the tools via standard JSON function calls, entirely bypassing the hallucinated tags.

2- Soften the Prompt (Graceful Degradation): On the flip side, if you definitely aren't passing tool schemas to the API, ensure your system prompt isn't forcing it into an agentic loop. If you stick to standard conversational phrasing (e.g., "Create a Python script..." instead of "This is an execution task"), the model gracefully falls back to normal markdown code blocks without hallucinating the tags.

Let us know if you're still seeing this after verifying your client is successfully passing the tool schemas in the payload!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment