Instructions to use google/gemma-1.1-7b-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/gemma-1.1-7b-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="google/gemma-1.1-7b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("google/gemma-1.1-7b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-1.1-7b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use google/gemma-1.1-7b-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/gemma-1.1-7b-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-1.1-7b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/google/gemma-1.1-7b-it

SGLang

How to use google/gemma-1.1-7b-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/gemma-1.1-7b-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-1.1-7b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/gemma-1.1-7b-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-1.1-7b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use google/gemma-1.1-7b-it with Docker Model Runner:
```
docker model run hf.co/google/gemma-1.1-7b-it
```

TemplateError: System role not supported

#15

by luogy - opened Apr 12, 2024

Discussion

luogy

Apr 12, 2024

The question is the same as the title.
Does Gemma-1.1 not support{"role": "system", "content": system_prompt}？

     27 messages = [
     28     {
     29         "role": "system",
   (...)
     33     {"role": "user", "content": user_input},
     34 ]
---> 36 prompt = pipe.tokenizer.apply_chat_template(messages,
     37                                             tokenize=False,
     38                                             add_generation_prompt=True)
     41 outputs = pipe(prompt,
     42                max_new_tokens=256,
     43                do_sample=True,
     44                temperature=0.7,
     45                top_k=50,
     46                top_p=0.95)
     48 generated_outputs = outputs[0]["generated_text"]
...
File ~/mambaforge/envs/ollama/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1828, in PreTrainedTokenizerBase._compile_jinja_template.<locals>.raise_exception(message)
   1827 def raise_exception(message):
-> 1828     raise TemplateError(message)

TemplateError: System role not supported

ThinkingRock

Apr 27, 2024

I am also having this issue

gusthema

Google org Apr 29, 2024

System instructions is not supported directly but you can start the prompt with the instruction "you are an expert in computer and ..." or something like that and the model will follow

areumtecnologia

May 2, 2024

•

edited May 2, 2024

Hello!
This way, the user will be able to modify the original purposes of the model. Ideally, the user should not be able to modify the model's operating principles. How to do this?

seoilgun

Jun 26, 2024

•

edited Jun 26, 2024

omg

azrai99

Jun 26, 2024

This comment has been hidden

MKR-AI

Dec 6, 2024

is there any update on this?

lkv

Google org Jan 8, 2025

Hi @luogy ,

Gemma-1.1 likely doesn't support the "system" role format used by other models. The error is being raised during the call to apply_chat_template(), which suggests that the tokenizer or template handler does not accept "system" as a valid role. If you want the model to follow certain instructions or be in a specific role, include those instructions directly in the user_input instead of using a separate system role.

Thank you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment