Instructions to use google/recurrentgemma-9b-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/recurrentgemma-9b-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="google/recurrentgemma-9b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("google/recurrentgemma-9b-it")
model = AutoModelForCausalLM.from_pretrained("google/recurrentgemma-9b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/recurrentgemma-9b-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/recurrentgemma-9b-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/recurrentgemma-9b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/google/recurrentgemma-9b-it

SGLang

How to use google/recurrentgemma-9b-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/recurrentgemma-9b-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/recurrentgemma-9b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/recurrentgemma-9b-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/recurrentgemma-9b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use google/recurrentgemma-9b-it with Docker Model Runner:
```
docker model run hf.co/google/recurrentgemma-9b-it
```

RecurrentGemmaForCausalLM.forward() got an unexpected keyword argument 'position_ids'

#16

by FreeHugsForRobots - opened Oct 12, 2024

Discussion

FreeHugsForRobots

Oct 12, 2024

This error happens when running the model on a HuggingFace dedicated endpoint (AWS US-East-1, A100), for any input including the provided examples.

Is the configuration not using the latest Huggingface version?

Renu11

Google org Oct 14, 2024

@FreeHugsForRobots , It seems the Transformers version is not updated to this HF endpoint. Please try again by installing the latest Transformers version and let us know if the issue still persists. You can upgrade the Transformers version using !pip install --upgrade transformers or !pip install -U transformers.

nbroad

Oct 18, 2024

If you add a requirements.txt file to the repo with an updated version of transformers, it should work. I created a PR that has this file. When deploying to inference endpoints, you can specify
revision=37ad739d64675c6cbd90f9e9897d8aa85a35b814 in the advanced configuration

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment