Instructions to use gorilla-llm/gorilla-openfunctions-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use gorilla-llm/gorilla-openfunctions-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="gorilla-llm/gorilla-openfunctions-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gorilla-llm/gorilla-openfunctions-v2")
model = AutoModelForCausalLM.from_pretrained("gorilla-llm/gorilla-openfunctions-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use gorilla-llm/gorilla-openfunctions-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gorilla-llm/gorilla-openfunctions-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorilla-llm/gorilla-openfunctions-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/gorilla-llm/gorilla-openfunctions-v2

SGLang

How to use gorilla-llm/gorilla-openfunctions-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "gorilla-llm/gorilla-openfunctions-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorilla-llm/gorilla-openfunctions-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "gorilla-llm/gorilla-openfunctions-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorilla-llm/gorilla-openfunctions-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use gorilla-llm/gorilla-openfunctions-v2 with Docker Model Runner:
```
docker model run hf.co/gorilla-llm/gorilla-openfunctions-v2
```

Updated with a working chat_template

by CISCai - opened Mar 2, 2024

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

-1

CISCai

Mar 2, 2024

After the recent README.md changes and some experimentation I have come up with the following chat_template, which I'm currently using in my GGUFs.

Minor fix for #3, now only generates functions in last message to not waste multi-turn context.

Fixes #2.

Updated with a working chat_template1de4d721

CharlieJi

Gorilla LLM (UC Berkeley) org Apr 18, 2024

Thanks for contributing! We really appreciate it. Sorry for the late response! We found that different chat templates are used in various packages, so we haven't updated the official chat_template and recommend developers use our official get_prompt(.) function instead as ground truth to avoid inconsistencies. We'll update the system prompt in chat_template from Deepseek to Gorilla LLM. Thanks for catching this again!

We'll close this issue. Let us know if you have additional questions / concerns, we'll reopen this thread. Thanks again!

CharlieJi changed pull request status to closed Apr 18, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment