Instructions to use Trelis/Meta-Llama-3-8B-Instruct-function-calling with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Trelis/Meta-Llama-3-8B-Instruct-function-calling with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Trelis/Meta-Llama-3-8B-Instruct-function-calling")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Trelis/Meta-Llama-3-8B-Instruct-function-calling")
model = AutoModelForCausalLM.from_pretrained("Trelis/Meta-Llama-3-8B-Instruct-function-calling", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Trelis/Meta-Llama-3-8B-Instruct-function-calling with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Trelis/Meta-Llama-3-8B-Instruct-function-calling"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Trelis/Meta-Llama-3-8B-Instruct-function-calling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Trelis/Meta-Llama-3-8B-Instruct-function-calling

SGLang

How to use Trelis/Meta-Llama-3-8B-Instruct-function-calling with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Trelis/Meta-Llama-3-8B-Instruct-function-calling" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Trelis/Meta-Llama-3-8B-Instruct-function-calling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Trelis/Meta-Llama-3-8B-Instruct-function-calling" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Trelis/Meta-Llama-3-8B-Instruct-function-calling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use Trelis/Meta-Llama-3-8B-Instruct-function-calling with Docker Model Runner:
```
docker model run hf.co/Trelis/Meta-Llama-3-8B-Instruct-function-calling
```

not working well when function calling

by huayranus - opened Apr 23, 2024

Discussion

huayranus

Apr 23, 2024

I tried both curl command as well as openai chat completion calls, not working well, the generation varies

RonanMcGovern

Trelis org Apr 23, 2024

Thanks for testing this out.

Curl Commands
This is tricky to do because you need to have the prompt correctly formatted. One option is to copy-paste the sample formatted prompt on the model card and use that.
OpenAI Style requests
The problem you're having there is that the end token for Llama 3 is not the same as the eos_token in the tokenizer_config.json file. I've swapped the eos_token now, so if you re-run with an openai style endpoint (whether tgi or vllm) it should produce clean results.

Note still the following caveats:

The model correctly calls functions, but does not do a good job making use of function responses.
Unlike openchat 3.5 (the function calling fine-tune), the model is weak at chaining function calls.

huayranus

Apr 24, 2024

Thanks a lot, now it works!

RonanMcGovern changed discussion status to closed Apr 24, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment