Instructions to use ScaleGenAI/Llama3-8B-Function-Calling with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ScaleGenAI/Llama3-8B-Function-Calling with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ScaleGenAI/Llama3-8B-Function-Calling")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ScaleGenAI/Llama3-8B-Function-Calling")
model = AutoModelForCausalLM.from_pretrained("ScaleGenAI/Llama3-8B-Function-Calling")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ScaleGenAI/Llama3-8B-Function-Calling with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ScaleGenAI/Llama3-8B-Function-Calling"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ScaleGenAI/Llama3-8B-Function-Calling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ScaleGenAI/Llama3-8B-Function-Calling

SGLang

How to use ScaleGenAI/Llama3-8B-Function-Calling with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ScaleGenAI/Llama3-8B-Function-Calling" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ScaleGenAI/Llama3-8B-Function-Calling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ScaleGenAI/Llama3-8B-Function-Calling" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ScaleGenAI/Llama3-8B-Function-Calling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ScaleGenAI/Llama3-8B-Function-Calling with Docker Model Runner:
```
docker model run hf.co/ScaleGenAI/Llama3-8B-Function-Calling
```

tejeshbhalla commited on Apr 26, 2024

Commit

2057304

verified ·

1 Parent(s): 32ad6d4

Update README.md

Browse files

Files changed (1) hide show

README.md +39 -0

README.md CHANGED Viewed

@@ -1,3 +1,42 @@
 ---
 license: llama3
 ---

 ---
 license: llama3
 ---
+This model is intended for use in environments where automated function calling capabilities are required to enhance data manipulation and retrieval tasks. It is particularly useful in scenarios involving complex data analysis, where users can query data interactively through natural language commands.
+The model has a specific format for tool calling that is :
+<|begin_of_text|>
+<|start_header_id|>system<|end_header_id|>
+        Only use these tools while answering
+        **python_repl_tool** - Run python code on the dataframe.
+          Parameters:
+            code: python code to run in the exec function.
+            chain_of_thought: chain of thought behind using the tool
+this is what my dataframe looks like :
+                                  Financials                 Metrics          Flags
+                          Revenue    Expenses    Profit    ROI    Valid    Outlier
+Year Quarter Region
+2021 Q1      North     120000.0     85000.0     35000.0    0.29   True     False
+             South     95000.0      55000.0     40000.0    0.42   False    False
+     Q2      North     130000.0     60000.0     70000.0    0.54   True     True
+             South     100000.0     70000.0     30000.0    0.30   True     False
+2022 Q1      North     125000.0     75000.0     50000.0    0.40   False    False
+<|eot_id|>
+<|start_header_id|>user<|end_header_id|>
+What is year 2021 quarter 1 revenue for region North??
+<|eot_id|>
+<|start_header_id|>assistant<|end_header_id|>
+function_call: {"name": "python_repl_tool", "arguments": {"code": "df.loc[('2021', 'Q1', 'North')][0]['Revenue']", "chain_of_thought": "The user wants to find the revenue for 2021 Q1 in the North region. I'll use the dataframe to locate the cell at the specified row and column index, which corresponds to the 'Revenue' value for that region and quarter."}}
+<|eot_id|>
+<|start_header_id|>user<|end_header_id|>
+function_response: {"answer": "80000"}
+<|eot_id|>
+<|start_header_id|>assistant<|end_header_id|>
+Ai will respond with function_call if it needs to call a function
+User will respond with either error or the response if it was a tool like : function_response {json response}