Instructions to use NexaAI/Octopus-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NexaAI/Octopus-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NexaAI/Octopus-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NexaAI/Octopus-v2")
model = AutoModelForCausalLM.from_pretrained("NexaAI/Octopus-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Local Apps Settings

vLLM

How to use NexaAI/Octopus-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NexaAI/Octopus-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NexaAI/Octopus-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/NexaAI/Octopus-v2

SGLang

How to use NexaAI/Octopus-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NexaAI/Octopus-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NexaAI/Octopus-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NexaAI/Octopus-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NexaAI/Octopus-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use NexaAI/Octopus-v2 with Docker Model Runner:
```
docker model run hf.co/NexaAI/Octopus-v2
```

Octopus-v2

Commit History

Sintayew4

0f9f2a8
verified

Sintayew4 commited on Apr 12, 2024

Update README.md

0e82a0e
verified

alexchen4ai commited on Apr 12, 2024

Update README.md

5bfa5fe
verified

alexchen4ai commited on Apr 12, 2024

Update README.md

b8b5434
verified

alexchen4ai commited on Apr 12, 2024

Update README.md

103d19a
verified

alexchen4ai commited on Apr 12, 2024

update

0bef7a2
verified

alexchen4ai commited on Apr 12, 2024

polish

a26d9d4
verified

alexchen4ai commited on Apr 6, 2024

update

942a5e6
verified

alexchen4ai commited on Apr 6, 2024

Update README.md

a291655
verified

alexchen4ai commited on Apr 6, 2024

Update README.md

c2ec018
verified

alexchen4ai commited on Apr 6, 2024

Upload android_benchmark.xlsx

74bc7c1
verified

nexa4ai commited on Apr 6, 2024

Fix model ID (#1)

d65d651
verified

alexchen4ai

osanseviero commited on Apr 6, 2024

GemmaForCausalLM import error fix (#3)

9572755
verified

alexchen4ai

Tonic commited on Apr 6, 2024