Instructions to use Nexusflow/Athene-V2-Chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nexusflow/Athene-V2-Chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nexusflow/Athene-V2-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Nexusflow/Athene-V2-Chat")
model = AutoModelForCausalLM.from_pretrained("Nexusflow/Athene-V2-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nexusflow/Athene-V2-Chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nexusflow/Athene-V2-Chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Athene-V2-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nexusflow/Athene-V2-Chat

SGLang

How to use Nexusflow/Athene-V2-Chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nexusflow/Athene-V2-Chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Athene-V2-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nexusflow/Athene-V2-Chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Athene-V2-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nexusflow/Athene-V2-Chat with Docker Model Runner:
```
docker model run hf.co/Nexusflow/Athene-V2-Chat
```

Draft Model of Speculative Decoding

by nagug - opened Feb 11, 2025

Discussion

nagug

Feb 11, 2025

Do you have any suggestions of which draft models would play nicely with this mode. BTW. Qwen2.5 7B instruct seem to have different vocab size and not working. May be i am doing something wrong.

mhenning

Feb 27, 2025

According to the models config.json (https://huggingface.co/Nexusflow/Athene-V2-Chat/blob/main/config.json) and (https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/config.json), the vocab_size is the same. As this is a finetune of Qwen 2.5 72B, another Qwen 2.5 model as draft model makes the most sense. Do you maybe use quantized versions which report a different vocabulary size?

nagug

Feb 27, 2025

Ah.. that makes sense. i was using a AWQ version using VLLM. (as i have limit for GPU). does that mean i should try with a AWQ version of Qwen 7b? not sure that will bring any improvement..

mhenning

Feb 27, 2025

I can't speak for this model, but I tried with Llama 3.3 70B AWQ and Llama 3.2 3B AWQ as draft model, and while it was running on vLLM, I got less tokens/sec than without the draft model. I'm not sure yet why. The acceptance rate was okay-ish with 0.72.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment