Instructions to use Nexusflow/Athene-70B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nexusflow/Athene-70B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nexusflow/Athene-70B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Nexusflow/Athene-70B")
model = AutoModelForCausalLM.from_pretrained("Nexusflow/Athene-70B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nexusflow/Athene-70B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nexusflow/Athene-70B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Athene-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nexusflow/Athene-70B

SGLang

How to use Nexusflow/Athene-70B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nexusflow/Athene-70B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Athene-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nexusflow/Athene-70B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nexusflow/Athene-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nexusflow/Athene-70B with Docker Model Runner:
```
docker model run hf.co/Nexusflow/Athene-70B
```

Question Regarding the Legality of Using CC BY-NC 4.0 License in This Project

by jiacheo - opened Aug 14, 2024

Discussion

jiacheo

Aug 14, 2024

Hello,

First of all, thank you for your contributions to the community. I have been reviewing and using this project, and I noticed that it is based on Meta Llama 3 and has been released under the CC BY-NC 4.0 license. However, I have some concerns about the legality of using this license, especially in terms of compliance with the original project’s license.

According to the Meta Llama 3 Community License Agreement, this agreement grants users a “non-exclusive, worldwide, non-transferable, royalty-free limited license” to use, reproduce, distribute, copy, create derivative works of, and modify the Llama Materials.

However, the license also specifies the following requirements:

1.	Section 1.b: Redistribution and Use
•	“If you distribute or make available the Llama Materials (or any derivative works thereof), including another AI model, you shall provide a copy of this Agreement with any such Llama Materials; and prominently display ‘Built with Meta Llama 3’ on a related website, user interface, blog post, about page, or product documentation.”
•	The license does not impose general restrictions on commercial use, except as outlined in Section 2 regarding additional commercial terms.
2.	Section 2: Additional Commercial Terms
•	If an entity or its affiliates have more than 700 million monthly active users, they must request a special license from Meta to continue exercising the rights under this agreement.

Given these clauses, I have the following concerns:

1.	Commercial Use Restriction: By choosing the CC BY-NC 4.0 license, which explicitly restricts commercial use, there may be a conflict with the original Meta Llama 3 license, which does not impose a blanket restriction on commercial use except under specific conditions. Changing the license to CC BY-NC 4.0 might violate the terms of the Meta Llama 3 license, particularly in restricting the commercial use of derivative works.
2.	License Compatibility: The terms of the Meta Llama 3 license and those of CC BY-NC 4.0 might be incompatible, especially regarding the redistribution and use of derivative works. Could this lead to potential legal non-compliance with the Meta Llama 3 community license requirements?

I would appreciate clarification on whether explicit permission from Meta has been obtained to use the CC BY-NC 4.0 license for this project. If not, does this license choice align with the Meta Llama 3 community license?

Thank you for your hard work, and I hope this issue can be addressed to ensure the project remains compliant with all applicable licenses.

Best regards,
Thanks again!

banghua

Nexusflow org Aug 19, 2024

Thank you for raising this point. We have been working on this and just got to update our license here to be fully compliant with the llama3 license: Nexusflow Research License

banghua changed discussion status to closed Aug 19, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment