Instructions to use HuggingFaceH4/zephyr-7b-beta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceH4/zephyr-7b-beta with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="HuggingFaceH4/zephyr-7b-beta")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use HuggingFaceH4/zephyr-7b-beta with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceH4/zephyr-7b-beta"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceH4/zephyr-7b-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/HuggingFaceH4/zephyr-7b-beta

SGLang

How to use HuggingFaceH4/zephyr-7b-beta with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceH4/zephyr-7b-beta" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceH4/zephyr-7b-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceH4/zephyr-7b-beta" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceH4/zephyr-7b-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use HuggingFaceH4/zephyr-7b-beta with Docker Model Runner:
```
docker model run hf.co/HuggingFaceH4/zephyr-7b-beta
```

Zephyr-7B-alpha on LLM leaderboard, why not Zephyr-7b-beta?

#11

by YokaiKoibito - opened Nov 2, 2023

Discussion

YokaiKoibito

Nov 2, 2023

I'd love to get the LLM leaderboard's opinion on how Zephyr-7B-beta compares to other Mistral finetunes. Could we get it submitted for evaluation?

YokaiKoibito changed discussion title from Zephyr-7B-alpha in on LLM leaderboard, why not Zephyr-7b-beta to Zephyr-7B-alpha in on LLM leaderboard, why not Zephyr-7b-beta? Nov 2, 2023

YokaiKoibito changed discussion title from Zephyr-7B-alpha in on LLM leaderboard, why not Zephyr-7b-beta? to Zephyr-7B-alpha on LLM leaderboard, why not Zephyr-7b-beta? Nov 2, 2023

lewtun

Hugging Face H4 org Nov 4, 2023

Thanks for the reminder @YokaiKoibito ! I think we do have the evals, but forgot to make them public. Maybe @clefourrier or @SaylorTwift can add them now?

clefourrier

Hugging Face H4 org Nov 6, 2023

@lewtun Cannot find these evals in the private datasets, would you be OK resubmitting?

YokaiKoibito

Nov 7, 2023

•

edited Nov 7, 2023

@lewtun I'm still keen to see this ASAP: I'd love to move our use case up from Alpha to Beta.

clefourrier

Hugging Face H4 org Nov 8, 2023

@YokaiKoibito feel free to submit it if you'd like :)

YokaiKoibito

Nov 9, 2023

It's not my model, and anyway, it looks like it's already in the Pending Evaluation Queue (twice).

clefourrier

Hugging Face H4 org Nov 9, 2023

You can submit models that are not yours as long as they are public :)
If it's pending, it will get evaluated in the next weeks, since we are catching up on the queue now! And models appearing two times have usually been submitted at different precisions (it's explained in our FAQ if you want more details)
Closing then, since it's going to be evaluated!

clefourrier changed discussion status to closed Nov 9, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment