Instructions to use Open-Orca/Mistral-7B-OpenOrca with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Open-Orca/Mistral-7B-OpenOrca with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Open-Orca/Mistral-7B-OpenOrca")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Open-Orca/Mistral-7B-OpenOrca")
model = AutoModelForCausalLM.from_pretrained("Open-Orca/Mistral-7B-OpenOrca", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Open-Orca/Mistral-7B-OpenOrca with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Open-Orca/Mistral-7B-OpenOrca"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Open-Orca/Mistral-7B-OpenOrca",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Open-Orca/Mistral-7B-OpenOrca

SGLang

How to use Open-Orca/Mistral-7B-OpenOrca with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Open-Orca/Mistral-7B-OpenOrca" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Open-Orca/Mistral-7B-OpenOrca",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Open-Orca/Mistral-7B-OpenOrca" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Open-Orca/Mistral-7B-OpenOrca",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Open-Orca/Mistral-7B-OpenOrca with Docker Model Runner:
```
docker model run hf.co/Open-Orca/Mistral-7B-OpenOrca
```

bleysg commited on Oct 10, 2023

Commit

0b9c924

1 Parent(s): 7233ac8

Update README.md

Browse files

Update with official HF Leaderboard scores.

Files changed (1) hide show

README.md +8 -8

README.md CHANGED Viewed

@@ -24,7 +24,7 @@ We use [OpenChat](https://huggingface.co/openchat) packing, trained with [Axolot
 This release is trained on a curated filtered subset of most of our GPT-4 augmented data.
 It is the same subset of our data as was used in our [OpenOrcaxOpenChat-Preview2-13B model](https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-13B).
-**HF Leaderboard evals place this model as #2 for all models smaller than 30B at release time, outperforming all but one 13B model.**
 This release provides a first: a fully open model with class-breaking performance, capable of running fully accelerated on even moderate consumer GPUs.
 Our thanks to the Mistral team for leading the way here.
@@ -112,20 +112,20 @@ pip install git+https://github.com/huggingface/transformers
 ## HuggingFace Leaderboard Performance
 We have evaluated using the methodology and tools for the HuggingFace Leaderboard, and find that we have dramatically improved upon the base model.
-We find **105%** of the base model's performance on HF Leaderboard evals, averaging **65.33**.
-At release time, this beats all 7B models, and all but one 13B.
 ![HF Leaderboard](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca/resolve/main/Images/MistralOrca7BHFLeaderboard.png)
 | Metric | Value |
 |-----------------------|-------|
-| MMLU (5-shot)         | 61.73 |
-| ARC (25-shot)         | 63.57 |
-| HellaSwag (10-shot)   | 83.79 |
-| TruthfulQA (0-shot)   | 52.24 |
-| Avg.                  | 65.33 |
 We use [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to run the benchmark tests above, using the same version as the HuggingFace LLM Leaderboard.

 This release is trained on a curated filtered subset of most of our GPT-4 augmented data.
 It is the same subset of our data as was used in our [OpenOrcaxOpenChat-Preview2-13B model](https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-13B).
+**HF Leaderboard evals place this model as #1 for all models smaller than 30B at release time, outperforming all other 7B and 13B models!**
 This release provides a first: a fully open model with class-breaking performance, capable of running fully accelerated on even moderate consumer GPUs.
 Our thanks to the Mistral team for leading the way here.
 ## HuggingFace Leaderboard Performance
 We have evaluated using the methodology and tools for the HuggingFace Leaderboard, and find that we have dramatically improved upon the base model.
+We find **106%** of the base model's performance on HF Leaderboard evals, averaging **65.84**.
+At release time, this beats all 7B and 13B models!
 ![HF Leaderboard](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca/resolve/main/Images/MistralOrca7BHFLeaderboard.png)
 | Metric | Value |
 |-----------------------|-------|
+| MMLU (5-shot)         | 62.24 |
+| ARC (25-shot)         | 64.08 |
+| HellaSwag (10-shot)   | 83.99 |
+| TruthfulQA (0-shot)   | 53.05 |
+| Avg.                  | 65.84 |
 We use [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to run the benchmark tests above, using the same version as the HuggingFace LLM Leaderboard.