Instructions to use mistralai/Mistral-7B-Instruct-v0.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mistralai/Mistral-7B-Instruct-v0.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mistralai/Mistral-7B-Instruct-v0.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Install mistral-common:
pip install --upgrade mistral-common
# Start the vLLM server:
vllm serve "mistralai/Mistral-7B-Instruct-v0.2" --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.2

SGLang

How to use mistralai/Mistral-7B-Instruct-v0.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mistralai/Mistral-7B-Instruct-v0.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mistralai/Mistral-7B-Instruct-v0.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mistralai/Mistral-7B-Instruct-v0.2 with Docker Model Runner:
```
docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.2
```

Was this model based of Mistral-7B-v0.2 from the start?

#72

by stduhpf - opened Mar 24, 2024

Discussion

stduhpf

Mar 24, 2024

•

edited Mar 24, 2024

The recent changes to the README are a bit confusing. It used to say it was an imporved version of Mistral-7B-Instruct-v0.1, and based on Mistral-7B-v0.1. The weights haven't changed, but the base model is no longer the same as before?

FusionCow

Mar 25, 2024

Ai flux made a video about it

xzuyn

Mar 25, 2024

They probably just copied the README from the previous version and didn't properly update it until now.

MaziyarPanahi

Mar 25, 2024

They probably just copied the README from the previous version and didn't properly update it until now.

They only have like 5 models, I mean it’s not that hard to double check after months after the release.
I believe they didn’t want people to know there is a base v0.2 either for strategic reasons or just avoid people keep bugging them about using an instruct without having the pretrained model. (I am just happy we have a new open-source model, so thank you 🙏)

xzuyn

Mar 25, 2024

•

edited Mar 25, 2024

They probably just copied the README from the previous version and didn't properly update it until now.

They only have like 5 models, I mean it’s not that hard to double check after months after the release.
I believe they didn’t want people to know there is a base v0.2 either for strategic reasons or just avoid people keep bugging them about using an instruct without having the pretrained model. (I am just happy we have a new open-source model, so thank you 🙏)

Maybe, but since it came out all I've seen is people assuming it's a different base (and being sad that Mistral didn't release it's base alongside the Instruct version as they did with the v0.1 version) since it's config isn't using the same SWA window as mistralai/Mistral-7B-v0.1 or mistralai/Mistral-7B-Instruct-v0.1. Only recently with people posting about this README have I seen anything thinking otherwise.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment