Instructions to use mistralai/Mistral-7B-Instruct-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mistralai/Mistral-7B-Instruct-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mistralai/Mistral-7B-Instruct-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Install mistral-common:
pip install --upgrade mistral-common
# Start the vLLM server:
vllm serve "mistralai/Mistral-7B-Instruct-v0.1" --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.1

SGLang

How to use mistralai/Mistral-7B-Instruct-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mistralai/Mistral-7B-Instruct-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mistralai/Mistral-7B-Instruct-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mistralai/Mistral-7B-Instruct-v0.1 with Docker Model Runner:
```
docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.1
```

How to fine tune ?

#12

by NickyNicky - opened Sep 28, 2023

Discussion

NickyNicky

Sep 28, 2023

same title ^|

IshaanGupta2311

Sep 28, 2023

same ques, i want to explore an application of these 7B models that i have been thinking about for quite some time

NajiAboo

Sep 28, 2023

looking for same

Lazycuber

Sep 28, 2023

still experimenting with colab finetuning...

silvacarl

Sep 28, 2023

looking for same here as well. so far its amazing...

nadahlberg

Sep 28, 2023

Seems to be mostly working with the default qlora.py script with one small change.

This line was causing trouble because mistral's model.config.pad_token_id returns None:

"unk_token": tokenizer.convert_ids_to_tokens(
     model.config.pad_token_id if model.config.pad_token_id != -1 else tokenizer.pad_token_id
),

Adding a None check seems to fix:

"unk_token": tokenizer.convert_ids_to_tokens(
    model.config.pad_token_id if 
    model.config.pad_token_id is not None and model.config.pad_token_id != -1 
    else tokenizer.pad_token_id
),

silvacarl

Sep 28, 2023

•

edited Sep 28, 2023

thx. i am not even sure yet if it needs fine tuning, we are running a bunch of tests on it.

it seems to be the best 7B model we have ever seen. it may outperform 13B and possibly 70B models for certain use cases.

NickyNicky

Sep 29, 2023

•

edited Sep 29, 2023

it's very fast.

use:
-When flash attention 2 is used and with an inference of approximately 600 tokens <20 seconds if I remember correctly.
-Without flash attention 2, inferences from 100 tokens take +440 seconds.

fine-tune (SFTTrainer):
-gpu a100 colab 40GB
-for a 15k dataset it takes approximately 1H
-14 credits 1H
-per_device_train_batch_size= 6 ---->>> use gpu 36GB

Lazycuber

Sep 29, 2023

Anyone made a colab notebook for finetuning?

sscpr

Oct 1, 2023

Seems to be mostly working with the default qlora.py script with one small change.

using this script and your change seems to work but I had to repull in the latest transformers version 4.34 to get it to work

NickM2002

Oct 2, 2023

Seems to be mostly working with the default qlora.py script with one small change.

using this script and your change seems to work but I had to repull in the latest transformers version 4.34 to get it to work

I also had to repull transformers with
“
pip install git+https://github.com/huggingface/transformers
“