Instructions to use mistralai/Mistral-7B-Instruct-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mistralai/Mistral-7B-Instruct-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mistralai/Mistral-7B-Instruct-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Install mistral-common:
pip install --upgrade mistral-common
# Start the vLLM server:
vllm serve "mistralai/Mistral-7B-Instruct-v0.1" --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.1

SGLang

How to use mistralai/Mistral-7B-Instruct-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mistralai/Mistral-7B-Instruct-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mistralai/Mistral-7B-Instruct-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mistralai/Mistral-7B-Instruct-v0.1 with Docker Model Runner:
```
docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.1
```

Mistral-Instruct can't count characters

#79

by KeithCu - opened Nov 19, 2023

Discussion

KeithCu

Nov 19, 2023

I've been playing around with Mistral, and think it would be an awesome LLM for LibreOffice, for everything from helping with grammar checking, to fixing spreadsheet errors, to helping format a resume to look more professional, and even being a research assistant.

However, it fails at a few easy commands. Here are my prompts and results.

I want to check your ability to count characters. Given the following sentence "She go to the store." Assuming the first character is index 0, what is the character index and length of the word "go" in the sentence?

Mistral-Instruct: The character index of the word "go" in the sentence "She goes to the store" would be from index 12 to index 15 (inclusive). This means that there are a total of 4 characters in the word "go".

ParisNeo created a dataset for this: https://huggingface.co/datasets/ParisNeo/Word_in_Sentence_Database

And for grammar checking:

Please perform a grammar check on the sentence at the end and provide a list of errors.
For each error, describe the problem, suggest a fix, and include the character index of the start and length of the mistaken letters suggested to be replaced with the fix.
Make sure the fix is minimal (the fewest letters that need to be changed) and if fixing one error makes another go away, don’t include the subsequent error.
Put into a JSON format so it can be easily loaded and used by others.
Sentence for Grammar Check: "She go to the store."

Mistral-Instruct:
{
"Errors": [
{
"Problem": "Subject-verb agreement",
"Fix": "goes",
"StartIndex": 0,
"Length": 3
},
{
"Problem": "Missing article before noun",
"Fix": "a",
"StartIndex": 4,
"Length": 1
}
],
"CorrectedSentence": "She goes to the store."
}

andysalerno

Nov 19, 2023

I'm not an expert, nor am I associated with Mistral :D But I don't think Mistral (or GPT-4, or GPT-3.5) could handle this prompt the way you are describing it, since it involves counting characters, and LLMs infamously do not understand characters well at all. Their view of the world is tokens.

I just asked ChatGPT 3.5:

reverse the string "Even gpt 3.5 can't reverse strings very well"

with the response:

Sure, here is the reversed string:

"llew yrev sgnirver t'nac 5.3 tpg nevE"

I asked ChatGPT 4, and amazingly it got it right, and for a second I thought they somehow upgraded ChatGPT to understand on a character-by-character basis. But then I saw the little "analysis" button and clicked it and realized that ChatGPT decided to execute Python to reverse the string :D

KeithCu

Nov 19, 2023

GPT 3.5 and 4, and Claude (usually) handle it without any problems. However, nearly every other one screws it up! It's like I found a bug in the entire LLM world ;-)

They definitely understand characters from any programming experience, but I think this sort of stuff is outside their training. Usually it is to write code to calculate where a word is in a sentence, but in this case, when it's changing part of a sentence, I need it to be able to tell me where.

It's possible to get LLMs to use Python to calculate word indexes, but if there are multiple uses of the same word in a sentence, it could more confusing to figure out what it is talking about.

Otherwise I'm just so impressed at Mistral's ability to be a helper for an app like LibreOffice (which has ~100M users) without any fine-tuning.

I did find a workaround is to have it generate a new complete sentence, and then the app figures out what to change.

Please perform a grammar check on the sentence at the end and return the simplest error you find.
Describe the problem, and create an improved sentence to fix that error.
Put into a JSON format so it can be easily loaded and used by others.
Sentence for Grammar Check: "She go to the store."

If I just have it (hopefully without hallucination!) return the entire improved sentence, the app can figure out what minimal characters to replace. (You would never replace the entire sentence if only some of it changed because it could have different formatting in different spots.)

With that change, it generally seems to do a pretty great job as a grammar checker.

LibreOffice has integrated Python support, here's a simple Python grammar checker that plugs into LibreOffice and could be used as a baseline in case anyone wants to work on it: https://cgit.freedesktop.org/libreoffice/lightproof/

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment