Instructions to use NousResearch/Yarn-Mistral-7b-128k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NousResearch/Yarn-Mistral-7b-128k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NousResearch/Yarn-Mistral-7b-128k", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NousResearch/Yarn-Mistral-7b-128k", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("NousResearch/Yarn-Mistral-7b-128k", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NousResearch/Yarn-Mistral-7b-128k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NousResearch/Yarn-Mistral-7b-128k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Yarn-Mistral-7b-128k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/NousResearch/Yarn-Mistral-7b-128k

SGLang

How to use NousResearch/Yarn-Mistral-7b-128k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NousResearch/Yarn-Mistral-7b-128k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Yarn-Mistral-7b-128k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NousResearch/Yarn-Mistral-7b-128k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Yarn-Mistral-7b-128k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use NousResearch/Yarn-Mistral-7b-128k with Docker Model Runner:
```
docker model run hf.co/NousResearch/Yarn-Mistral-7b-128k
```

sliding_window = 131072? Sliding window attention doesn't work for 128?

by keyishen - opened Nov 3, 2023

Discussion

keyishen

Nov 3, 2023

Does it mean yarn-mistral version discards the sliding window attention used in Mistral-7B?

keyishen changed discussion title from sliding_window = 131072? to sliding_window = 131072? Sliding window attention doesn't work for 128? Nov 3, 2023

bloc97

NousResearch org Nov 3, 2023

•

edited Nov 3, 2023

Sliding window does work, but in order to take advantage of the full 128k context, you should set it to 128k. Smaller windows will lower VRAM requirements but will degrade PPL and reduce the context size that the model can retrieve information from.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment