Instructions to use NousResearch/Yarn-Mistral-7b-64k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NousResearch/Yarn-Mistral-7b-64k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NousResearch/Yarn-Mistral-7b-64k", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NousResearch/Yarn-Mistral-7b-64k", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("NousResearch/Yarn-Mistral-7b-64k", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NousResearch/Yarn-Mistral-7b-64k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NousResearch/Yarn-Mistral-7b-64k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Yarn-Mistral-7b-64k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/NousResearch/Yarn-Mistral-7b-64k

SGLang

How to use NousResearch/Yarn-Mistral-7b-64k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NousResearch/Yarn-Mistral-7b-64k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Yarn-Mistral-7b-64k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NousResearch/Yarn-Mistral-7b-64k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Yarn-Mistral-7b-64k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use NousResearch/Yarn-Mistral-7b-64k with Docker Model Runner:
```
docker model run hf.co/NousResearch/Yarn-Mistral-7b-64k
```

Request for Model Sharding to Support Lower-End PCs

by Firejowl - opened Nov 8, 2023

Discussion

Firejowl

Nov 8, 2023

Hello NousResearch team,

I hope this message finds you well. I am reaching out to inquire about the possibility of sharding the Yarn-Mistral-7b-64k model. The aim of this request is to facilitate users with lower-end PCs to run the model more effectively.

As you're aware, the computational demands of large models can be a barrier to entry for individuals without access to high-end hardware. By providing a sharded version of the Yarn-Mistral-7b-64k model, we can democratize access, enabling a broader range of users to experiment with and benefit from this impressive model.

Moreover, sharding could also benefit users who prefer to run models on cloud-based platforms like Google Colab or Kaggle, where there may be limitations on resources or users seek to optimize their allocated compute time.

I understand that model sharding can come with its own set of challenges, but I believe the benefits to the community could be significant. If there's any possibility to consider this, or if there are alternative solutions that could accommodate the needs mentioned above, I'd love to hear your thoughts.

Thank you for your time and for your contributions to the AI community.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment