Instructions to use cloudyu/mistral_28B_instruct_v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cloudyu/mistral_28B_instruct_v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="cloudyu/mistral_28B_instruct_v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("cloudyu/mistral_28B_instruct_v0.1")
model = AutoModelForCausalLM.from_pretrained("cloudyu/mistral_28B_instruct_v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use cloudyu/mistral_28B_instruct_v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cloudyu/mistral_28B_instruct_v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cloudyu/mistral_28B_instruct_v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/cloudyu/mistral_28B_instruct_v0.1

SGLang

How to use cloudyu/mistral_28B_instruct_v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cloudyu/mistral_28B_instruct_v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cloudyu/mistral_28B_instruct_v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cloudyu/mistral_28B_instruct_v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cloudyu/mistral_28B_instruct_v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use cloudyu/mistral_28B_instruct_v0.1 with Docker Model Runner:
```
docker model run hf.co/cloudyu/mistral_28B_instruct_v0.1
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

 this is a 128 layers model based on mistral architecture 
 sft by vicgalle/alpaca-gpt4.
 template is "{instruction} {inputs} \n {output}"

example output

<s> write a story about new yrok city.  
 Once upon a time, in the heart of the United States, there was a city known as New York City. It was a bustling metropolis, teeming with people from all walks of life. The streets were alive with energy and excitement, as people went about their daily routines, working hard to make ends meet.

The city was a melting pot of cultures, where people from all over the world had come to start anew. They brought with them their traditions, their customs, and their dreams, creating a unique tapestry that made up the very essence of New York City.

One day, as the sun set over the skyscrapers, a young woman named Maya stepped off the subway, her eyes wide with wonder. She had just moved to the city from a small town in the midwest, and she couldn't believe her luck at having landed a job as a journalist for one of the biggest news organizations in the city.

As she made her way through the crowded streets, she took in everything around her - the sounds of horns honking, the smells of street food, the sight of people from all races and ethnicities living side by side. It was a world unlike anything she had ever experienced before, and it both excited and scared her.

Maya soon found herself immersed in the fast-paced world of journalism, reporting on everything from local events to national news. She worked long hours, but she loved every moment of it. She had finally found her place in the world, and she knew that this was only the beginning of her journey.

Over the next few years, Maya continued to explore the city, uncovering its secrets and sharing its stories with the world. She met people from all walks of life, each one leaving an indelible mark on her heart.

And even as the city changed and grew, Maya remained forever enchanted by its magic. For she had discovered something truly special about New York City: no matter how big or bright the world became, it would always be her home.</s>

Downloads last month: 67

Safetensors

Model size

28B params

Tensor type

BF16

Model tree for cloudyu/mistral_28B_instruct_v0.1

Quantizations

2 models