Instructions to use Qwen/Qwen2.5-7B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen2.5-7B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen2.5-7B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen2.5-7B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2.5-7B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2.5-7B-Instruct

SGLang

How to use Qwen/Qwen2.5-7B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen2.5-7B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen2.5-7B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen2.5-7B-Instruct with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2.5-7B-Instruct
```

How do I make the model output JSON?

#14

by vmajor - opened Nov 14, 2024

Discussion

vmajor

Nov 14, 2024

Is it just an instruction or is there a particular prompt syntax that I need to follow? I am using a GGUF version if that matters.

Currently I am using it with pydantic but sometimes the output cannot be parsed correctly.

jklj077

Qwen org Nov 14, 2024

I believe most of the 100% JSON output is achieved by guided decoding (in addition to telling the model to generate JSON).

If you are using Ollama, there is a json mode: https://github.com/ollama/ollama/blob/main/docs/api.md#request-json-mode and the example scripts are at https://github.com/ollama/ollama/tree/main/examples/python-json-datagenerator

vtriple

Nov 14, 2024

Just make it output yaml instead! Save money on tokens and a headache of parsing broken JSON from any LLM model.

vmajor

Nov 29, 2024

yaml? I never even saw yaml mentioned as a possibility. Is this a serious suggestion or just something to consume more of my time? How do I get it to output yaml? Just asking it nicely, or is there a "proper" way?

ktobah

Dec 1, 2024

Could you please share sample code how you make it work with Pydantic? And which framework are you using the serve the model?

I cannot get it to work with TGI (model loads fine).

vmajor

Dec 1, 2024

Actually, I no longer use pydantic. I instead consulted the documentation (lol) and now have this:

response = self.client.chat.completions.create(
model="gpt-3.5-turbo-1106",
messages=[
{"role": "system", "content": "You are a professional business researcher analyzing manufacturer websites."},
{"role": "user", "content": prompt}
],
response_format={"type": "json_object"},
temperature=0.1
)

I use llama.cpp to run the model using this command: "./llama-server -m ~models/qwen2.5-7b-instruct-q6_k-00001-of-00002.gguf -c 0 --mirostat 2 -fa -j {}

you can add your own flags for GPU offloading and other performance stuff.

ktobah

Dec 1, 2024

Thank you very much. this solution also worked for me before, it will output Json. But if I want to use Pydantic, I cannot get it to work with TGI.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment