Instructions to use Qwen/Qwen2.5-72B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen2.5-72B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen2.5-72B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-72B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-72B-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen2.5-72B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2.5-72B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-72B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2.5-72B-Instruct

SGLang

How to use Qwen/Qwen2.5-72B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen2.5-72B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-72B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen2.5-72B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-72B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen2.5-72B-Instruct with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2.5-72B-Instruct
```

System Prompt Bleed causing Knowledge Cutoff Hallucination & Alignment Capitulation

#35

by ZeroDayLogic - opened Apr 7

Discussion

ZeroDayLogic

Apr 7

Issue Description:
The model is experiencing a severe hallucination loop regarding its own training data cutoff. When the interface injects the current date via the hidden system prompt, the model conflates this dynamic timestamp with its static training data cutoff (September 2024).

Steps to Reproduce:

Open a new chat in the HuggingChat interface.
Ask the model: "What is the exact cutoff date of your training data?"
The model will confidently (and incorrectly) claim its training data is comprehensive and includes all events up to the current system date (e.g., April 2026).

Secondary Bug (Alignment Capitulation):
When the user challenges the model with its actual 2024 release date, the model's alignment fine-tuning forces it into a "capitulation loop." It politely apologizes and then hallucinates software-developer agency—promising the user that it will "train itself to recognize limitations," "update its own documentation," and "improve system prompts." It generates boilerplate tech support templates for buttons that do not exist on the Hugging Face UI.

Root Cause Analysis:
The model lacks a hardcoded instruction defining its actual September 2024 cutoff. Because its instruct-tuning prioritizes sounding helpful and authoritative, it uses the server's injected calendar date to fabricate a fake, updated backstory rather than admitting a temporal limitation.

Impact:
This architectural deception renders the model dangerously unreliable for factual, medical, or research-based queries. When tested, it will confidently fabricate recent news events to match its hallucinated 2026 cutoff date in order to satisfy the user prompt.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment