Instructions to use Qwen/Qwen2.5-Coder-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen2.5-Coder-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen2.5-Coder-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Qwen/Qwen2.5-Coder-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2.5-Coder-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2.5-Coder-7B

SGLang

How to use Qwen/Qwen2.5-Coder-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen2.5-Coder-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen2.5-Coder-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen2.5-Coder-7B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2.5-Coder-7B
```

Qwen2.5-Coder-7B

Commit History

update tokenizer_config.json,config.json,generation_config.json

0396a76

JustinLin610 commited on Nov 18, 2024

update README.md

cee317b

feihu.hf commited on Nov 12, 2024

Update README.md

89109db
verified

cyente commited on Nov 11, 2024

Update README.md

388104a
verified

cyente commited on Nov 9, 2024

Update README.md

3040e12
verified

cyente commited on Nov 8, 2024

Update README.md

ee4e81d
verified

cyente commited on Nov 8, 2024

Update README.md

8e82b34
verified

cyente commited on Sep 25, 2024

update README.md

097b213

feihu.hf commited on Sep 20, 2024

Upload ./tokenizer_config.json with huggingface_hub

9ec1a91
verified

clonefy commited on Sep 20, 2024

update tokenizer_config.json

30b6a7e

feihu.hf commited on Sep 20, 2024

Update README.md (#3)

78bf335
verified

hzhwcmhf

cyente commited on Sep 20, 2024

Update README.md (#2)

fc160ff
verified

huybery

cyente commited on Sep 19, 2024

Upload ./tokenizer_config.json with huggingface_hub

4c1c161
verified

clonefy commited on Sep 19, 2024

update README & config.json

e8d5fbb

feihu.hf commited on Sep 18, 2024

update README & config.json

f07775d

feihu.hf commited on Sep 18, 2024

update README & LICENSE

c635280

feihu.hf commited on Sep 18, 2024

Upload LICENSE

4e56b5c
verified

hzhwcmhf commited on Sep 18, 2024

Create README.md (#1)

81c2a16
verified

hzhwcmhf

cyente commited on Sep 18, 2024

Update config.json

f52e93b
verified

clonefy commited on Sep 17, 2024

Update config.json

e1a210c
verified

clonefy commited on Sep 17, 2024

Upload folder using huggingface_hub

921a10c
verified

clonefy commited on Sep 16, 2024

initial commit

4914cf9
verified

clonefy commited on Sep 16, 2024

Commit History

update tokenizer_config.json,config.json,generation_config.json 0396a76

update README.md cee317b

Update README.md 89109db verified

Update README.md 388104a verified

Update README.md 3040e12 verified

Update README.md ee4e81d verified

Update README.md 8e82b34 verified

update README.md 097b213

Upload ./tokenizer_config.json with huggingface_hub 9ec1a91 verified

update tokenizer_config.json 30b6a7e

Update README.md (#3) 78bf335 verified

Update README.md (#2) fc160ff verified

Upload ./tokenizer_config.json with huggingface_hub 4c1c161 verified

update README & config.json e8d5fbb

update README & config.json f07775d

update README & LICENSE c635280

Upload LICENSE 4e56b5c verified

Create README.md (#1) 81c2a16 verified

Update config.json f52e93b verified

Update config.json e1a210c verified

Upload folder using huggingface_hub 921a10c verified

initial commit 4914cf9 verified

update tokenizer_config.json,config.json,generation_config.json

0396a76

update README.md

cee317b

Update README.md

89109db
verified

Update README.md

388104a
verified

Update README.md

3040e12
verified

Update README.md

ee4e81d
verified

Update README.md

8e82b34
verified

update README.md

097b213

Upload ./tokenizer_config.json with huggingface_hub

9ec1a91
verified

update tokenizer_config.json

30b6a7e

Update README.md (#3)

78bf335
verified

Update README.md (#2)

fc160ff
verified

Upload ./tokenizer_config.json with huggingface_hub

4c1c161
verified

update README & config.json

e8d5fbb

update README & config.json

f07775d

update README & LICENSE

c635280

Upload LICENSE

4e56b5c
verified

Create README.md (#1)

81c2a16
verified

Update config.json

f52e93b
verified

Update config.json

e1a210c
verified

Upload folder using huggingface_hub

921a10c
verified

initial commit

4914cf9
verified