Instructions to use ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Local Apps Settings

How to use ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1

SGLang

How to use ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1 with Docker Model Runner:
```
docker model run hf.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v1
```

Answers without <think> tag

by Eryk-Chmielewski - opened Jan 22, 2025

Discussion

Eryk-Chmielewski

Jan 22, 2025

Hi, I'm previously check 32b than this,
in 32b version there is always tag.

But here is missing:

I don't know if base model has it.

lrl-modelcloud

Jan 22, 2025

•

edited Jan 23, 2025

@Eryk-Chmielewski Yes, we have also noticed this. It seems to be due to our calibration data, which caused the ‘think’ tag to disappear. We are plan update our calibration data to make it compatible with the think process of the deepseek-r1 series model.

Qubitium

ModelCloud.AI org Jan 23, 2025

@Eryk-Chmielewski Are you getting the wrong answers to your logic/math questions without the <think> outputs?

Eryk-Chmielewski

Jan 23, 2025

I would say that it behaves more like in CoT prompting.

It does not have that first it tries to understand the question and plan the actions and after finishing think execute them.

It more immediately starts solving the task but as CoT.

Eryk-Chmielewski

Jan 23, 2025

But I've noticed other problems too, he can stop halfway through an answer:

Don't answer anythging or answer out of the blue:

lrl-modelcloud

Jan 23, 2025

@Eryk-Chmielewski Hi, we have fixed the issue with the think tag and released the new Votex V2 version. https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment