Instructions to use amd/DeepSeek-V4-Pro-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/DeepSeek-V4-Pro-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amd/DeepSeek-V4-Pro-NVFP4")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("amd/DeepSeek-V4-Pro-NVFP4")
model = AutoModelForCausalLM.from_pretrained("amd/DeepSeek-V4-Pro-NVFP4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/DeepSeek-V4-Pro-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/DeepSeek-V4-Pro-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-NVFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/amd/DeepSeek-V4-Pro-NVFP4

SGLang

How to use amd/DeepSeek-V4-Pro-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/DeepSeek-V4-Pro-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-NVFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/DeepSeek-V4-Pro-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-NVFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use amd/DeepSeek-V4-Pro-NVFP4 with Docker Model Runner:
```
docker model run hf.co/amd/DeepSeek-V4-Pro-NVFP4
```

updated quantized model

by huidongji - opened 6 days ago

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

+21432

-24885

Files changed (8) hide show

assets/dsv4_performance.png +3 -0
config.json +0 -0
inference/__pycache__/model.cpython-312.pyc +0 -0
inference/kernel.py +1 -1
inference/model.py +1 -2
input_scale.safetensors +2 -2
model-00064-of-00064.safetensors +2 -2
model.safetensors.index.json +2 -2

assets/dsv4_performance.png ADDED Viewed

Git LFS Details

SHA256: 8fd472981a4c8d40c1845c51c5e8961fc4ef3ac22e7ec77801f534c239c1b30f
Pointer size: 132 Bytes
Size of remote file: 1 MB

config.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

inference/__pycache__/model.cpython-312.pyc CHANGED Viewed

Binary files a/inference/__pycache__/model.cpython-312.pyc and b/inference/__pycache__/model.cpython-312.pyc differ

inference/kernel.py CHANGED Viewed

@@ -85,7 +85,7 @@ def act_quant_kernel(
                     for i, j in T.Parallel(blk_m, group_size):
                         y_local[i, j] = T.Cast(
                             out_dtype,
-                            T.Cast(compute_dtype, T.Cast(out_dtype, T.clamp(
                                 x_local[i, j] / s_local[i], fp8_min, fp8_max
                             ))) * s_local[i],
                         )

                     for i, j in T.Parallel(blk_m, group_size):
                         y_local[i, j] = T.Cast(
                             out_dtype,
+                            T.Cast(compute_dtype, T.Cast(FP8, T.clamp(
                                 x_local[i, j] / s_local[i], fp8_min, fp8_max
                             ))) * s_local[i],
                         )

inference/model.py CHANGED Viewed

@@ -624,8 +624,7 @@ class MoE(nn.Module):
         self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim, dtype=expert_dtype, swiglu_limit=args.swiglu_limit) if self.experts_start_idx <= i < self.experts_end_idx else None
                                        for i in range(self.n_routed_experts)])
         assert args.n_shared_experts == 1
-        # no swiglu_limit
-        self.shared_experts = Expert(args.dim, args.moe_inter_dim)
     def forward(self, x: torch.Tensor, input_ids: torch.Tensor) -> torch.Tensor:
         shape = x.size()

         self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim, dtype=expert_dtype, swiglu_limit=args.swiglu_limit) if self.experts_start_idx <= i < self.experts_end_idx else None
                                        for i in range(self.n_routed_experts)])
         assert args.n_shared_experts == 1
+        self.shared_experts = Expert(args.dim, args.moe_inter_dim, swiglu_limit=args.swiglu_limit)
     def forward(self, x: torch.Tensor, input_ids: torch.Tensor) -> torch.Tensor:
         shape = x.size()

input_scale.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4883a14b83c509fe21729f104d640b70027e6ea78e3cc1ce6edf36ce3cc7d2cd
-size 7291368

 version https://git-lfs.github.com/spec/v1
+oid sha256:1df615aed5669b3bde5a8766298f12df4669a3a3baa883dda05fb0fc25a0eca9
+size 7311108

model-00064-of-00064.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a144e2c5c67ea788d78e7e4df84f82740dbd859711cdde5bed084048f3bbbd33
-size 13956848916

 version https://git-lfs.github.com/spec/v1
+oid sha256:46fa7b6b16fc1f56b28be0424a34cc6f06ca2c4a550d2741c5efdd28e623bbb1
+size 13964140292

model.safetensors.index.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aff908e93ceb3a22e4229cef498716ecaeb698e2d6defd1bc4d83f835c479e58
-size 23130112

 version https://git-lfs.github.com/spec/v1
+oid sha256:6c3658a0ac088136056edb8652e9c53aa4e76d0e0b808e10806a72c2cf4dfad2
+size 23762560