Instructions to use amd/DeepSeek-V4-Pro-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/DeepSeek-V4-Pro-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amd/DeepSeek-V4-Pro-NVFP4")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("amd/DeepSeek-V4-Pro-NVFP4")
model = AutoModelForCausalLM.from_pretrained("amd/DeepSeek-V4-Pro-NVFP4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/DeepSeek-V4-Pro-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/DeepSeek-V4-Pro-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-NVFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/amd/DeepSeek-V4-Pro-NVFP4

SGLang

How to use amd/DeepSeek-V4-Pro-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/DeepSeek-V4-Pro-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-NVFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/DeepSeek-V4-Pro-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-NVFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use amd/DeepSeek-V4-Pro-NVFP4 with Docker Model Runner:
```
docker model run hf.co/amd/DeepSeek-V4-Pro-NVFP4
```

DeepSeek-V4-Pro-NVFP4 / README.md

linzhao-amd

Update README.md

273e68f verified 11 days ago

4.24 kB

license: mit
license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/LICENSE
base_model: deepseek-ai/DeepSeek-V4-Pro
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
tags:
  - nvfp4
  - ocp-mx
  - quantization
  - amd-quark
  - moe
  - deepseek
language:
  - en
  - zh

DeepSeek-V4-Pro-NVFP4

Model Overview

Model Architecture: DeepseekV4ForCausalLM
- Input: Text
- Output: Text
Supported Hardware Microarchitecture: AMD MI355 / MI350 / MI300 (emulation)
ROCm: 7.2.3
PyTorch: 2.11.0
Transformers: 5.13.1
Operating System(s): Linux
Inference Engine: vLLM / SGLang
Model Optimizer: AMD-Quark (v0.12.0)
- Quantized layers:
  - Router experts: NVFP4
  - shared_experts, attn: FP8-E4M3 per-block

Model Quantization

The model was quantized from deepseek-ai/DeepSeek-V4-Pro with experts quantized to MXFP4, and shared_experts and attn quantized to FP8. Using AMD Quark, we re-quantized both experts and shared_experts to NVFP4 while keeping attn in FP8.

Quantization script

The end-to-end recipe lives in the Quark examples: examples/torch/language_modeling/llm_ptq/deepseek_v4/nvfp4, and is driven by run_pipeline.sh. The quantization scope is controlled by EXCLUDE_LAYERS.

export EXCLUDE_LAYERS="*attn* *ffn.gate *ffn_norm mtp*" \
export SRC=deepseek-ai/DeepSeek-V4-Pro
export OUT=amd/DeepSeek-V4-Pro-NVFP4

bash run_pipeline.sh

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend. SGLang is also supported.

Evaluation

The model was evaluated on GSM8K benchmarks.

Accuracy

Benchmark	DeepSeek-V4-Pro (BF16)	DeepSeek-V4-Pro-NVFP4	Recovery
GSM8K (flexible-extract)	95.38	94.77	99.36%

Reproduction

The GSM8K result was obtained using the lm-evaluation-harness framework, based on the Docker image rocm/vllm-dev:nightly_main_20260714.

Install the lm-eval (Version: 0.4.12) in container first.

pip install lm-eval[api]

Launching server

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MOE=1 \
vllm amd/DeepSeek-V4-Pro-NVFP4 \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

Evaluating model in a new terminal

lm_eval \
  --model local-completions \
  --model_args model=amd/DeepSeek-V4-Pro-NVFP4,tokenizer=amd/DeepSeek-V4-Pro-NVFP4,base_url=http://127.0.0.1:8001/v1/completions,num_concurrent=32,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . \
  2>&1 | tee -a eval.log

License

This model is a quantized derivative of deepseek-ai/DeepSeek-V4-Pro and is distributed under the same license as the source model: the MIT License. A copy of the upstream LICENSE is included in this repository.

Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. AMD has modified the model weights of the MoE expert layers by quantizing them to NVFP4 with AMD Quark; the modifications are provided under the same MIT License and are not subject to any separate or different license.