README.md

---
license: mit
license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/LICENSE
base_model: deepseek-ai/DeepSeek-V4-Pro
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
tags:
  - nvfp4
  - ocp-mx
  - quantization
  - amd-quark
  - moe
  - deepseek
language:
  - en
  - zh
---

# DeepSeek-V4-Pro-NVFP4

## Model Overview

- **Model Architecture:** DeepseekV4ForCausalLM
  - **Input:** Text
  - **Output:** Text
- **Supported Hardware Microarchitecture:** AMD MI355 / MI350 / MI300 (emulation)
- **ROCm:** 7.2.3
- **PyTorch:** 2.11.0
- **Transformers:** 5.13.1
- **Operating System(s):** Linux
- **Inference Engine:** [vLLM](https://docs.vllm.ai/en/latest/) / [SGLang](https://docs.sglang.ai/)
- **Model Optimizer:** [AMD-Quark](https://quark.docs.amd.com/latest/index.html) (v0.12.0)
  - **Quantized layers:**
    - Router `experts`: NVFP4
    - `shared_experts`, `attn`: FP8-E4M3 per-block


## Model Quantization

The model was quantized from [deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) with
`experts` quantized to MXFP4, and `shared_experts` and `attn` quantized to FP8. Using [AMD Quark](https://quark.docs.amd.com/latest/index.html), 
we re-quantized both `experts` and `shared_experts` to NVFP4 while keeping `attn` in FP8.

### Quantization script


The end-to-end recipe lives in the Quark examples:
`examples/torch/language_modeling/llm_ptq/deepseek_v4/nvfp4`, and is driven by
`run_pipeline.sh`. The quantization scope is controlled by `EXCLUDE_LAYERS`.

```bash
export EXCLUDE_LAYERS="*attn* *ffn.gate *ffn_norm mtp*" \
export SRC=deepseek-ai/DeepSeek-V4-Pro
export OUT=amd/DeepSeek-V4-Pro-NVFP4

bash run_pipeline.sh
```





## Deployment

### Use with vLLM

This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/)
backend. [SGLang](https://docs.sglang.ai/) is also supported.

## Evaluation

The model was evaluated on GSM8K benchmarks. 

### Accuracy

<table>
  <tr>
   <td><strong>Benchmark</strong>
   </td>
   <td><strong>DeepSeek-V4-Pro (BF16) </strong>
   </td>
   <td><strong>DeepSeek-V4-Pro-NVFP4 </strong>
   </td>
   <td><strong>Recovery</strong>
   </td>
  </tr>
  <tr>
   <td>GSM8K (flexible-extract)
   </td>
   <td>95.38
   </td>
   <td>94.77
   </td>
   <td>99.36%
   </td>
  </tr>

  </tr>
</table>




### Reproduction

The GSM8K result was obtained using the `lm-evaluation-harness` framework, based on the Docker image `rocm/vllm-dev:nightly_main_20260714`.

Install the lm-eval `(Version: 0.4.12)` in container first.
```
pip install lm-eval[api]
```

#### Launching server
```
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MOE=1 \
vllm amd/DeepSeek-V4-Pro-NVFP4 \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
```

#### Evaluating model in a new terminal
```
lm_eval \
  --model local-completions \
  --model_args model=amd/DeepSeek-V4-Pro-NVFP4,tokenizer=amd/DeepSeek-V4-Pro-NVFP4,base_url=http://127.0.0.1:8001/v1/completions,num_concurrent=32,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . \
  2>&1 | tee -a eval.log
```
## License

This model is a quantized derivative of
[deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)
and is distributed under the same license as the source model: the
[MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/LICENSE).
A copy of the upstream LICENSE is included in this repository.

Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.
AMD has modified the model weights of the MoE expert layers by quantizing them to
NVFP4 with AMD Quark; the modifications are provided under the same MIT License and
are not subject to any separate or different license.