intel-ai/DeepSeek-V3.2-NVFP4-AutoRound

DeepSeek-V3.2 MXFP8 W8G32 with NVFP4 MoE Experts

Model Details

This checkpoint is a mixed-precision quantized version of DeepSeek-V3.2, produced with Intel AutoRound 0.17.0 using a model-free workflow.

The general quantization configuration is MXFP8 W8A8 with group size 32. The MoE expert layers are configured to use NVFP4. The eh_proj and indexer modules are excluded from quantization. Please follow the original model's license and usage terms.

Property	Value
Base model	deepseek-ai/DeepSeek-V3.2
Quantization	Mixed MXFP8 and NVFP4
General weight / activation format	MXFP8 W8A8
General group size	32
MoE expert quantization	NVFP4
Activation scaling	Dynamic, per configuration
Excluded modules	eh_proj, indexer
Quantization workflow	Model-free
AutoRound version	0.17.0
Packing format	auto_round:llm_compressor

How to Use

Install the vLLM plugin from the main branch of Intel Neural Compressor into the same Python environment as vLLM. The plugin must be installed even when activation QDQ is disabled: its inc_nvfp4 entry point registers the AutoRound NVFP4 format with vLLM.

git clone https://github.com/intel/neural-compressor.git
cd neural-compressor
python -m pip install --no-build-isolation -e benchmark/vllm-qdq-plugin/

On hardware that supports NVFP4, set VLLM_QDQ=0 to use vLLM's native NVFP4 kernels. The plugin remains necessary to register the checkpoint's AutoRound NVFP4 format. Adjust the checkpoint path and parallelism options for your environment.

VLLM_QDQ=1 vllm serve /home/xinhe/models/deepseek-ai/DeepSeek-V3.2-NVFP4/DeepSeek-V3.2-mxfp-w8g32 \
  --served-model-name dsv32 \
  --port 8001 \
  --tensor-parallel-size 4 \
  --max-model-len 102400 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 20000 \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --tokenizer-mode deepseek_v32 \
  --tool-call-parser deepseek_v32 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v3 \
  --kernel-config.enable_flashinfer_autotune=False

This setup uses DeepSeek-V3.2's tokenizer and tool-call format. The requested context length, concurrency, and GPU memory utilization require sufficient GPU memory; reduce them as needed for your hardware.

Evaluation

Each benchmark was evaluated in five runs. The quantized scores report the arithmetic mean across the five runs; scores are percentages and sample counts are per run. The Raw FP8 baseline is from the original model evaluated with an FP8 KV cache. Recovery is the quantized score divided by the Raw FP8 baseline.

Benchmark	Raw FP8 baseline	MXFP8 + NVFP4 (5-run mean)	Recovery	Samples per run
GPQA Diamond	84.17	81.92	97.33%	198
AIME 2025	92.22	92.67	100.49%	30
GSM8K	97.45	97.41	99.96%	1319
PIQA	94.29	94.39	100.11%	1838
LiveCodeBench v6	80.39	79.89	99.38%	175

EvalScope

With the vLLM server running, use one EvalScope command for GPQA Diamond, AIME 2025, GSM8K, PIQA, and LiveCodeBench v6. The generation settings enable DeepSeek reasoning:

evalscope eval \
  --model dsv32 \
  --eval-type openai_api \
  --api-key EMPTY \
  --timeout 36000 \
  --datasets gpqa_diamond aime25 gsm8k piqa live_code_bench \
  --dataset-args '{"live_code_bench":{"subset_list":["v6"]}}' \
  --generation-config '{"temperature":1.0,"top_p":0.95,"n":1,"extra_body":{"chat_template_kwargs":{"enable_thinking":true,"reasoning_effort":"max"}},"max_tokens":64000}' \
  --eval-batch-size 256 \
  --api-url http://127.0.0.1:8001/v1

For comparable results, keep the benchmark version, dataset subset, generation configuration, and evaluation harness consistent across runs.

Generate the Model

The following is the model-free AutoRound command used to generate the quantized checkpoint:

AR_DISABLE_META_LOAD=1 \
CUDA_VISIBLE_DEVICES=1,2,3 \
auto-round models/deepseek-ai/DeepSeek-V3.2 \
  --scheme MXFP8 \
  --layer_config '{mlp.experts:{scheme:NVFP4}}' \
  --ignore_layers eh_proj,indexer \
  --output_dir models/deepseek-ai/DeepSeek-V3.2-NVFP4-MOE \
  --iters 0 \
  --dataset ultrachat_200k \
  --device_map auto

Update --output_dir to the desired destination. The directory name in this example reflects the original generation command; this README describes the checkpoint at DeepSeek-V3.2-mxfp-w8g32.

Ethical Considerations and Limitations

The model may produce factually incorrect, biased, or otherwise harmful output. It should not be relied on as the sole source of factual information. Perform appropriate safety, quality, and performance testing before deployment, and follow the original model's terms and applicable laws.

Disclaimer

This README is for informational purposes and does not constitute legal advice. Consult qualified counsel regarding the model license and any intended commercial use.

Citation

Please cite the original DeepSeek-V3.2 work and the AutoRound paper when using this checkpoint:

@misc{deepseekai2025deepseekv32,
  title={DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models},
  author={DeepSeek-AI},
  year={2025}
}

@article{cheng2023optimize,
  title={Optimize weight rounding via signed gradient descent for the quantization of llms},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}