README.md

---
license: mit
license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/LICENSE
base_model: deepseek-ai/DeepSeek-V4-Flash
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
tags:
  - ocp-mx
  - quantization
  - amd-quark
  - moe
  - deepseek
---

# DeepSeek-V4-Flash-MXFP4

## Model Overview

- **Model Architecture:** DeepseekV4ForCausalLM
  - **Input:** Text
  - **Output:** Text
- **Supported Hardware Microarchitecture:** AMD MI355 / MI350 (gfx950)
- **ROCm:** 7.2.0
- **PyTorch:** 2.9.1
- **Transformers:** 5.13.1
- **Operating System(s):** Linux
- **Inference Engine:** [vLLM](https://docs.vllm.ai/en/latest/)
- **Model Optimizer:** [AMD-Quark](https://quark.docs.amd.com/latest/index.html) (v0.12.0)
  - **Quantized layers:** All routed + shared MoE expert projections. All other modules (attention, the MoE router gate, norms, embeddings, the output head, and the MTP block) are excluded and kept in original precision.
  - **Weight quantization:** OCP MXFP4, Static
  - **Activation quantization:** OCP MXFP4, Dynamic

## Model Quantization

Quantized from [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)
with [AMD Quark](https://quark.docs.amd.com/latest/index.html). The pipeline
re-quantizes only the MoE expert weights and activations to MXFP4. All non-expert
modules are kept as-is via the exclude list.

### Quantization script

```python
from quark.torch import ModelQuantizer
from quark.torch.quantization.config.template import LLMTemplate

template = LLMTemplate.get('deepseek_v4')
qconfig = template.get_config(scheme='mxfp4')
ModelQuantizer(qconfig).direct_quantize_checkpoint(
    pretrained_model_path='<DSV4_Flash_src_path>',
    save_path='<output_dir>',
    keep_excluded_layers_as_original_model_state=True,
)
```

## Deployment

### Use with vLLM

This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend based on the Docker image `rocm/vllm-dev:nightly_main_20260714`. vLLM and lm_eval are both installed from source. 

## Evaluation

The model was evaluated on gsm8k (8-shot) benchmark using the vLLM framework.

### Accuracy

| Benchmark | deepseek-ai/DeepSeek-V4-Flash | amd/DeepSeek-V4-Flash-MXFP4(this model) | Recovery |
|---|---:|---:|---:|
| GSM8K (flexible-extract) | 95.00 | 94.92 | 99.9% |

### Reproduction

The GSM8K results were obtained using the lm-eval framework, based on the Docker image `rocm/vllm-dev:nightly_main_20260714`.

#### Launching server
```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
vllm serve amd/DeepSeek-V4-Flash-MXFP4 --tensor-parallel-size 4 --kv-cache-dtype fp8 \
  --trust-remote-code --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
```
#### Evaluating model in a new terminal
```bash
lm_eval --model local-completions \
    --model_args model=amd/DeepSeek-V4-Flash-MXFP4,base_url=http://localhost:30000/v1/completions,tokenized_requests=False,num_concurrent=32 \
    --tasks gsm8k --batch_size auto --num_fewshot 8
```

## License

This model is a quantized derivative of
[deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)
and is distributed under the same license as the source model: the
[MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/LICENSE).
A copy of the upstream LICENSE is included in this repository.

Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.
AMD has modified the model weights of the MoE expert layers by quantizing them to
MXFP4 with AMD Quark; the modifications are provided under the same MIT License and
are not subject to any separate or different license.