| --- |
| base_model: |
| - unsloth/DeepSeek-R1-0528-BF16 |
| language: |
| - en |
| library_name: transformers |
| license: mit |
| --- |
| |
| # Model Overview |
|
|
| - **Model Architecture:** DeepSeek-R1-0528 |
| - **Input:** Text |
| - **Output:** Text |
| - **Supported Hardware Microarchitecture:** AMD MI350/MI355 |
| - **ROCm**: 7.0 |
| - **PyTorch**: 2.8.0 |
| - **Transformers**: 4.56.1 |
| - **Operating System(s):** Linux |
| - **Inference Engine:** [SGLang](https://docs.sglang.ai/)/[vLLM](https://docs.vllm.ai/en/latest/) |
|
|
| # Model Details |
|
|
| This model is adapted from `unsloth/DeepSeek-R1-0528-BF16` by adding an MTP layer (specifically, the 61st layer) in the `modeling_deepseek.py` file. With this modification, it is possible to use AMD-Quark to quantize the DeepSeek-R1-0528 model with the MTP layer included. |
|
|
| **Important Notes:** |
| - When loading this model, you must set `trust_remote_code=True` to ensure that changes related to the MTP layer in `modeling_deepseek.py` take effect. |
| - After loading this model with `transformers`, **evaluation should NOT be performed directly**. The reason is that the forward function for the added MTP layer in `modeling_deepseek.py` is implemented only for calibration during the quantization process, so computation is not guaranteed to be the same as the original DeepSeek-R1-0528. |
| - Therefore, when quantizing with AMD-Quark, you **must add the `--skip_evaluation` option** to skip the evaluation step and only perform quantization. |
| |
| For further details or issues, please refer to the [AMD-Quark](https://quark.docs.amd.com/latest/index.html) documentation or contact the respective developers. |
| |
| Below is an example of how to quantize this model: |
| |
| ```bash |
| cd Quark/examples/torch/language_modeling/llm_ptq/ |
| exclude_layers="lm_head *self_attn* *mlp.gate *eh_proj" |
| python3 quantize_quark.py --model_dir $MODEL_DIR \ |
| --quant_scheme w_mxfp4_a_mxfp4 \ |
| --num_calib_data 32 \ |
| --output_dir $output_dir \ |
| --exclude_layers $exclude_layers \ |
| --dataset pileval \ |
| --multi_gpu \ |
| --model_export hf_format \ |
| --trust_remote_code \ |
| --skip_evaluation \ |
| --seq_len 512 |
| ``` |
| For further details or issues, please refer to the [AMD-Quark](https://quark.docs.amd.com/latest/index.html) documentation or contact the respective developers. |
| |
| # License |
| Modifications Copyright(c) 2025 Advanced Micro Devices, Inc. All rights reserved. |
