| | --- |
| | base_model: |
| | - unsloth/DeepSeek-R1-0528-BF16 |
| | language: |
| | - en |
| | library_name: transformers |
| | license: mit |
| | --- |
| | |
| | # Model Overview |
| |
|
| | - **Model Architecture:** DeepSeek-R1-0528 |
| | - **Input:** Text |
| | - **Output:** Text |
| | - **Supported Hardware Microarchitecture:** AMD MI350/MI355 |
| | - **ROCm**: 7.0 |
| | - **PyTorch**: 2.8.0 |
| | - **Transformers**: 4.56.1 |
| | - **Operating System(s):** Linux |
| | - **Inference Engine:** [SGLang](https://docs.sglang.ai/)/[vLLM](https://docs.vllm.ai/en/latest/) |
| |
|
| | # Model Details |
| |
|
| | In the original `modeling_deepseek.py` file from the `unsloth/DeepSeek-R1-0528-BF16` model, there is no definition or implementation of the MTP (Multi-Token-Predictor) layer. As a result, when you load the original model, there is no MTP layer included, and MTP-specific quantization cannot be performed. |
| |
|
| | To enable MTP layer loading and quantization, this model is adapted from `unsloth/DeepSeek-R1-0528-BF16` by adding an MTP layer in the `modeling_deepseek.py` file. With this modification, it is possible to use [AMD-Quark](https://quark.docs.amd.com/latest/index.html) to quantize the DeepSeek-R1-0528 model with the MTP layer included. |
| |
|
| | **Important Notes:** |
| | - When loading this model, you must set `trust_remote_code=True` to ensure that changes related to the MTP layer in `modeling_deepseek.py` take effect. |
| | - After loading this model with `transformers`, **evaluation should NOT be performed directly**. The reason is that the forward function for the added MTP layer in `modeling_deepseek.py` is implemented only for calibration during the quantization process, so computation is not guaranteed to be the same as the original DeepSeek-R1-0528. |
| | - Therefore, when quantizing with AMD-Quark, you **must add the `--skip_evaluation` option** to skip the evaluation step and only perform quantization. |
| | - To skip quantization for the MTP layers, set `exclude_layers="lm_head *self_attn* *mlp.gate *eh_proj *shared_head.head model.layers.61.*"`. |
| | |
| | Below is an example of how to quantize this model: |
| | |
| | ```bash |
| | cd Quark/examples/torch/language_modeling/llm_ptq/ |
| | exclude_layers="lm_head *self_attn* *mlp.gate *eh_proj *shared_head.head" |
| | python3 quantize_quark.py --model_dir $MODEL_DIR \ |
| | --quant_scheme w_mxfp4_a_mxfp4 \ |
| | --num_calib_data 32 \ |
| | --output_dir $output_dir \ |
| | --exclude_layers $exclude_layers \ |
| | --dataset pileval \ |
| | --multi_gpu \ |
| | --model_export hf_format \ |
| | --trust_remote_code \ |
| | --skip_evaluation \ |
| | --seq_len 512 |
| | ``` |
| | For further details or issues, please refer to the [AMD-Quark](https://quark.docs.amd.com/latest/index.html) documentation or contact the respective developers. |
| | |
| | # License |
| | Modifications Copyright(c) 2025 Advanced Micro Devices, Inc. All rights reserved. |
