kewang2 commited on
Commit
3a3d2a0
·
verified ·
1 Parent(s): db6920b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -2
README.md CHANGED
@@ -21,12 +21,14 @@ license: mit
21
 
22
  # Model Details
23
 
24
- This model is adapted from `unsloth/DeepSeek-R1-0528-BF16` by adding an MTP layer (specifically, the 61st layer) in the `modeling_deepseek.py` file. With this modification, it is possible to use AMD-Quark to quantize the DeepSeek-R1-0528 model with the MTP layer included.
 
 
25
 
26
  **Important Notes:**
27
  - When loading this model, you must set `trust_remote_code=True` to ensure that changes related to the MTP layer in `modeling_deepseek.py` take effect.
28
  - After loading this model with `transformers`, **evaluation should NOT be performed directly**. The reason is that the forward function for the added MTP layer in `modeling_deepseek.py` is implemented only for calibration during the quantization process, so computation is not guaranteed to be the same as the original DeepSeek-R1-0528.
29
- - Therefore, when quantizing with [AMD-Quark](https://quark.docs.amd.com/latest/index.html), you **must add the `--skip_evaluation` option** to skip the evaluation step and only perform quantization.
30
 
31
  Below is an example of how to quantize this model:
32
 
 
21
 
22
  # Model Details
23
 
24
+ In the original `modeling_deepseek.py` file from the `unsloth/DeepSeek-R1-0528-BF16` model, there is no definition or implementation of the MTP (Multi-Token-Predictor) layer. As a result, when you load the original model, there is no MTP layer included, and MTP-specific quantization cannot be performed.
25
+
26
+ To enable MTP layer loading and quantization, this model is adapted from `unsloth/DeepSeek-R1-0528-BF16` by adding an MTP layer in the `modeling_deepseek.py` file. With this modification, it is possible to use [AMD-Quark](https://quark.docs.amd.com/latest/index.html) to quantize the DeepSeek-R1-0528 model with the MTP layer included.
27
 
28
  **Important Notes:**
29
  - When loading this model, you must set `trust_remote_code=True` to ensure that changes related to the MTP layer in `modeling_deepseek.py` take effect.
30
  - After loading this model with `transformers`, **evaluation should NOT be performed directly**. The reason is that the forward function for the added MTP layer in `modeling_deepseek.py` is implemented only for calibration during the quantization process, so computation is not guaranteed to be the same as the original DeepSeek-R1-0528.
31
+ - Therefore, when quantizing with AMD-Quark, you **must add the `--skip_evaluation` option** to skip the evaluation step and only perform quantization.
32
 
33
  Below is an example of how to quantize this model:
34