Update README.md
Browse files
README.md
CHANGED
|
@@ -21,12 +21,14 @@ license: mit
|
|
| 21 |
|
| 22 |
# Model Details
|
| 23 |
|
| 24 |
-
|
|
|
|
|
|
|
| 25 |
|
| 26 |
**Important Notes:**
|
| 27 |
- When loading this model, you must set `trust_remote_code=True` to ensure that changes related to the MTP layer in `modeling_deepseek.py` take effect.
|
| 28 |
- After loading this model with `transformers`, **evaluation should NOT be performed directly**. The reason is that the forward function for the added MTP layer in `modeling_deepseek.py` is implemented only for calibration during the quantization process, so computation is not guaranteed to be the same as the original DeepSeek-R1-0528.
|
| 29 |
-
- Therefore, when quantizing with
|
| 30 |
|
| 31 |
Below is an example of how to quantize this model:
|
| 32 |
|
|
|
|
| 21 |
|
| 22 |
# Model Details
|
| 23 |
|
| 24 |
+
In the original `modeling_deepseek.py` file from the `unsloth/DeepSeek-R1-0528-BF16` model, there is no definition or implementation of the MTP (Multi-Token-Predictor) layer. As a result, when you load the original model, there is no MTP layer included, and MTP-specific quantization cannot be performed.
|
| 25 |
+
|
| 26 |
+
To enable MTP layer loading and quantization, this model is adapted from `unsloth/DeepSeek-R1-0528-BF16` by adding an MTP layer in the `modeling_deepseek.py` file. With this modification, it is possible to use [AMD-Quark](https://quark.docs.amd.com/latest/index.html) to quantize the DeepSeek-R1-0528 model with the MTP layer included.
|
| 27 |
|
| 28 |
**Important Notes:**
|
| 29 |
- When loading this model, you must set `trust_remote_code=True` to ensure that changes related to the MTP layer in `modeling_deepseek.py` take effect.
|
| 30 |
- After loading this model with `transformers`, **evaluation should NOT be performed directly**. The reason is that the forward function for the added MTP layer in `modeling_deepseek.py` is implemented only for calibration during the quantization process, so computation is not guaranteed to be the same as the original DeepSeek-R1-0528.
|
| 31 |
+
- Therefore, when quantizing with AMD-Quark, you **must add the `--skip_evaluation` option** to skip the evaluation step and only perform quantization.
|
| 32 |
|
| 33 |
Below is an example of how to quantize this model:
|
| 34 |
|