Update README.md
Browse files
README.md
CHANGED
|
@@ -11,7 +11,9 @@ Please follow the license of the original model.
|
|
| 11 |
|
| 12 |
## How To Use
|
| 13 |
### INT4 Inference
|
| 14 |
-
|
|
|
|
|
|
|
| 15 |
```python
|
| 16 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 17 |
import transformers
|
|
@@ -121,6 +123,7 @@ Here are the key points about the company:
|
|
| 121 |
* **Open-Source Contribution:** DeepSeak has made significant contributions to the open-source community. They have released powerful models like **DeepSeek-Coder** (focused on code generation and programming tasks) and the weights for earlier versions of their LLMs, allowing developers and researchers worldwide
|
| 122 |
--------------------------------------------------
|
| 123 |
"""
|
|
|
|
| 124 |
|
| 125 |
### Generate the model
|
| 126 |
Mian branch is required if the model is fp8 and the device supports fp8 https://github.com/intel/auto-round
|
|
|
|
| 11 |
|
| 12 |
## How To Use
|
| 13 |
### INT4 Inference
|
| 14 |
+
Potential overflow/underflow issues have been observed on CUDA, primarily due to kernel limitations.
|
| 15 |
+
For better accuracy, we recommend deploying the model on CPU or using [our INT4 mixed version](https://huggingface.co/Intel/DeepSeek-V3.1-int4-mixed-AutoRound)
|
| 16 |
+
|
| 17 |
```python
|
| 18 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 19 |
import transformers
|
|
|
|
| 123 |
* **Open-Source Contribution:** DeepSeak has made significant contributions to the open-source community. They have released powerful models like **DeepSeek-Coder** (focused on code generation and programming tasks) and the weights for earlier versions of their LLMs, allowing developers and researchers worldwide
|
| 124 |
--------------------------------------------------
|
| 125 |
"""
|
| 126 |
+
```
|
| 127 |
|
| 128 |
### Generate the model
|
| 129 |
Mian branch is required if the model is fp8 and the device supports fp8 https://github.com/intel/auto-round
|