Update README.md
Browse files
README.md
CHANGED
|
@@ -8,15 +8,17 @@ tags:
|
|
| 8 |
|
| 9 |
This model is a mixed int4 model with group_size 128 and asymmetric quantization of [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5/) generated by [intel/auto-round](https://github.com/intel/auto-round). Please follow the license of the original model.
|
| 10 |
|
| 11 |
-
|
| 12 |
|
|
|
|
| 13 |
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
-
pip install git+https://github.com/vllm-project/vllm.git@main
|
| 16 |
|
| 17 |
-
```
|
| 18 |
pip install git+https://github.com/huggingface/transformers.git
|
| 19 |
-
|
| 20 |
|
| 21 |
|
| 22 |
|
|
|
|
| 8 |
|
| 9 |
This model is a mixed int4 model with group_size 128 and asymmetric quantization of [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5/) generated by [intel/auto-round](https://github.com/intel/auto-round). Please follow the license of the original model.
|
| 10 |
|
| 11 |
+
**The model is quantized with pure RTN mode**
|
| 12 |
|
| 13 |
+
### vllm inference
|
| 14 |
|
| 15 |
+
**Setup**
|
| 16 |
+
~~~bash
|
| 17 |
+
pip install git+https://github.com/vllm-project/vllm.git@main
|
| 18 |
|
|
|
|
| 19 |
|
|
|
|
| 20 |
pip install git+https://github.com/huggingface/transformers.git
|
| 21 |
+
~~~bash
|
| 22 |
|
| 23 |
|
| 24 |
|