Intel
/

GLM-5-int4-mixed-AutoRound

text-generation-inference

4-bit precision

Model card Files Files and versions

wenhuach commited on Feb 26

Commit

3dc616e

·

verified ·

1 Parent(s): 6cd3fc1

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -25,7 +25,7 @@ pip install git+https://github.com/huggingface/transformers.git
 **MTP has not been supported, we will try to fix it later.**
 ~~~bash
-CUDA_VISIBLE_DEVICES=1,2,3,4 vllm serve  /xuehaosu/glm5_whc/GLM-5-w4g128/  \
      --tensor-parallel-size 4 \
      --gpu-memory-utilization 0.85 \
      --tool-call-parser glm47 \

 **MTP has not been supported, we will try to fix it later.**
 ~~~bash
+CUDA_VISIBLE_DEVICES=1,2,3,4 vllm serve  Intel/GLM-5-int4-mixed-AutoRound  \
      --tensor-parallel-size 4 \
      --gpu-memory-utilization 0.85 \
      --tool-call-parser glm47 \