Instructions to use zai-org/glm-4-9b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use zai-org/glm-4-9b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="zai-org/glm-4-9b", trust_remote_code=True)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("zai-org/glm-4-9b", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use zai-org/glm-4-9b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "zai-org/glm-4-9b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zai-org/glm-4-9b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/zai-org/glm-4-9b
- SGLang
How to use zai-org/glm-4-9b with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "zai-org/glm-4-9b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zai-org/glm-4-9b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "zai-org/glm-4-9b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zai-org/glm-4-9b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use zai-org/glm-4-9b with Docker Model Runner:
docker model run hf.co/zai-org/glm-4-9b
TypeError: ChatGLM4Tokenizer._pad() got an unexpected keyword argument 'padding_side'
File "/cpfs01/user/xiaolinchen/opencompass_fork/opencompass/openicl/icl_inferencer/icl_ppl_inferencer.py", line 113, in inference
prompt_token_num = self.model.get_token_len_from_template(prompt, mode='ppl')
File "/cpfs01/user/xiaolinchen/opencompass_fork/opencompass/models/base.py", line 225, in get_token_len_from_template
token_lens = [self.get_token_len(prompt) for prompt in prompts]
File "/cpfs01/user/xiaolinchen/opencompass_fork/opencompass/models/base.py", line 225, in
token_lens = [self.get_token_len(prompt) for prompt in prompts]
File "/cpfs01/user/xiaolinchen/opencompass_fork/opencompass/models/huggingface_above_v4_33.py", line 638, in get_token_len
t = self.tokenizer(m, add_special_tokens=add_special_tokens)
File "/cpfs01/user/xiaolinchen/miniconda3/envs/oc_lmdeploy/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3016, in call
encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
File "/cpfs01/user/xiaolinchen/miniconda3/envs/oc_lmdeploy/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3126, in _call_one
return self.encode_plus(
File "/cpfs01/user/xiaolinchen/miniconda3/envs/oc_lmdeploy/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3202, in encode_plus
return self._encode_plus(
File "/cpfs01/user/xiaolinchen/miniconda3/envs/oc_lmdeploy/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 801, in _encode_plus
return self.prepare_for_model(
File "/cpfs01/user/xiaolinchen/miniconda3/envs/oc_lmdeploy/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3698, in prepare_for_model
encoded_inputs = self.pad(
File "/cpfs01/user/xiaolinchen/miniconda3/envs/oc_lmdeploy/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3500, in pad
encoded_inputs = self._pad(
TypeError: ChatGLM4Tokenizer._pad() got an unexpected keyword argument 'padding_side'
两种解决方案:一是删除padding_side观察会不会出错,二是可以修改tokenizer实现
