zai-org

/

glm-4-9b-chat

like 706

Follow

Z.ai 13.5k

我已经使用transformers==4.40.2，为什么报错

File "/works/app/src/llmtuner/data/template.py", line 118, in _convert_elements_to_ids
token_ids += [tokenizer.convert_tokens_to_ids(elem.get("token"))]
File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 645, in convert_tokens_to_ids
return self._convert_token_to_id_with_added_voc(tokens)
File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 658, in _convert_token_to_id_with_added_voc
return self._convert_token_to_id(token)
File "/root/.cache/huggingface/modules/transformers_modules/tokenization_chatglm.py", line 95, in _convert_token_to_id
return self.mergeable_ranks[token]
KeyError: 'sop'
谢谢，帮忙看下