'ChatGLMTokenizer' object has attribute 'sp_tokenizer'
#109
by LavanyaSuresh - opened
I tried to convert the model to openvino , By using below comments ,
Safetensors Models are still missing from Chatglm-6B https://huggingface.co/THUDM/chatglm-6b/tree/main ,& working model https://huggingface.co/THUDM/chatglm3-6b/tree/main Please have a compare,
Still facing sp_tokenizer issue and Https response issue..
Usage:
python 3.10.11
Ov latest toolkit
Could you please to resolve the issue ..
我使用THUDM/chatglm-6b的tokenizer也遇到了类似的情况：
Python：3.11.10
transformers：4.45.2
降级到4.40，只有GLM-4有继续维护到这个版本
看了之前的帖子，用了新版本：PiaoYang/chatglm-6b
tokenizer = AutoTokenizer.from_pretrained("PiaoYang/chatglm-6b",trust_remote_code=True)
可以使用。