用最新版Transformers报错的解决办法 AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'. Did you mean: '_tokenize'? return self.sp_tokenizer.num_tokens
#106
by PiaoYang - opened
使用最新版Transformers对Tokenizer进行加载的时候会报错
AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'. Did you mean: '_tokenize'?
return self.sp_tokenizer.num_tokens
解决办法：使用修复版的Tokenizer
将THUDM/chatglm-6b替换为PiaoYang/chatglm-6b即可。同时如果有revision参数，需要去掉。
tokenizer = AutoTokenizer.from_pretrained("PiaoYang/chatglm-6b", trust_remote_code=True)
PiaoYang changed discussion status to closed