IndexError: piece id is out of range.
#5
by godcrying - opened
使用github上的项目 https://github.com/hiyouga/LLaMA-Factory/ 微调后，对话报错 IndexError: piece id is out of range.
使用github上的项目 https://github.com/hiyouga/LLaMA-Factory/ 微调后，对话报错 IndexError: piece id is out of range.
参考https://huggingface.co/THUDM/chatglm3-6b/commit/ea563876364622a0a5c24e6b71db0b93a9861ba0#d2h-069285
在tokenization_chatglm.py里新增两行代码
We have been paying attention to this problem for a long time. Although we suspect that it may be caused by the user did not input in the correct format, we still added this line of code . Minimize the occurrence of accidents.
thank you for your suggestion
ZHANGYUXUAN-zR changed discussion status to closed