CodexTrouter / tokenizer.json
prelington's picture
Create tokenizer.json
8a2b266 verified
Invalid JSON: Unexpected token 'r', "from transf"... is not valid JSON
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
tokenizer.save_pretrained("./ProTalk/")