CodexTrouter / tokenizer.json

Create tokenizer.json

8a2b266 verified 4 months ago

141 Bytes

Invalid JSON: Unexpected token 'r', "from transf"... is not valid JSON

	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
	tokenizer.save_pretrained("./ProTalk/")