File size: 141 Bytes
8a2b266 |
1 2 3 4 5 |
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
tokenizer.save_pretrained("./ProTalk/")
|
8a2b266 |
1 2 3 4 5 |
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
tokenizer.save_pretrained("./ProTalk/")
|