deepl-project / save_tokenizer.py
picket-cliff's picture
Upload 46 files
43176db verified
raw
history blame contribute delete
202 Bytes
from transformers import DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
tokenizer.save_pretrained('saved_model')
print("Tokenizer saved to saved_model")