POSNEGPRETRAINED / tokenizer_config.json
chaosbringerc's picture
Update tokenizer_config.json
e405e67 verified
Raw
History Blame Contribute Delete
221 Bytes
Invalid JSON:Unexpected token 'r', "from transf"... is not valid JSON
from transformers import AutoTokenizer
# Load the correct base tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Save tokenizer config properly
tokenizer.save_pretrained("./POSNEG-Dataset-API")