crypto-mini-embed / tokenizer.json
0xcubin's picture
Create tokenizer.json
fb82757 verified
{
"model": {
"type": "character",
"characters": "abcdefghijklmnopqrstuvwxyz0123456789 .,:;-_/\"'?!#@$%^&*()[]{}"
},
"unk_token": "[UNK]",
"pad_token": "[PAD]"
}