Mini / tokenizer.json
Ananthusajeev190's picture
Upload 8 files
8749343 verified
{
"type": "byte-level",
"vocab_size": 256,
"special_tokens": {
"pad": 0,
"unk": 1
}
}