Micro / tokenizer_config.json
Ananthusajeev190's picture
Upload 6 files
eeecd10 verified
{
"tokenizer_type": "char",
"vocab_file": "vocab.json",
"do_lower_case": false,
"unk_token": "<UNK>",
"pad_token": "<PAD>",
"bos_token": "<BOS>",
"eos_token": "<EOS>"
}