thai-nlp-toolkit / tokenizer /tokenizer_config.json
puttimej's picture
Upload tokenizer/tokenizer_config.json with huggingface_hub
18e411d verified
Raw
History Blame Contribute Delete
215 Bytes
{
"vocab_size": 500,
"model_file": "thai_bpe.model",
"special_tokens": {
"pad_id": 0,
"unk_id": 1,
"bos_id": 2,
"eos_id": 3,
"cls_id": 4,
"sep_id": 5,
"mask_id": 6
}
}