Gunulhona
/

tb_tokenizer_big

Model card Files Files and versions

tb_tokenizer_big / special_tokens_map.json

Gunulhona's picture

add tokenizer

1d6f15b almost 4 years ago

history blame contribute delete

112 Bytes

{"bos_token": "[BOS]", "eos_token": "[EOS]", "unk_token": "[UNK]", "pad_token": "[PAD]", "mask_token": "[MASK]"}