ueg-classifier / tokenizer /tokenizer_config.json
rufatronics's picture
Add trained tokenizer
01e1741 verified
Raw
History Blame Contribute Delete
164 Bytes
{
"vocab_size": 32000,
"max_seq_len": 128,
"pad_id": 0,
"cls_id": 2,
"sep_id": 3,
"pad_token": "[PAD]",
"cls_token": "[CLS]",
"sep_token": "[SEP]"
}