my-tokenizer / config.json
Max1798's picture
Upload folder using huggingface_hub
c519dd6 verified
{
"model_type": "bpe",
"tokenizer_class": "BertWordPieceTokenizer",
"do_lower_case": true,
"pad_token_id": 3,
"unk_token_id": 0,
"sep_token_id": 2,
"cls_token_id": 1,
"mask_token_id": 4,
"vocab_size": 1000
}