Hugging Face's logo

hf-internal-testing
/
test-two-tokenizers

Model card Files
xet
Community
sgugger commited on
Commit
d19581c
·
1 Parent(s): 41dd093

Initial commit

Browse files
Files changed (5) hide show
  1. special_tokens_map.json +1 -0
  2. tokenizer.4.0.0.json +0 -0
  3. tokenizer.json +0 -0
  4. tokenizer_config.json +1 -0
  5. vocab.txt +0 -0
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
tokenizer.4.0.0.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"do_lower_case": false, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "bert-base-cased", "fast_tokenizer_files": ["tokenizer.4.0.0.json"], "tokenizer_class": "BertTokenizer"}
vocab.txt ADDED
The diff for this file is too large to render. See raw diff