Splend1dchan commited on
Commit
94360f6
·
1 Parent(s): a6d85fe

change tokenizer to bert

Browse files
special_tokens_map.json DELETED
@@ -1 +0,0 @@
1
- {"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}
 
 
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
The diff for this file is too large to render. See raw diff
 
vocab.json DELETED
The diff for this file is too large to render. See raw diff
 
merges.txt → vocab.txt RENAMED
The diff for this file is too large to render. See raw diff