timodonnell
/

protein-docs-tokenizer

timodonnell commited on Apr 7

Commit

83f597d

verified ·

1 Parent(s): ff742de

Upload tokenizer

Files changed (2) hide show

tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json CHANGED Viewed

@@ -16,7 +16,7 @@
       "single_word": false,
       "special": true
     },
-    "2840": {
       "content": "<UNK>",
       "lstrip": false,
       "normalized": false,

       "single_word": false,
       "special": true
     },
+    "2839": {
       "content": "<UNK>",
       "lstrip": false,
       "normalized": false,