WillHeld
/

marin-tomol

WillHeld commited on Jan 25

Commit

a1ae6c9

verified ·

1 Parent(s): 476beed

Add tokenizer

Files changed (2) hide show

tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json CHANGED Viewed

@@ -24,7 +24,7 @@
       "single_word": false,
       "special": true
     },
-    "5768": {
       "content": "[UNK]",
       "lstrip": false,
       "normalized": false,

       "single_word": false,
       "special": true
     },
+    "7292": {
       "content": "[UNK]",
       "lstrip": false,
       "normalized": false,