lemon-mint
/

KoEnNanoLMv2-Tokenizer

lemon-mint commited on Jul 9, 2024

Commit

b7c7f4e

verified ·

1 Parent(s): 13c33f9

Upload tokenizer

Files changed (2) hide show

tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json CHANGED Viewed

@@ -32,7 +32,7 @@
       "single_word": false,
       "special": true
     },
-    "55734": {
       "content": "<mask>",
       "lstrip": true,
       "normalized": false,

       "single_word": false,
       "special": true
     },
+    "65052": {
       "content": "<mask>",
       "lstrip": true,
       "normalized": false,