KTibow
/

bytetokenizer

KTibow commited on Apr 11, 2024

Commit

3d42eec

verified ·

1 Parent(s): 7cbf8b3

Update tokenizer_config.json

Files changed (1) hide show

tokenizer_config.json CHANGED Viewed

@@ -15,7 +15,7 @@
       "rstrip": false,
       "single_word": false,
       "special": true
-    },
   },
   "unk_token": "[UNK]",
   "mask_token": "[MASK]",

       "rstrip": false,
       "single_word": false,
       "special": true
+    }
   },
   "unk_token": "[UNK]",
   "mask_token": "[MASK]",