ArthurConmy
/

bugtest-tokenizer

Model card Files Files and versions

ArthurConmy commited on Nov 6, 2023

Commit

05f8cec

·

1 Parent(s): 4dae664

Update tokenizer.json

Files changed (1) hide show

tokenizer.json +1 -3

tokenizer.json CHANGED Viewed

@@ -41,9 +41,7 @@
     "continuing_subword_prefix": null,
     "end_of_word_suffix": null,
     "fuse_unk": false,
-    "vocab": {
-      "hello": 0
-    },
     "merges": []
   }
 }

     "continuing_subword_prefix": null,
     "end_of_word_suffix": null,
     "fuse_unk": false,
+    "vocab": {},
     "merges": []
   }
 }