davda54
/

wiki-retrieval-patch-base

Model card Files Files and versions

davda54 commited on Nov 22, 2023

Commit

34edb11

·

1 Parent(s): e7828e2

Upload tokenizer.json

Files changed (1) hide show

tokenizer.json +1 -1

tokenizer.json CHANGED Viewed

@@ -55,7 +55,7 @@
     "pretokenizers": [
       {
         "type": "ByteLevel",
-        "add_prefix_space": false,
         "trim_offsets": true,
         "use_regex": true
       },

     "pretokenizers": [
       {
         "type": "ByteLevel",
+        "add_prefix_space": true,
         "trim_offsets": true,
         "use_regex": true
       },