RA-ALTA
/

tokenizer-ar

suchirsalhan commited on 8 days ago

Commit

a76cdb7

verified ·

1 Parent(s): 90447ca

Final Rescue: Full vocab restoration with Unigram scores and Metaspace fix

Files changed (1) hide show

tokenizer.json CHANGED Viewed

@@ -43,7 +43,7 @@
   "normalizer": null,
   "pre_tokenizer": {
     "type": "Metaspace",
-    "replacement": " ",
     "prepend_scheme": "always",
     "split": true
   },
@@ -112,7 +112,7 @@
   },
   "decoder": {
     "type": "Metaspace",
-    "replacement": " ",
     "prepend_scheme": "always",
     "split": true
   },

   "normalizer": null,
   "pre_tokenizer": {
     "type": "Metaspace",
+    "replacement": "▁",
     "prepend_scheme": "always",
     "split": true
   },
   },
   "decoder": {
     "type": "Metaspace",
+    "replacement": "▁",
     "prepend_scheme": "always",
     "split": true
   },