ardauzunoglu
/

kth_tokenizer

Model card Files Files and versions

ardauzunoglu commited on 29 days ago

Commit

90aed0b

·

verified ·

1 Parent(s): 35dbba9

Upload tokenizer

Files changed (1) hide show

tokenizer.json +6 -5

tokenizer.json CHANGED Viewed

@@ -73,11 +73,12 @@
       "K6": 21,
       "K7": 22,
       "K8": 23,
-      "K9": 24,
-      "K10": 25,
-      "END": 26,
-      "SELECT": 27,
-      "SEP": 28
     },
     "unk_token": "<unk>"
   }

       "K6": 21,
       "K7": 22,
       "K8": 23,
+      "EQUALS": 24,
+      "END": 25,
+      "K9": 26,
+      "K10": 27,
+      "SELECT": 28,
+      "SEP": 29
     },
     "unk_token": "<unk>"
   }