gsaltintas commited on
Commit
e049556
·
verified ·
1 Parent(s): a860110

Upload mod_tokenizers/flexitok--mod-tokenizers-individual_vocab.json with huggingface_hub

Browse files
mod_tokenizers/flexitok--mod-tokenizers-individual_vocab.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"<unk>": [0], "~SPECIAL~ALIGNED~BOS~SYMBOL~": [1], "</s>": [2], "<pad>": [3], "mod": [4], "=": [5], " ": [6], "0": [7], "1": [8], "2": [9], "3": [10], "4": [11], "5": [12], "6": [13], "7": [14], "8": [15], "9": [16]}