Hindi
Marathi
English
tokenizers
tokenizer
bpe
hinglish
minglish
code-mixed
indic
nlp
research-paper
akshar-32k / tokenizer.json
Sujalvc's picture
Initial release of Akshar: The High-Efficiency Desi Tokenizer
0c87312 verified
raw
history contribute delete
2.3 MB
File too large to display, you can check the raw version instead.