Hindi
Marathi
English
tokenizers
tokenizer
bpe
hinglish
minglish
code-mixed
indic
nlp
research-paper
akshar-32k / tokenizer.json

Commit History

Initial release of Akshar: The High-Efficiency Desi Tokenizer
0c87312
verified

Sujalvc commited on

Upload Akshar-32k tokenizer and README
5a32612
verified

Sujalvc commited on