konko
/

bornomala-bengali-tokenizer

Token Classification

Model card Files Files and versions

bornomala-bengali-tokenizer / tokenizer.json

konko's picture

v0.2: 64k literary-weighted tokenizer, beats IndicBERTv2 on every held-out register

3ba25d0 verified 8 days ago

History Contribute Delete

4.76 MB

File too large to display, you can check the raw version instead.