konko
/

bornomala-bengali-tokenizer

Token Classification

Model card Files Files and versions

bornomala-bengali-tokenizer / tokenizer.json

Commit History

v0.2: 64k literary-weighted tokenizer, beats IndicBERTv2 on every held-out register

3ba25d0
verified

konko commited on 8 days ago

Add Bornomala Bengali tokenizer v0.1 (card + tokenizer files)

91965dc
verified

konko commited on 9 days ago