thedeba
/

deb-bpe

Model card Files Files and versions

deb-bpe / tokenizer_config.json

Commit History

Add bilingual Bengali-English BPE tokenizer (100k vocab, byte_fallback)

9a0fd03
verified

thedeba commited on May 30