Spaces:

oddadmix
/

Arabic-Tokenizer-Comparison

Sleeping

App Files Files Community

Arabic-Tokenizer-Comparison / app.py

Commit History

Only Emhotob among Arabic tokenizers; fix byte-level BPE token display

e841b14
verified

oddadmix commited on 9 days ago

Arabic tokenizer comparison: 9 tokenizers, MSA/dialect/code-switch

0b999a6
verified

oddadmix commited on 9 days ago