ThisenEkanayake
/

sinhala-tokenizer

Model card Files Files and versions

sinhala-tokenizer

Commit History

tokenizer class updated

e337c70

Thisen Ekanayake commited on Oct 1, 2025

renamed to "tokenizer"

6e0c6f7

Thisen Ekanayake commited on Oct 1, 2025

renamed

49d2aac

Thisen Ekanayake commited on Oct 1, 2025

updated tokenizer class

3841b6e

Thisen Ekanayake commited on Sep 30, 2025

Add custom Sinhala SentencePiece tokenizer

01e5ddf

Thisen Ekanayake commited on Sep 30, 2025

initial commit

48fae19
verified

ThisenEkanayake commited on Sep 30, 2025