SindhiLM-Tokenizer-v1 / tokenizer.json

Commit History

Upgraded architecture: Unigram + Metaspace fix + 32k Vocab limit + Morpheme splitting
be8d37f
verified

aakashMeghwar01 commited on

Upgraded architecture: Unigram + Metaspace fix + 32k Vocab limit + Morpheme splitting
f82968f
verified

aakashMeghwar01 commited on