tokenizer-fineweb_ar / tokenizer.json

Commit History

Register tokenizer as HF SentencePiece tokenizer
8d82b02
verified

suchirsalhan commited on