tokenizer-fineweb_pl / tokenizer.json

Commit History

Register tokenizer as HF SentencePiece tokenizer
5e7a8b5
verified

suchirsalhan commited on