tokenizer-fineweb_de / tokenizer_config.json

Commit History

Register tokenizer as HF SentencePiece tokenizer
2d9aed2
verified

suchirsalhan commited on