Push tokenizer again
#4
by tomaarsen HF Staff - opened
In this PR, I'm repushing the tokenizer with a newer version of transformers, with the goal of also generating the tokenizer.json used by the tokenizers-backed fast tokenizers. There should not be any changes in the performance of the tokenizer, the only difference is that Transformers and Sentence Transformers can now directly use the fast tokenizer without having to convert it from the slow tokenizer. This should make loading the model faster.
- Tom Aarsen
tomaarsen changed pull request status to merged