Upload tokenizer
#22
by ybelkada - opened
Uploads the transformers compatible version of the tokenizer, which is the same as https://huggingface.co/mistral-community/Codestral-22B-v0.1
Thanks! Codestral also has a suffix and prefix token that’s useful for FIM functionality, but we can add this later one supported by transformers
patrickvonplaten changed pull request status to merged