Where did you get HF Tokenizer?
#2
by
Heejoong
- opened
Hello.
https://huggingface.co/mistralai/Devstral-Small-2507/tree/main
There are not tokenizer.json and tokenizer_config.json
So, I have some difficulties for SFT Training.
However, in your repository(https://huggingface.co/lmstudio-community/Devstral-Small-2507-MLX-bf16/tree/main), it looks that there is HF tokenizer.
How can you...?
Is there a way to convert sentence piece tokenizer to HF tokenizer? or can I use this tokenizer? https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/tree/main
Thank you in advance.