Hugging Face's logo

zirui3
/
llm-multilingual-tokenizer

Model card Files
xet
Community

YAML Metadata Warning: empty or missing yaml metadata in repo card

Check out the documentation for more information.

summary

multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.

  • vocab size: 100k
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support