Hugging Face's logo

openeurollm
/
tokenizer-128k

tokenizer
sentencepiece
bpe
multilingual
european-languages
Model card Files
xet
Community
tokenizer-128k / tokenizer_config.json

Commit History

Upload 128k vocab SentencePiece BPE tokenizer (trained on 175GB)
43f50d5
verified

timpal0l commited on