SentencePiece-based Hugging Face tokenizer

This repository contains a Hugging Face tokenizer converted from a SentencePiece model.

Load

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("lanstat0123/multilingual_tokenizer")

print(tokenizer.special_tokens_map)
print(len(tokenizer))

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support