metadata
library_name: transformers
tags:
- tokenizer
- sentencepiece
SentencePiece-based Hugging Face tokenizer
This repository contains a Hugging Face tokenizer converted from a SentencePiece model.
Load
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("lanstat0123/multilingual_tokenizer")
Special tokens
print(tokenizer.special_tokens_map)
print(len(tokenizer))