| library_name: transformers | |
| tags: | |
| - tokenizer | |
| - sentencepiece | |
| # SentencePiece-based Hugging Face tokenizer | |
| This repository contains a Hugging Face tokenizer converted from a SentencePiece model. | |
| ## Load | |
| ```python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("lanstat0123/multilingual_tokenizer") | |
| ``` | |
| ## Special tokens | |
| ```python | |
| print(tokenizer.special_tokens_map) | |
| print(len(tokenizer)) | |
| ``` | |