README.md

---
library_name: transformers
tags:
- tokenizer
- sentencepiece
---

# SentencePiece-based Hugging Face tokenizer

This repository contains a Hugging Face tokenizer converted from a SentencePiece model.

## Load

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("lanstat0123/multilingual_tokenizer")
```

## Special tokens

```python
print(tokenizer.special_tokens_map)
print(len(tokenizer))
```