README.md

# Cohere `rerank-multilingual-v2.0` tokenizer 

This is the tokenizer for the [Cohere Rerank Model](https://txt.cohere.com/rerank/).

You can load it with the transformers library like this:
```python

from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("Cohere/rerank-multilingual-v2.0")

text = "Hello World, this is my input string!"

enc = tokenizer(text)

print("Encoded input:")

print(enc)



inv_vocab = {v: k for k, v in tokenizer.vocab.items()}

tokens = [inv_vocab[token_id] for token_id in enc['input_ids']]

print("Tokens:")

print(tokens)



number_of_tokens = len(enc['input_ids'])

print("Number of tokens:", number_of_tokens)

```