Add tokenizer

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,21 @@
+# Cohere `rerank-multilingual-v3.0` tokenizer
+
+This is the tokenizer for the [Cohere Rerank](https://cohere.com/rerank) Multilingual v3 model.
+
+You can load it with the tokenizers library like this:
+
+```python
+from tokenizers import Tokenizer
+
+tokenizer = Tokenizer.from_pretrained("Cohere/rerank-multilingual-v3.0")
+text = "Hello World, this is my input string!"
+enc = tokenizer.encode(text)
+print("Encoded input:")
+print(enc.ids)
+
+print("Tokens:")
+print(enc.tokens)
+
+number_of_tokens = len(enc.ids)
+print("Number of tokens:", number_of_tokens)
+```

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f39413cfc8829996be621f4fbea8b76f09b6a9d07724fa4fbab32292cb80d5f
+size 16316179