Upload folder using huggingface_hub
Browse files
README.md
CHANGED
|
@@ -31,7 +31,7 @@ A **Byte-Level BPE** tokenizer trained on **rus_Cyrl** data from Fineweb-2-HQ.
|
|
| 31 |
```python
|
| 32 |
from transformers import AutoTokenizer
|
| 33 |
|
| 34 |
-
tokenizer = AutoTokenizer.from_pretrained("
|
| 35 |
tokens = tokenizer.encode("Hello, world!")
|
| 36 |
```
|
| 37 |
|
|
|
|
| 31 |
```python
|
| 32 |
from transformers import AutoTokenizer
|
| 33 |
|
| 34 |
+
tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_rus_Cyrl_32000")
|
| 35 |
tokens = tokenizer.encode("Hello, world!")
|
| 36 |
```
|
| 37 |
|