gsaltintas commited on
Commit
e118a93
·
verified ·
1 Parent(s): 9b94857

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -31,7 +31,7 @@ A **Byte-Level BPE** tokenizer trained on **rus_Cyrl** data from Fineweb-2-HQ.
31
  ```python
32
  from transformers import AutoTokenizer
33
 
34
- tokenizer = AutoTokenizer.from_pretrained("<repo_id>")
35
  tokens = tokenizer.encode("Hello, world!")
36
  ```
37
 
 
31
  ```python
32
  from transformers import AutoTokenizer
33
 
34
+ tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_rus_Cyrl_32000")
35
  tokens = tokenizer.encode("Hello, world!")
36
  ```
37