gsaltintas commited on
Commit
2dd180a
·
verified ·
1 Parent(s): ee881d5

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -31,7 +31,7 @@ A **Byte-Level BPE** tokenizer trained on **dan_Latn** data from Fineweb-2-HQ.
31
  ```python
32
  from transformers import AutoTokenizer
33
 
34
- tokenizer = AutoTokenizer.from_pretrained("<repo_id>")
35
  tokens = tokenizer.encode("Hello, world!")
36
  ```
37
 
 
31
  ```python
32
  from transformers import AutoTokenizer
33
 
34
+ tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_dan_Latn_16000")
35
  tokens = tokenizer.encode("Hello, world!")
36
  ```
37