carry121 commited on
Commit
72ff25b
·
verified ·
1 Parent(s): ebfa1b4

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -20,7 +20,7 @@ Bilingual (Chinese / English) SentencePiece **Unigram** tokenizer with a
20
  ```python
21
  from transformers import AutoTokenizer
22
 
23
- tok = AutoTokenizer.from_pretrained("<namespace>/k3-32k-tokenizer")
24
  ids = tok.encode("深度学习是机器学习的一个分支。")
25
  # BOS (<s>, id=2) is prepended automatically (add_bos_token=True).
26
  ```
 
20
  ```python
21
  from transformers import AutoTokenizer
22
 
23
+ tok = AutoTokenizer.from_pretrained("carry121/k3-32k-tokenizer")
24
  ids = tok.encode("深度学习是机器学习的一个分支。")
25
  # BOS (<s>, id=2) is prepended automatically (add_bos_token=True).
26
  ```