goryden
/

mn_tokenizer

Model card Files Files and versions

goryden commited on May 13, 2025

Commit

8838402

·

verified ·

1 Parent(s): 4f74d9d

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -6,10 +6,10 @@ language:
 ---
 ## Usage
 ```python
-from transformers import T5Tokenizer
 text = "сайн уу"
-mn_tokenizer = T5Tokenizer(vocab_file="goryden/mn_tokenizer")
 tokens = mn_tokenizer.tokenize(text)
 encoded = mn_tokenizer.encode(text)

 ---
 ## Usage
 ```python
+from transformers import AutoTokenizer
 text = "сайн уу"
+AutoTokenizer.from_pretrained("goryden/mn_tokenizer")
 tokens = mn_tokenizer.tokenize(text)
 encoded = mn_tokenizer.encode(text)