Shu-vi
/

Russian_BPE_Tokenizer_16k

Model card Files Files and versions

Shu-vi commited on Oct 10, 2025

Commit

c41999b

·

verified ·

1 Parent(s): bd4d957

Update README.md

Files changed (1) hide show

README.md +1 -2

README.md CHANGED Viewed

@@ -65,11 +65,10 @@ metrics:
 ```python
 from tokenizers import Tokenizer
-# Загрузка токенизатора (если загружен с Hugging Face Hub)
 tokenizer = Tokenizer.from_pretrained("Shu-vi/russian-bpe-tokenizer-16k")
 # Пример
-text = "В Москве прошёл международный экономический форум."
 encoded = tokenizer.encode(text)
 print("Токены:", encoded.tokens)

 ```python
 from tokenizers import Tokenizer
 tokenizer = Tokenizer.from_pretrained("Shu-vi/russian-bpe-tokenizer-16k")
 # Пример
+text = "В Казани в 2024 прошёл БРИКС."
 encoded = tokenizer.encode(text)
 print("Токены:", encoded.tokens)