Upload folder using huggingface_hub

README.md

@@ -26,13 +26,13 @@ A Korean language tokenizer with 96,000 vocabulary size, optimized for Korean te
 
 ## Usage
 
-### With Transformers Library
+### From Hugging Face Hub
 
 ```python
 from transformers import PreTrainedTokenizerFast
 
-# Load the tokenizer
-tokenizer = PreTrainedTokenizerFast.from_pretrained("./tokenizer_hf-96k")
+# Load the tokenizer from Hugging Face Hub
+tokenizer = PreTrainedTokenizerFast.from_pretrained("mrcha033/YunMin-tokenizer-96k")
 
 # Tokenize Korean text
 text = "안녕하세요, 한국어 토크나이저입니다."
@@ -47,20 +47,6 @@ decoded_text = tokenizer.decode(token_ids)
 print(f"Decoded: {decoded_text}")
 ```
 
-### With Tokenizers Library
-
-```python
-from tokenizers import Tokenizer
-
-# Load tokenizer
-tokenizer = Tokenizer.from_file("./tokenizer_hf-96k/tokenizer.json")
-
-# Encode text
-encoding = tokenizer.encode("안녕하세요, 한국어 토크나이저입니다.")
-print(f"Tokens: {encoding.tokens}")
-print(f"IDs: {encoding.ids}")
-```
-
 ## Special Tokens
 
 - `<unk>` - Unknown token