Add AutoTokenizer support (trust_remote_code)

README.md

@@ -33,6 +33,7 @@ pip install git+https://huggingface.co/Ethosoft/turk-tokenizer
 
 ## Quick Start
 
+**Direct usage:**
 ```python
 from turk_tokenizer import TurkTokenizer
 
@@ -43,6 +44,22 @@ for t in tokens:
     print(t["token"], t["token_type"], t["morph_pos"])
 ```
 
+**HuggingFace AutoTokenizer:**
+```python
+from transformers import AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("Ethosoft/turk-tokenizer", trust_remote_code=True)
+out = tok("İstanbul'da meeting'e katılamadım")
+
+out["input_ids"]             # hash-stable int IDs
+out["attention_mask"]        # [1, 1, 1, ...]
+out["token_type_ids"]        # 0=root, 1=suffix, 2=bpe, 3=punct, 4=num, 5=url/social
+out["morphological_tokens"]  # full morphological dicts
+
+# Batch:
+out = tok(["Türkçe metin.", "Another sentence."])
+```
+
 Output:
 ```
 <uppercase_word>  ROOT    0