\---

license: apache-2.0

language: 

  - ur

  - pa

  - sd

  - ps

  - fa

  - ku

tags:

  - tokenizer

  - multilingual

  - pakistan

  - ir

\---



\# Strategic Multilingual Tokenizer



Optimized for Pakistan, Iran, and South Asian languages. Provides up to 3X better compression on Urdu, Punjabi, Sindhi, and Persian compared to Mistral and Qwen2.



\## Usage



```python

from tokenizers import Tokenizer

tok = Tokenizer.from\_file("strategic\_tokenizer\_gpt4.json")

enc = tok.encode("مصنوعی ذہانت دنیا کو بدل رہی ہے")