YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

---

license: apache-2.0

language:

- ur

- pa

- sd

- ps

- fa

- ku

tags:

- tokenizer

- multilingual

- pakistan

- ir

---

# Strategic Multilingual Tokenizer

Optimized for Pakistan, Iran, and South Asian languages. Provides up to 3X better compression on Urdu, Punjabi, Sindhi, and Persian compared to Mistral and Qwen2.

## Usage


from tokenizers import Tokenizer

tok = Tokenizer.from\_file("strategic\_tokenizer\_gpt4.json")

enc = tok.encode("مصنوعی ذہانت دنیا کو بدل رہی ہے")

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support