File size: 663 Bytes
e0146f3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | \---
license: apache-2.0
language:
  - ur
  - pa
  - sd
  - ps
  - fa
  - ku
tags:
  - tokenizer
  - multilingual
  - pakistan
  - ir
\---
\# Strategic Multilingual Tokenizer
Optimized for Pakistan, Iran, and South Asian languages. Provides up to 3X better compression on Urdu, Punjabi, Sindhi, and Persian compared to Mistral and Qwen2.
\## Usage
```python
from tokenizers import Tokenizer
tok = Tokenizer.from\_file("strategic\_tokenizer\_gpt4.json")
enc = tok.encode("مصنوعی ذہانت دنیا کو بدل رہی ہے")
|