Initial commit: KashTok tokenizer

README.md

@@ -0,0 +1,93 @@
+---
+language:
+- ks
+- kas
+license: apache-2.0
+tags:
+- tokenizer
+- kashmiri
+- low-resource
+- perso-arabic
+- character
+library_name: transformers
+---
+
+# Character Tokenizer for Kashmiri
+
+A Character tokenizer for Kashmiri (ISO 639-3: `kas`) trained on
+**KS-LIT-3M**, a 3.1M-word literary corpus. Released as part of the
+**KashTok** study (Malik et al., 2026), the first systematic
+linguist-verified tokenization comparison for Kashmiri.
+
+## Quick Start
+
+```python
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("Omarrran/Kashmiri_Char_Tokenizer")
+
+text = "کٲشِر زَبان چھِیہٕ خٲص زَبان"
+encoding = tokenizer(text, return_tensors="pt")
+print(tokenizer.tokenize(text))
+print(encoding.input_ids)
+```
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| Tokenizer type | Character |
+| Vocabulary size | 133 |
+| Training corpus | KS-LIT-3M (2.47M words, 129,672 train segments) |
+| Special tokens | `[PAD]` `[UNK]` `[CLS]` `[SEP]` `[MASK]` |
+| Max sequence length | 512 |
+| Pre-tokenization | NFC + KS_CHAR_MAP normalization |
+
+## Evaluation Metrics
+
+Computed on **16,209 truly-unseen** held-out test segments from KS-LIT-3M:
+
+| Metric | Value |
+|---|---|
+| Composite Quality Score (CQS) | 0.3107 |
+| Fertility (↓ better) | 5.2453 |
+| Diacritic Preservation Score (↑) | 0.0000 |
+| Morphological Boundary Alignment (↑) | 0.2104 |
+| Out-of-Vocabulary rate (↓) | 0.0000 |
+| Reconstruction (char-level, ↑) | N/A |
+
+See [the paper](https://arxiv.org/) for full evaluation methodology and
+the linguist-verified gold morpheme reference.
+
+## Recommended Use
+
+**Primary use case:** ASR/OCR post-correction, character-level models, error analysis
+
+133 vocab — one token per Unicode codepoint. Fertility 5.25.
+
+## Companion Repositories
+
+The other four KashTok tokenizers are also available for direct comparison:
+
+- [Kashmiri_Char_Tokenizer](https://huggingface.co/Omarrran/Kashmiri_Char_Tokenizer)
+- [Kashmiri_Word_Tokenizer](https://huggingface.co/Omarrran/Kashmiri_Word_Tokenizer)
+- [Kashmiri_WordPiece_Tokenizer](https://huggingface.co/Omarrran/Kashmiri_WordPiece_Tokenizer)
+- [Kashmiri_BPE_Tokenizer](https://huggingface.co/Omarrran/Kashmiri_BPE_Tokenizer)
+- [Kashmiri_Unigram_Tokenizer](https://huggingface.co/Omarrran/Kashmiri_Unigram_Tokenizer)
+
+## Citation
+
+```bibtex
+@article{malik2026kashtok,
+  title  = {KashTok: Tokenizing Kashmiri at Scale with Novel
+            Diacritic- and Morphology-Aware Metrics},
+  author = {Malik, Haq Nawaz and Nissar, Nahfid and others},
+  year   = {2026}
+}
+```
+
+## Linguistic Verification
+
+Every Kashmiri character, diacritic, and morpheme split used in the
+evaluation of this tokenizer was confirmed by a native-Kashmiri-speaker
+linguistic review (40 consonants, 7 vowels, 11 diacritics, 26 gold
+morpheme splits).

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "pad_token": "[PAD]",
+  "unk_token": "[UNK]",
+  "cls_token": "[CLS]",
+  "sep_token": "[SEP]",
+  "mask_token": "[MASK]"
+}

tokenizer.json

@@ -0,0 +1,229 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "[PAD]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "[UNK]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "[CLS]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "[SEP]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "[MASK]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": {
+    "type": "NFC"
+  },
+  "pre_tokenizer": {
+    "type": "Split",
+    "pattern": {
+      "String": ""
+    },
+    "behavior": "Isolated",
+    "invert": false
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {}
+  },
+  "decoder": null,
+  "model": {
+    "type": "WordLevel",
+    "vocab": {
+      "[PAD]": 0,
+      "[UNK]": 1,
+      "[CLS]": 2,
+      "[SEP]": 3,
+      "[MASK]": 4,
+      "[BOS]": 5,
+      "[EOS]": 6,
+      "\n": 7,
+      " ": 8,
+      "!": 9,
+      "\"": 10,
+      "'": 11,
+      "-": 12,
+      ".": 13,
+      "«": 14,
+      "»": 15,
+      "؁": 16,
+      "،": 17,
+      "؎": 18,
+      "ؐ": 19,
+      "ؑ": 20,
+      "ؒ": 21,
+      "ؓ": 22,
+      "ؔ": 23,
+      "؛": 24,
+      "؟": 25,
+      "ؠ": 26,
+      "ء": 27,
+      "آ": 28,
+      "أ": 29,
+      "ؤ": 30,
+      "إ": 31,
+      "ئ": 32,
+      "ا": 33,
+      "ب": 34,
+      "ت": 35,
+      "ث": 36,
+      "ج": 37,
+      "ح": 38,
+      "خ": 39,
+      "د": 40,
+      "ذ": 41,
+      "ر": 42,
+      "ز": 43,
+      "س": 44,
+      "ش": 45,
+      "ص": 46,
+      "ض": 47,
+      "ط": 48,
+      "ظ": 49,
+      "ع": 50,
+      "غ": 51,
+      "ف": 52,
+      "ق": 53,
+      "ك": 54,
+      "ل": 55,
+      "م": 56,
+      "ن": 57,
+      "ه": 58,
+      "و": 59,
+      "ً": 60,
+      "ٍ": 61,
+      "َ": 62,
+      "ُ": 63,
+      "ِ": 64,
+      "ّ": 65,
+      "ْ": 66,
+      "ٓ": 67,
+      "ٔ": 68,
+      "ٕ": 69,
+      "ٖ": 70,
+      "ٗ": 71,
+      "٘": 72,
+      "ٚ": 73,
+      "ٛ": 74,
+      "ٟ": 75,
+      "٠": 76,
+      "١": 77,
+      "٢": 78,
+      "٣": 79,
+      "٤": 80,
+      "٥": 81,
+      "٦": 82,
+      "٧": 83,
+      "٨": 84,
+      "٩": 85,
+      "٪": 86,
+      "٭": 87,
+      "ٮ": 88,
+      "ٮ۪": 89,
+      "ٰ": 90,
+      "ٲ": 91,
+      "ٳ": 92,
+      "ٹ": 93,
+      "پ": 94,
+      "ٿ": 95,
+      "ڀ": 96,
+      "چ": 97,
+      "ڈ": 98,
+      "ڑ": 99,
+      "ژ": 100,
+      "ڙ": 101,
+      "ک": 102,
+      "ڪ": 103,
+      "ڮ": 104,
+      "گ": 105,
+      "ں": 106,
+      "ھ": 107,
+      "ہ": 108,
+      "ۂ": 109,
+      "ۃ": 110,
+      "ۄ": 111,
+      "ۅ": 112,
+      "ۆ": 113,
+      "ی": 114,
+      "ۍ": 115,
+      "ے": 116,
+      "ۓ": 117,
+      "۔": 118,
+      "۪": 119,
+      "ۭ": 120,
+      "۰": 121,
+      "۱": 122,
+      "۲": 123,
+      "۳": 124,
+      "۴": 125,
+      "۵": 126,
+      "۶": 127,
+      "۷": 128,
+      "۸": 129,
+      "۹": 130,
+      "﴾": 131,
+      "﴿": 132
+    },
+    "unk_token": "[UNK]"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}