|
|
| --- |
| license: apache-2.0 |
| --- |
| |
| # Overview |
|
|
| This repository contains a **Grapheme-Aware Tokenizer (GAT)** specifically trained for **Kannada**, designed to handle the unique orthographic and phonological structure of the language. Unlike traditional subword tokenizers such as BPE, SentencePiece, or WordPiece, this tokenizer operates at the **grapheme level**, improving representation fidelity and reducing subword fragmentation. |
|
|
| --- |
|
|
| # Available Vocabulary Sizes |
|
|
| This repository includes **three tokenizer variants**: |
|
|
| | Vocabulary | File | |
| |------------|-------| |
| | **8k** | `GAT_Kannada_8k.json` | |
| | **16k** | `GAT_Kannada_16k.json` | |
| | **32k** | `GAT_Kannada_32k.json` *(recommended)* | |
|
|
| --- |
|
|
| # Why Grapheme-Aware Preprocessing? |
|
|
| Kannada is an **Abugida** script where a single grapheme (akshara) may be composed of: |
|
|
| - multiple consonants |
| - a halant (virama) |
| - vowel diacritics (matra) |
|
|
| For example: |
|
|
| ಕ್ರಿ |
|
|
| is **one grapheme**, but composed of multiple Unicode codepoints. |
|
|
| ಕ್ + ರ್ + ಿ → 3–4 fragments |
|
|
| ### Problem with BPE / SentencePiece / WordPiece |
|
|
| These tokenizers operate at the byte or character level: |
|
|
| This results in: |
|
|
| - stable semantic units |
| - better compression |
| - more efficient tokenization |
| - |
| ### GAT Solution |
|
|
| GAT applies a **custom grapheme parser** that merges the components into **one atomic unit**: |
|
|
| GAT uses a rule-based finite-state parser that correctly handles: |
|
|
| - consonants |
| - vowels |
| - halants |
| - vowel signs |
| - anusvara & visarga |
|
|
| <p align="center"> |
| <img src="./GAT-algo.png" width="650"/> |
| </p> |
|
|
| After grapheme segmentation, **Byte Pair Encoding (BPE)** is applied to learn higher-level merges. |
|
|
| --- |
|
|
| # Training Data |
|
|
| Tokenizer training uses a **composite 4.5M-sentence Kannada corpus**: |
|
|
| 1. **Samanantar Dataset** (AI4Bharat) |
| 2. **Kannada-Instruct Dataset** (Cognitive Lab) |
|
|
| This provides broad coverage of conversational, literary, and instruction-following Kannada. |
|
|
| --- |
|
|
| # Tokenizer Metrics |
|
|
| These metrics evaluate tokenizer quality independent of any downstream NLP model. |
|
|
| ## **Compression Ratio (CR)** |
| Higher = better (larger text compressed into fewer bytes) |
|
|
| ## **Fertility Score (FS)** |
| Lower = better (#tokens produced per grapheme/character) |
|
|
| ### **Results for CR and FS** |
|
|
| GAT consistently showed better compression ratio and fertility score across vocab sizes . |
|
|
| --- |
| CR : 3.5 -> 3.9 -> 4.8 |
|
|
| --- |
| FS : 2.1 -> 1.8 -> 1.6 |
|
|
| --- |
|
|
| # 💻 Usage Example |
|
|
| ### Load the 32k tokenizer |
|
|
| ```python |
| from transformers import PreTrainedTokenizerFast |
| |
| tokenizer = PreTrainedTokenizerFast.from_pretrained( |
| "varuni/GAT-K", |
| tokenizer_file="GAT_Kannada_32k.json" |
| ) |
| |
| text = "ನಿಮ್ಮ ಹೆಸರು ಏನು?" |
| print(tokenizer.encode(text)) |
| ``` |
| # Related work : |
|
|
| - M. Velayuthan and K. Sarveswaran, “Egalitarian Language Representation in Language Models: It All Begins with Tokenizers,” COLING 2025. |
| arXiv:2409.11501 [cs.CL]. DOI: https://doi.org/10.48550/arXiv.2409.11501 |
|
|
| - Unicode Normalization and Grapheme Parsing of Indic Languages, 2023. [Online]. Available: https://arxiv.org/abs/2306.01743 |
|
|
| - M. K. H. and A. Giri, “Orthographic Syllable Pair Encoding for Language Modelling Tasks in Indic Languages,” in 2023 IEEE MIT Undergraduate Research Technology Conference (URTC), Cambridge, MA, USA, pp. 1–6, 2023. |
| DOI: https://doi.org/10.1109/URTC60662.2023.10534970 |
| |
| - R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), 2016. |
| [Online]. Available: https://arxiv.org/abs/1508.07909 |
|
|
|
|
|
|
