| --- |
| language: |
| - kab |
| tags: |
| - kabyle |
| - taqbaylit |
| - tokenizer |
| - sentence-segmentation |
| - stanza |
| - onnx |
| - tatoeba |
| license: apache-2.0 |
| library_name: stanza |
| pipeline_tag: token-classification |
| --- |
| |
| # Kabyle Stanza Tokenizer |
|
|
| Sentence tokenizer for Kabyle (Taqbaylit) language, trained on Tatoeba corpus designed to be used on MiniSBD. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |----------|-------| |
| | Language | Kabyle (`kab`) | |
| | Type | Sentence tokenizer | |
| | Architecture | CNN + BiLSTM | |
| | Training data | Tatoeba Kabyle sentences (~789K sentences) | |
| | Dev F1 | 99.19% | |
| | Token F1 | 99.96% | |
| | Sentence F1 | 98.43% | |
| | ONNX size | 0.62 MB | |
| | Vocab size | 223 characters | |
|
|
| ## Files |
|
|
| - `kab.onnx` — ONNX runtime model |
| - `kab.pt` — PyTorch checkpoint |
| - `vocab.json` — Character vocabulary (223 entries) |
| - `config.json` — Model hyperparameters |
| - `tokenizer_config.json` — HF tokenizer config |
|
|
| ## Usage |
|
|
| ### With Stanza (PyTorch) |
| ```python |
| import stanza |
| nlp = stanza.Pipeline( |
| lang="kab", |
| processors="tokenize", |
| tokenize_model_path="kab.pt" |
| ) |
| doc = nlp("Amcic ha-t-an deg uxxam-nneɣ. Teciḍ fell-as?") |
| for sent in doc.sentences: |
| print(sent.text) |
| # Amcic ha-t-an deg uxxam-nneɣ. |
| # Teciḍ fell-as? |
| ``` |
|
|
| ### With ONNX Runtime |
| ```python |
| import onnxruntime as ort |
| import numpy as np |
| |
| sess = ort.InferenceSession("kab_tokenizer.onnx") |
| # units: (batch, seq_len) int64 — char IDs from vocab.json |
| # features: (batch, seq_len, 5) float32 — Stanza features |
| units = np.zeros((1, 100), dtype=np.int64) |
| features = np.zeros((1, 100, 5), dtype=np.float32) |
| outputs = sess.run(None, {"units": units, "features": features}) |
| # outputs[0] shape: (batch, seq_len, 3) — logits for B/I/O |
| ``` |
|
|
| ## Tokenization Examples |
|
|
| | Input | Tokens | |
| |-------|--------| |
| | `Ad tseddumt ɣer Taskriwt.` | `['Ad', 'tseddumt', 'ɣer', 'Taskriwt', '.']` | |
| | `Aweḍ ɣer Tezmalt.` | `['Aweḍ', 'ɣer', 'Tezmalt', '.']` | |
| | `Tettawḍem ɣer Kendira.` | `['Tettawḍem', 'ɣer', 'Kendira', '.']` | |
| | `Efk-asen tizwal-nni.` | `['Efk-asen', 'tizwal-nni', '.']` | |
| | `Melmi ara ad d-taɣeḍ lmitra?` | `['Melmi', 'ara', 'ad', 'd-taɣeḍ', 'lmitra', '?']` | |
|
|
| ## Character Set |
|
|
| The tokenizer handles standard Kabyle Latin characters including: |
| - `ɛ` / `Ɛ` (open e) |
| - `ɣ` / `Ɣ` (voiced velar fricative) |
| - `ṭ`, `ḍ`, `č`, `ǧ` (emphatic and palatal consonants) |
| - Standard ASCII + punctuation |
|
|
| ## Citation |
|
|
| If you use this model, please cite: |
| - Tatoeba project: https://tatoeba.org |
| - Stanza: Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, Christopher D. Manning. 2020. [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages.](https://aclanthology.org/2020.acl-demos.14/) ACL. |
|
|
| ## License |
|
|
| Apache-2.0 |
