---
language:
- kab
tags:
- kabyle
- taqbaylit
- tokenizer
- sentence-segmentation
- stanza
- onnx
- tatoeba
license: apache-2.0
library_name: stanza
pipeline_tag: token-classification
---

# Kabyle Stanza Tokenizer

Sentence tokenizer for Kabyle (Taqbaylit) language, trained on Tatoeba corpus designed to be used on MiniSBD.

## Model Details

| Property | Value |
|----------|-------|
| Language | Kabyle (`kab`) |
| Type | Sentence tokenizer |
| Architecture | CNN + BiLSTM |
| Training data | Tatoeba Kabyle sentences (~789K sentences) |
| Dev F1 | 99.19% |
| Token F1 | 99.96% |
| Sentence F1 | 98.43% |
| ONNX size | 0.62 MB |
| Vocab size | 223 characters |

## Files

- `kab.onnx` — ONNX runtime model
- `kab.pt` — PyTorch checkpoint
- `vocab.json` — Character vocabulary (223 entries)
- `config.json` — Model hyperparameters
- `tokenizer_config.json` — HF tokenizer config

## Usage

### With Stanza (PyTorch)
```python
import stanza
nlp = stanza.Pipeline(
    lang="kab", 
    processors="tokenize", 
    tokenize_model_path="kab.pt"
)
doc = nlp("Amcic ha-t-an deg uxxam-nneɣ. Teciḍ fell-as?")
for sent in doc.sentences:
    print(sent.text)
# Amcic ha-t-an deg uxxam-nneɣ.
# Teciḍ fell-as?
```

### With ONNX Runtime
```python
import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("kab_tokenizer.onnx")
# units: (batch, seq_len) int64 — char IDs from vocab.json
# features: (batch, seq_len, 5) float32 — Stanza features
units = np.zeros((1, 100), dtype=np.int64)
features = np.zeros((1, 100, 5), dtype=np.float32)
outputs = sess.run(None, {"units": units, "features": features})
# outputs[0] shape: (batch, seq_len, 3) — logits for B/I/O
```

## Tokenization Examples

| Input | Tokens |
|-------|--------|
| `Ad tseddumt ɣer Taskriwt.` | `['Ad', 'tseddumt', 'ɣer', 'Taskriwt', '.']` |
| `Aweḍ ɣer Tezmalt.` | `['Aweḍ', 'ɣer', 'Tezmalt', '.']` |
| `Tettawḍem ɣer Kendira.` | `['Tettawḍem', 'ɣer', 'Kendira', '.']` |
| `Efk-asen tizwal-nni.` | `['Efk-asen', 'tizwal-nni', '.']` |
| `Melmi ara ad d-taɣeḍ lmitra?` | `['Melmi', 'ara', 'ad', 'd-taɣeḍ', 'lmitra', '?']` |

## Character Set

The tokenizer handles standard Kabyle Latin characters including:
- `ɛ` / `Ɛ` (open e)
- `ɣ` / `Ɣ` (voiced velar fricative)
- `ṭ`, `ḍ`, `č`, `ǧ` (emphatic and palatal consonants)
- Standard ASCII + punctuation

## Citation

If you use this model, please cite:
- Tatoeba project: https://tatoeba.org
- Stanza: Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, Christopher D. Manning. 2020. [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages.](https://aclanthology.org/2020.acl-demos.14/) ACL.

## License

Apache-2.0