MorphBPE tokenizer for Swahili (XNLI)
Faithful implementation of MorphBPE (Asgari et al., Findings of ACL 2026): a standard BPE tokenizer whose merges were constrained during training to never cross a morpheme boundary. Inference is identical to standard BPE on raw text — no morphological analyzer is required at test time.
| Parameter | Value |
|---|---|
| Vocabulary size | 18,414 |
| Special tokens | <s>, <pad>, </s>, <unk>, <mask> |
| Training data | facebook/xnli / sw / train split (785,404 sentences) |
| Normalization | NFC + lowercase (built into the tokenizer) |
| Pre-tokenizer | Whitespace (built into the tokenizer) |
| Morpheme source (training only) | Morfessor Baseline (unsupervised) |
| Metric (validation+test) | MorphBPE | plain BPE |
|---|---|---|
| Fertility (ϕ) | 1.6360 | 1.1483 |
| Tokens/char | 0.2898 | 0.2034 |
| Morph. Edit Distance (µₑ, lower=better) | 0.6564 | 1.4438 |
| Morph. Consistency F1 (µ_c, higher=better) | 0.6168 | 0.1711 |
Usage
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("NIRVLab/xnli-morphbpe-sw-32k")
tok.tokenize("Watoto wanasoma vitabu vyao shuleni!") # raw text, no pre-segmentation
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support