README.md

---
language:
- am
- ti
- tig
- gez
- en
license: unknown
tags:
- tokenizer
- sentencepiece
- unigram
- geez-script
- amharic
- tigrinya
- tigre
- geez
- low-resource
---

# geez-en-shared-tokenizer

A shared SentencePiece **Unigram** tokenizer (vocab size **120,000**,
`byte_fallback=true`) covering Amharic, Tigrinya, Tigre, Classical Ge'ez, and
English. Built as a continuation of
["MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages"](https://github.com/hailaykidu/MoVoC)
(Teklehaymanot, Fazlija, Nejdl — L3S Research Center).

## Usage

```python
import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
print(sp.encode("አልሰበሩም", out_type=str))
```

## Training data — provenance and licenses

Trained on a temperature-sampled (`alpha=0.3`) mix of cleaned text from these
sources. **Licenses are mixed and one local source is unspecified** — review
before treating this as uniformly open-licensed for downstream redistribution
of *text generated by decoding*, though the vocabulary/subword-piece list
itself does not reproduce the training text.

| Language | Primary sources | License(s) |
|---|---|---|
| Amharic | `castorini/afriberta-corpus`, `michsethowusu/amharic-sentiments-corpus`, `yordanoswuletaw/amharic-pretraining-corpus`, `rasyosef/amharic-sentences-corpus`, `cis-lmu/Glot500`, `cis-lmu/GlotCC-V1`, `HuggingFaceFW/finetranslations` | per HF dataset card / CC0-1.0 / ODC-By |
| Tigrinya | `fgaim/GLOCR-Tigrinya`, `fgaim/tigrinya-squad`, `mewaeltsegay/TigrinyaLargeText`, `SIMBA9657/haddas-tigrinya-corpus`, `cis-lmu/Glot500`/`GlotCC-V1`, `HuggingFaceFW/finetranslations`, **local file `~/Ser/train.txt` (1.98M lines)** | per HF dataset card / CC0-1.0 / ODC-By / **unspecified (local)** |
| Tigre | `BeitTigreAI/tigre-data-lexicon`, `BeitTigreAI/tigre-data-monolingual-text` (primary, 490K rows) | CC-BY-SA-4.0 |
| Ge'ez | `Bedru/Eng-Geez` (2,107 pairs — the only real corpus found anywhere; see Limitations) | per HF dataset card |
| English | `translated_text` field from `HuggingFaceFW/finetranslations` | ODC-By |

Full per-source breakdown: see `SOURCES.md` in the
[GeezTokenizer](https://github.com/hailaykidu) project this was built from.

## Limitations

- **Bilin (byn) has no dedicated training data** — confirmed absent from
  every broad-coverage source checked (HF Hub, GlotCC-V1, FineTranslations,
  biblenlp-corpus-mmteb, GitHub). `byte_fallback=true` lets this tokenizer
  still encode Bilin text losslessly, but with no dedicated subword coverage.
- **Ge'ez is extremely low-resource** — only 2,107 sentence pairs of real
  training text exist anywhere that could be found; its ~1,800 unique
  sentences were each repeated ~478x during temperature-sampled balancing to
  give it fair representation in the shared vocabulary. This is a documented,
  independently-reconfirmed data-scarcity constraint (matches the MoVoC
  paper's own reported limitation for the same two languages), not a search
  gap.
- **Tigre text sourced from CommonCrawl-derived corpora (GlotCC,
  FineTranslations) is filtered by language-ID confidence (≥0.5) but should
  still be treated as secondary** to the dedicated `BeitTigreAI` corpus —
  automatic language-ID is known to confuse Tigre with the much more common
  Tigrinya.

## Validation

0% UNK rate across all 5 languages; fertility 1.2–1.4 tokens/word. Full
report and methodology in the source project's `04_validation/report.md`.

## Installation

```bash
pip install tokenizers huggingface_hub
```

## Tokenizer configuration

| Property | Value |
|---|---|
| Algorithm | SentencePiece Unigram |
| Vocabulary size | 120,000 |
| Byte fallback | enabled |

Character coverage, sampling temperature, normalisation rules, and the
per-language corpus sizes used for temperature sampling are **Not
documented**.

## Citation

If you use this artifact, please cite:

```bibtex
@inproceedings{teklehaymanot2025movoc,
  title     = {MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages},
  author    = {Teklehaymanot, Hailay Kidu and Fazlija, Dren and Nejdl, Wolfgang},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year      = {2025},
  url       = {https://arxiv.org/abs/2509.08812}
}
```