metadata
license: mit
base_model: none
tags:
- arabizi
- transliteration
- arabic-nlp
- arabic-dialects
pretty_name: ArabiziKit trained layer
ArabiziKit trained layer
The learned layer of the ArabiziKit hybrid transliteration system (https://github.com/rb2625/arabizi-kit), which converts Arabizi — Arabic written in Latin letters and digits (2 for hamza/qaf, 3 for ayn, 7 for ha, 5 for kha) — into Arabic script.
Three small, dependency-free components, all pure Python:
| component | file | what it does |
|---|---|---|
| word reading table | model.json |
Arabizi word → observed Arabic renderings with frequencies (1,155 entries), article-aware alignment (el etnein → الاثنين) |
| character trigram LM | model.json |
Laplace-smoothed trigram over corpus references, reranking each word's candidates toward natural Arabic |
| Naive Bayes dialect classifier | egyptian.json, levantine.json, maghrebi.json |
word tokens + Arabizi code markers; supplies the dialect hint automatically (only for dialects whose conventions change the engine's readings) |
Training
Trained on the calibration set plus the pipeline train/dev splits
(rb2625/arabizi-kit-corpus) and public dialect text. Held-out test and
external sets are excluded from training.
Usage
pip install arabizikit
arabizikit model train # rebuild the model from the corpus
arabizikit "bach n9ra" --model # learned layer on one sentence
arabizikit eval --data test.json --model # reproduce the paper's numbers
Results (exact@1 / hit@3 / CER, from the paper)
| set | rules, no hint | rules, oracle hint | learned layer |
|---|---|---|---|
| pipeline dev | 0.021 / 0.021 / 0.291 | 0.021 / 0.021 / 0.291 | 0.354 / 0.542 / 0.097 |
| pipeline test | 0.000 / 0.000 / 0.299 | 0.000 / 0.000 / 0.299 | 0.061 / 0.102 / 0.226 |
| Egyptian (external) | 0.376 / 0.526 / 0.236 | 0.376 / 0.526 / 0.236 | 0.348 / 0.502 / 0.248 |
| Levantine (external) | 0.296 / 0.429 / 0.076 | 0.296 / 0.429 / 0.076 | 0.270 / 0.360 / 0.096 |
| Moroccan Darija (external) | 0.035 / 0.057 / 0.235 | 0.088 / 0.115 / 0.180 | 0.053 / 0.080 / 0.207 |
Paper
"ArabiziKit: An Open, Hybrid, Benchmark-Driven Arabizi to Arabic-Script Transliteration System" (arXiv, cs.CL).