rabeeeehh/arabizi-kit-model

ArabiziKit trained layer

The learned layer of the ArabiziKit hybrid transliteration system (https://github.com/rb2625/arabizi-kit), which converts Arabizi — Arabic written in Latin letters and digits (2 for hamza/qaf, 3 for ayn, 7 for ha, 5 for kha) — into Arabic script.

Three small, dependency-free components, all pure Python:

component	file	what it does
word reading table	model.json	Arabizi word → observed Arabic renderings with frequencies (1,155 entries), article-aware alignment (el etnein → الاثنين)
character trigram LM	model.json	Laplace-smoothed trigram over corpus references, reranking each word's candidates toward natural Arabic
Naive Bayes dialect classifier	egyptian.json, levantine.json, maghrebi.json	word tokens + Arabizi code markers; supplies the dialect hint automatically (only for dialects whose conventions change the engine's readings)

Training

Trained on the calibration set plus the pipeline train/dev splits (rb2625/arabizi-kit-corpus) and public dialect text. Held-out test and external sets are excluded from training.

Usage

pip install arabizikit
arabizikit model train     # rebuild the model from the corpus
arabizikit "bach n9ra" --model              # learned layer on one sentence
arabizikit eval --data test.json --model    # reproduce the paper's numbers

Results (exact@1 / hit@3 / CER, from the paper)

set	rules, no hint	rules, oracle hint	learned layer
pipeline dev	0.021 / 0.021 / 0.291	0.021 / 0.021 / 0.291	0.354 / 0.542 / 0.097
pipeline test	0.000 / 0.000 / 0.299	0.000 / 0.000 / 0.299	0.061 / 0.102 / 0.226
Egyptian (external)	0.376 / 0.526 / 0.236	0.376 / 0.526 / 0.236	0.348 / 0.502 / 0.248
Levantine (external)	0.296 / 0.429 / 0.076	0.296 / 0.429 / 0.076	0.270 / 0.360 / 0.096
Moroccan Darija (external)	0.035 / 0.057 / 0.235	0.088 / 0.115 / 0.180	0.053 / 0.080 / 0.207

Paper

"ArabiziKit: An Open, Hybrid, Benchmark-Driven Arabizi to Arabic-Script Transliteration System" (arXiv, cs.CL).