| --- |
| license: mit |
| base_model: none |
| tags: |
| - arabizi |
| - transliteration |
| - arabic-nlp |
| - arabic-dialects |
| pretty_name: ArabiziKit trained layer |
| --- |
| |
| # ArabiziKit trained layer |
|
|
| The learned layer of the ArabiziKit hybrid transliteration system |
| (https://github.com/rb2625/arabizi-kit), which converts Arabizi โ Arabic |
| written in Latin letters and digits (2 for hamza/qaf, 3 for ayn, 7 for ha, |
| 5 for kha) โ into Arabic script. |
|
|
| Three small, dependency-free components, all pure Python: |
|
|
| | component | file | what it does | |
| | --- | --- | --- | |
| | word reading table | `model.json` | Arabizi word โ observed Arabic renderings with frequencies (1,155 entries), article-aware alignment (`el etnein` โ `ุงูุงุซููู`) | |
| | character trigram LM | `model.json` | Laplace-smoothed trigram over corpus references, reranking each word's candidates toward natural Arabic | |
| | Naive Bayes dialect classifier | `egyptian.json`, `levantine.json`, `maghrebi.json` | word tokens + Arabizi code markers; supplies the dialect hint automatically (only for dialects whose conventions change the engine's readings) | |
|
|
| ## Training |
|
|
| Trained on the calibration set plus the pipeline train/dev splits |
| (`rb2625/arabizi-kit-corpus`) and public dialect text. Held-out test and |
| external sets are excluded from training. |
|
|
| ## Usage |
|
|
| ```bash |
| pip install arabizikit |
| arabizikit model train # rebuild the model from the corpus |
| arabizikit "bach n9ra" --model # learned layer on one sentence |
| arabizikit eval --data test.json --model # reproduce the paper's numbers |
| ``` |
|
|
| ## Results (exact@1 / hit@3 / CER, from the paper) |
|
|
| | set | rules, no hint | rules, oracle hint | learned layer | |
| | --- | --- | --- | --- | |
| | pipeline dev | 0.021 / 0.021 / 0.291 | 0.021 / 0.021 / 0.291 | 0.354 / 0.542 / 0.097 | |
| | pipeline test | 0.000 / 0.000 / 0.299 | 0.000 / 0.000 / 0.299 | 0.061 / 0.102 / 0.226 | |
| | Egyptian (external) | 0.376 / 0.526 / 0.236 | 0.376 / 0.526 / 0.236 | 0.348 / 0.502 / 0.248 | |
| | Levantine (external) | 0.296 / 0.429 / 0.076 | 0.296 / 0.429 / 0.076 | 0.270 / 0.360 / 0.096 | |
| | Moroccan Darija (external) | 0.035 / 0.057 / 0.235 | 0.088 / 0.115 / 0.180 | 0.053 / 0.080 / 0.207 | |
|
|
| ## Paper |
|
|
| "ArabiziKit: An Open, Hybrid, Benchmark-Driven Arabizi to Arabic-Script |
| Transliteration System" (arXiv, cs.CL). |
|
|
