zewi_base: direct Runyankole ⇄ Acholi translation (pretrained base)
zewi_base is a 89M-parameter encoder-decoder Transformer that translates Runyankole → Acholi and
Acholi → Runyankole directly, with one model and no English on either side. The direction is set by a prefix token on the
source (<2ach> / <2nyn>; the tag names the target language).
This is a pretrained base, not a finished translator. It was trained only on machine-translated pairs: both the Runyankole side and the Acholi side of every training pair are output of one NLLB-based model (see Training). It has not been fine-tuned on human-translated data. Full test-set scores have not been run yet (see Evaluation).
Quick start
pip install torch sentencepiece safetensors huggingface_hub
import sys
from huggingface_hub import snapshot_download
folder = snapshot_download("amDANIEL2024/zewi_base") # add token="hf_..." if the repo is still private
sys.path.insert(0, folder)
from zewi import Translator
t = Translator(folder) # uses the GPU if there is one
acholi = t.translate("Okwombeka orugo kukatandika omu kwezi kw'okubanza 2000.", "nyn2ach")
print(acholi)
print(t.translate(acholi, "ach2nyn")) # back again
Directions are "nyn2ach" and "ach2nyn". translate takes a string or a list and accepts beam (default 4).
This is a custom PyTorch model, not a transformers model, so AutoModel will not load it.
Example outputs
Not yet added. Examples will be taken unedited from the SALT test set, mistakes included, once the evaluation has been run.
Model
| Architecture | pre-LN Transformer encoder-decoder, 6 + 6 layers, d_model 640, 8 heads, FFN 2560, GELU |
| Parameters | 89M (input embedding, decoder embedding and output projection are tied) |
| Tokenizer | shared SentencePiece unigram, 32,000 pieces, case preserved, digits split (spm.model); trained on Runyankole and Acholi text only |
| Context | 256 positions (sinusoidal); training pairs were limited to 128 pieces per side |
| Weights | model.safetensors (float32) |
Training
- Data: the Runyankole–Acholi pairs of tulati, used in both directions. In tulati every non-English column is a machine translation of one English column, made by Sunbird/translate-nllb-1.3b-salt; no human-written Runyankole or Acholi is in it. The English source column was not used for this model. This card does not report how good that translator is for Runyankole or Acholi.
- Filtering (counts from this run): of 1,534,699 tulati rows, 87,670 were dropped because they were flagged empty, degenerate, looks-like-English or length-outlier in either language (flagged in Runyankole: 35,277, in Acholi: 61,272). Then 1,805 pairs with identical sides (removed) and 574 exact duplicates were removed, leaving 1,444,650 pairs. Decontamination against SALT dev/test removed 2 pair(s), leaving 1,444,648; 2,000 of those were held out for validation. No further pair-level filter was used.
- Tokenizer: trained on tulati plus the text of the SALT train split (Runyankole and Acholi only). The model weights were trained on tulati only.
- Schedule: 5 epochs, 19,824 optimizer steps (about 25k target tokens per step), fp16 on one Kaggle T4, label smoothing 0.1, AdamW, linear warmup then inverse-sqrt decay (peak LR 3e-4).
Evaluation
Complete SALT dev/test scores (BLEU, chrF, chrF++): not run yet. SALT test has not been used for anything.
What has been measured so far, at the end of pretraining (step 19,824):
| Set | Direction | Loss | Quick chrF++ (greedy, first 128 sentences) |
|---|---|---|---|
| tulati validation (held-out machine-translated pairs) | nyn→ach | 1.405 | 52.62 |
| tulati validation (held-out machine-translated pairs) | ach→nyn | 1.461 | 52.40 |
| SALT dev (human translations) | nyn→ach | 3.547 | 34.99 |
| SALT dev (human translations) | ach→nyn | 3.629 | 37.24 |
The quick chrF++ is a noisy check on a small subset with greedy decoding; it is not comparable to full-set scores. SALT dev was only observed here, not used to choose anything.
Copy baseline on the complete SALT dev set (output the source sentence unchanged), sacreBLEU 2.5.1:
| Direction | BLEU | chrF | chrF++ |
|---|---|---|---|
| nyn→ach | 0.21 | 14.88 | 12.03 |
| ach→nyn | 0.21 | 12.76 | 10.60 |
Note that SALT's Runyankole and Acholi sentences are independent human translations of the same English sentence, so a reference in one language is not a translation of the reference in the other.
Limitations
- Both sides of every training pair are machine output, so the model inherits that translator's errors and style. Real human-written text (dialect, spelling, code-switching) may be handled worse than benchmark numbers suggest. The table above shows this: loss on held-out machine-translated pairs is far lower than loss on human-translated SALT dev.
- Meaning can be dropped or changed, especially for numbers, names and rare words. Do not use it for medical, legal or other high-stakes content without human review.
- Sentence-level model: inputs longer than about 128 pieces are truncated or degrade.
- Outputs have not been reviewed by an Acholi reader.
Files
model.safetensors weights · config.json architecture and training summary · spm.model tokenizer ·
zewi.py standalone inference code · README.md this card.
License
Released under the MIT License (as are tulati, the translation model that generated the training data, and SALT).
Author
Amanyire Daniel (amDANIEL2024) · domusdeiug@gmail.com · https://domusdeiug.com
- Downloads last month
- 22