zewi_base: direct Runyankole ⇄ Acholi translation (pretrained base)

zewi_base is a 89M-parameter encoder-decoder Transformer that translates Runyankole → Acholi and Acholi → Runyankole directly, with one model and no English on either side. The direction is set by a prefix token on the source (<2ach> / <2nyn>; the tag names the target language).

This is a pretrained base, not a finished translator. It was trained only on machine-translated pairs: both the Runyankole side and the Acholi side of every training pair are output of one NLLB-based model (see Training). It has not been fine-tuned on human-translated data. Full test-set scores have not been run yet (see Evaluation).

Quick start

pip install torch sentencepiece safetensors huggingface_hub
import sys
from huggingface_hub import snapshot_download

folder = snapshot_download("amDANIEL2024/zewi_base")     # add token="hf_..." if the repo is still private
sys.path.insert(0, folder)
from zewi import Translator

t = Translator(folder)                                   # uses the GPU if there is one
acholi = t.translate("Okwombeka orugo kukatandika omu kwezi kw'okubanza 2000.", "nyn2ach")
print(acholi)
print(t.translate(acholi, "ach2nyn"))                    # back again

Directions are "nyn2ach" and "ach2nyn". translate takes a string or a list and accepts beam (default 4). This is a custom PyTorch model, not a transformers model, so AutoModel will not load it.

Example outputs

Not yet added. Examples will be taken unedited from the SALT test set, mistakes included, once the evaluation has been run.

Model

Architecture pre-LN Transformer encoder-decoder, 6 + 6 layers, d_model 640, 8 heads, FFN 2560, GELU
Parameters 89M (input embedding, decoder embedding and output projection are tied)
Tokenizer shared SentencePiece unigram, 32,000 pieces, case preserved, digits split (spm.model); trained on Runyankole and Acholi text only
Context 256 positions (sinusoidal); training pairs were limited to 128 pieces per side
Weights model.safetensors (float32)

Training

  • Data: the Runyankole–Acholi pairs of tulati, used in both directions. In tulati every non-English column is a machine translation of one English column, made by Sunbird/translate-nllb-1.3b-salt; no human-written Runyankole or Acholi is in it. The English source column was not used for this model. This card does not report how good that translator is for Runyankole or Acholi.
  • Filtering (counts from this run): of 1,534,699 tulati rows, 87,670 were dropped because they were flagged empty, degenerate, looks-like-English or length-outlier in either language (flagged in Runyankole: 35,277, in Acholi: 61,272). Then 1,805 pairs with identical sides (removed) and 574 exact duplicates were removed, leaving 1,444,650 pairs. Decontamination against SALT dev/test removed 2 pair(s), leaving 1,444,648; 2,000 of those were held out for validation. No further pair-level filter was used.
  • Tokenizer: trained on tulati plus the text of the SALT train split (Runyankole and Acholi only). The model weights were trained on tulati only.
  • Schedule: 5 epochs, 19,824 optimizer steps (about 25k target tokens per step), fp16 on one Kaggle T4, label smoothing 0.1, AdamW, linear warmup then inverse-sqrt decay (peak LR 3e-4).

Evaluation

Complete SALT dev/test scores (BLEU, chrF, chrF++): not run yet. SALT test has not been used for anything.

What has been measured so far, at the end of pretraining (step 19,824):

Set Direction Loss Quick chrF++ (greedy, first 128 sentences)
tulati validation (held-out machine-translated pairs) nyn→ach 1.405 52.62
tulati validation (held-out machine-translated pairs) ach→nyn 1.461 52.40
SALT dev (human translations) nyn→ach 3.547 34.99
SALT dev (human translations) ach→nyn 3.629 37.24

The quick chrF++ is a noisy check on a small subset with greedy decoding; it is not comparable to full-set scores. SALT dev was only observed here, not used to choose anything.

Copy baseline on the complete SALT dev set (output the source sentence unchanged), sacreBLEU 2.5.1:

Direction BLEU chrF chrF++
nyn→ach 0.21 14.88 12.03
ach→nyn 0.21 12.76 10.60

Note that SALT's Runyankole and Acholi sentences are independent human translations of the same English sentence, so a reference in one language is not a translation of the reference in the other.

Limitations

  • Both sides of every training pair are machine output, so the model inherits that translator's errors and style. Real human-written text (dialect, spelling, code-switching) may be handled worse than benchmark numbers suggest. The table above shows this: loss on held-out machine-translated pairs is far lower than loss on human-translated SALT dev.
  • Meaning can be dropped or changed, especially for numbers, names and rare words. Do not use it for medical, legal or other high-stakes content without human review.
  • Sentence-level model: inputs longer than about 128 pieces are truncated or degrade.
  • Outputs have not been reviewed by an Acholi reader.

Files

model.safetensors weights · config.json architecture and training summary · spm.model tokenizer · zewi.py standalone inference code · README.md this card.

License

Released under the MIT License (as are tulati, the translation model that generated the training data, and SALT).

Author

Amanyire Daniel (amDANIEL2024) · domusdeiug@gmail.com · https://domusdeiug.com

Downloads last month
22
Safetensors
Model size
89.4M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train amDANIEL2024/zewi_base