atella_finetuned: direct Runyankole ⇄ Ateso translation

atella_finetuned is a 89M-parameter encoder-decoder Transformer that translates Runyankole → Ateso and Ateso → Runyankole directly, with one model and no English on either side. The direction is set by a prefix token on the source (<2teo> / <2nyn>; the tag names the target language).

It is atella_base (pretrained only on machine-translated pairs) further trained on the human-translated Runyankole–Ateso pairs of the SALT training set. Headline scores on the complete SALT test set are in Evaluation.

Quick start

pip install torch sentencepiece safetensors huggingface_hub
import sys
from huggingface_hub import snapshot_download

folder = snapshot_download("amDANIEL2024/atella_finetuned")   # add token="hf_..." if the repo is still private
sys.path.insert(0, folder)
from atella import Translator

t = Translator(folder)                                        # uses the GPU if there is one
ateso = t.translate("Okwombeka orugo kukatandika omu kwezi kw'okubanza 2000.", "nyn2teo")
print(ateso)
print(t.translate(ateso, "teo2nyn"))                        # back again

Directions are "nyn2teo" and "teo2nyn". translate takes a string or a list and accepts beam (default 4). This is a custom PyTorch model, not a transformers model, so AutoModel will not load it.

Evaluation

Complete SALT sets (500 sentences each), beam 4, sacreBLEU 2.5.1. Test is the headline. SALT dev was used to stop training and choose the checkpoint, so dev scores are slightly optimistic.

Split Direction System n BLEU chrF chrF++
test nyn→teo model 500 15.47 47.23 44.01
test nyn→teo copy baseline (source unchanged) 500 0.41 15.79 13.28
test teo→nyn model 500 10.19 45.46 40.60
test teo→nyn copy baseline (source unchanged) 500 0.40 14.87 12.69
dev nyn→teo model 500 16.27 48.33 45.11
dev nyn→teo copy baseline (source unchanged) 500 0.23 15.99 13.54
dev teo→nyn model 500 10.08 46.60 41.85
dev teo→nyn copy baseline (source unchanged) 500 0.22 15.13 12.99

Signatures: BLEU nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.5.1; chrF nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1; chrF++ nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.5.1.

The copy baseline outputs the source sentence unchanged and scores it against the reference. An English-pivot baseline was not run. Differences of 1 to 2 BLEU on a few hundred sentences are within noise.

Example outputs

The first 5 sentences of each SALT test file, not selected and not edited. The reference is the human translation. Mistakes are included; read the model output critically.

nyn → teo

Source Human reference Model output
Eby'eby'afaashoni biriyo nibigaruza akatwe. Ageutu ikampunin luka enape apolo bobo. Euta erionget loka ikapun alemun ikapun idis.
Omuzibwe n'endwara erikukwata abantu baingi omu kyanga. Erai adeka na ikamuni itunga luipu kotoma atutubet. Erai adeka na esuronokini itunga luipu katutubet.
Obwoki burungi, nibwija kuruga omukyanga eki. Esike lo ejok ebuni kosiep kalo. Esike loajokan nes ebuni alomun kaiboisit kana.
Akaba akwise naavuga egaaro obwo nyowe ndikwiruka. Auta ngesi aicak egali kauta eong akerit. Auta ngesi airenge erot nautar eong akerit.
Aby'obushoboorozi nibahhiga bundi nuuma abastigazi enkwata-buta abaayeyereire akatare k'omu nju Eutasi ipolisin aimo itelepai iarei luabilete ka akoko eiduka lokapolon. Isinyikokinitos apedorosio ace itunga luitolosete ibiasaran ibiasaranakec.

teo → nyn

Source Human reference Model output
Ageutu ikampunin luka enape apolo bobo. Eby'eby'afaashoni biriyo nibigaruza akatwe. Kampuni z'omupiira zitandikire kugaruka kutunguuka.
Erai adeka na ikamuni itunga luipu kotoma atutubet. Omuzibwe n'endwara erikukwata abantu baingi omu kyanga. N'endwara erikukwata abantu baingi omu kyanga.
Esike lo ejok ebuni kosiep kalo. Obwoki burungi, nibwija kuruga omukyanga eki. Esike nungi neeza aha rubaju oru.
Auta ngesi aicak egali kauta eong akerit. Akaba akwise naavuga egaaro obwo nyowe ndikwiruka. Akaba naavuga egaari enyima yaniruka.
Eutasi ipolisin aimo itelepai iarei luabilete ka akoko eiduka lokapolon. Aby'obushoboorozi nibahhiga bundi nuuma abastigazi enkwata-buta abaayeyereire akatare k'omu nju Pooriisi eriyo neronda teleki ibiri z'okwiba duuka mpango.

Model

Architecture pre-LN Transformer encoder-decoder, 6 + 6 layers, d_model 640, 8 heads, FFN 2560, GELU
Parameters 89M (input embedding, decoder embedding and output projection are tied)
Tokenizer shared SentencePiece unigram, 32,000 pieces, case preserved, digits split (spm.model); trained on Runyankole and Ateso text only
Context 256 positions (sinusoidal); training pairs were limited to 128 pieces per side
Weights model.safetensors (float32)

Training

  1. Pretraining (atella_base): 5 epochs, 17,426 optimizer steps on the Runyankole–Ateso pairs of tulati, both directions. In tulati every non-English column is a machine translation of one English column, made by Sunbird/translate-nllb-1.3b-salt; no human-written text is in it, so both sides of every pretraining pair are machine output. The English column was not used. Of 1,534,699 tulati rows, 127,539 (8.3%) were dropped because they were flagged empty, degenerate, looks-like-English or length-outlier in either language (flagged in Runyankole: 35,277; in Ateso: 104,732), then 1,015 identical-side pairs and 506 exact duplicates were removed, and 3 pair(s) that matched a SALT dev/test sentence; 1,405,636 pairs remained. See the atella_base card for details.
  2. Fine-tuning: the SALT training pairs (human translations) that have both languages, both directions, peak LR 6e-5, SALT dev loss checked every 100 steps, early stopping after 5 checks without improvement. The checkpoint with the lowest dev loss was kept (step 2,200). SALT test was not used in training or in choosing the checkpoint.

Limitations

  • The pretraining pairs are machine output on both sides, so the base model inherits that translator's errors and style. The translator's output was flagged (empty, degenerate, looks-like-English or length-outlier) for 2.3% of tulati rows in Runyankole and 6.8% in Ateso. These rates describe the generated data, not this model. This card does not report how good the translator is for Runyankole or Ateso.
  • Scores differ by direction: Ateso → Runyankole scores lower than Runyankole → Ateso on both dev and test (see Evaluation). The reason has not been established.
  • SALT's Runyankole and Ateso sentences are independent human translations of the same English sentence, so a reference in one language is not a translation of the reference in the other. Differences of 1 to 2 BLEU on 500 sentences are within noise.
  • Real human-written text beyond SALT (dialect, spelling, code-switching) may be handled worse than the benchmark suggests.
  • Meaning can be dropped or changed, especially for numbers, names and rare words. Do not use it for medical, legal or other high-stakes content without human review.
  • Sentence-level model: inputs longer than about 128 pieces are truncated or degrade.
  • Outputs have not been reviewed by a reader of Runyankole or Ateso.

Files

model.safetensors weights · config.json architecture and training summary · spm.model tokenizer · atella.py standalone inference code · README.md this card.

License

Released under the MIT License (as are tulati, the translation model that generated the training data, and SALT).

Author

Amanyire Daniel (amDANIEL2024) · domusdeiug@gmail.com · https://domusdeiug.com

Downloads last month
26
Safetensors
Model size
89.4M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for amDANIEL2024/atella_finetuned

Finetuned
(1)
this model

Dataset used to train amDANIEL2024/atella_finetuned