Hailay/entimt-en-tigrinya-mt

EnTiMT: English ↔ Tigrinya Machine Translation with a Transplanted MoVoC_Tok

This project builds a bidirectional English–Tigrinya translation model by taking a pretrained multilingual NLLB-200 checkpoint and transplanting its tokenizer for the MoVoC_Tok (a 120,000-vocabulary SentencePiece Unigram tokenizer shared across Amharic/Tigrinya/Tigre/Ge'ez/English, built earlier in this line of work) — then fully fine-tuning on a cleaned, deduplicated, multi-source parallel corpus. It extends the methodology of our own prior paper, Low-Resource English-Tigrinya MT: Leveraging Multilingual Models, Custom Tokenizers, and Clean Evaluation Benchmarks (Teklehaymanot, Gidey, Nejdl — LREC 2026), applying it to this newer, larger, 5-language-shared tokenizer instead of a narrower one.

Code: github.com/hailaykidu/EnTiMT

Pipeline

01_collection/       -> raw parallel data from every verified public source
02_cleaning/          -> normalize, filter, dedup (exact + MinHash near-dup), merge
03_tokenizer_integration/ -> transplant MoVoC_Tok into NLLB-200-distilled-600M
04_training/          -> bidirectional Seq2SeqTrainer fine-tune (SLURM)
05_evaluation/        -> BLEU / chrF / COMET on a held-out gold benchmark

1. Data collection

Every source below was verified by actually downloading it (OPUS API, GitHub clone, or reuse of an already-local repo from this session) -- see 01_collection/SOURCES.md in the code repo for the full table with license and exact pair counts. In short:

Source	Pairs	Quality
OPUS NLLB (mined)	1,398,173	noisy, web-mined
Travis Foundation Tigrinya-Parallel-Corpus	126,930	human-collected
OPUS CCAligned	7,658	noisy, web-mined
OPUS tico-19	6,142	professionally translated (COVID-19 domain)
HornMT	2,030	human-translated news, multi-way parallel
OPUS Tatoeba	74	human, tiny

FLORES-200 devtest (tir_Ethi/eng_Latn) is fetched separately and used only as the final held-out evaluation benchmark -- it is never mixed into the training pool.

2. Cleaning

clean_corpus.py normalizes (NFC, control-char/whitespace cleanup), strips mined-bitext artifacts found in the raw NLLB data (Bible verse numbers, [alt1//alt2] inline alternate-phrasing brackets), filters by length and length-ratio, keeps only Tigrinya-script-consistent pairs, then deduplicates (exact + MinHash near-duplicate over word 3-shingles, reusing the same approach as the MoVoC_Tok project's corpus cleaning).

One real bug caught and fixed during this step: Python's str.splitlines() splits on more than \n (including U+2028 LINE SEPARATOR, present 84 times in the raw NLLB English file and only 4 times in its Tigrinya counterpart) -- using it to read the two sides of a parallel file independently silently desynced their line-for-line alignment partway through the corpus. Switched to plain \n-splitting, which matches how the files are actually newline-delimited, and re-verified alignment by spot-checking matched line indices before proceeding.

3. Tokenizer transplant

transplant_tokenizer.py replaces facebook/nllb-200-distilled-600M's own ~256k-token tokenizer with MoVoC_Tok, and re-initializes the embedding matrix with a FOCUS-style informed initialization rather than random init:

1. Tokenize the cleaned training corpus with MoVoC_Tok itself, so the auxiliary embedding model sees the exact subword pieces this vocabulary produces.
2. Train a FastText skip-gram model directly on that tokenized text, at dim == 1024 (NLLB-200-distilled-600M's d_model) -- no projection needed since dimensions already match.
3. Use those vectors as the new embedding matrix, written in place via .data indexing after resize_token_embeddings(), which preserves tied encoder/decoder-input/LM-head weights.
4. The pretrained transformer body (attention/FFN weights) is kept as-is -- that is the actual transfer-learning payload; only the embedding layer changes.

Bidirectionality is handled with direction tags (>>tir<< / >>eng<<) prepended to the encoder input.

4. Training

train_mt.py builds a bidirectional training set (every pair trains both en->ti and ti->en, so 1,140,309 cleaned pairs become 2,280,618 training examples) and fully fine-tunes with Seq2SeqTrainer (no frozen layers -- see the tokenizer-transplant section above for why).

Training configuration:

Setting	Value
Base checkpoint	facebook/nllb-200-distilled-600M (tokenizer/embeddings replaced, body kept)
Train / dev examples	2,280,618 / 4,000 (bidirectional; dev held out from cleaning, distinct from FLORES-200)
Epochs	3
Per-device train / eval batch size	16 / 16
Gradient accumulation	4 (effective batch size 64)
Learning rate	3e-5
Max sequence length	128 tokens
Precision	bf16
Eval / save interval	every 2,000 steps
Model selection	load_best_model_at_end=True, metric_for_best_model="eval_loss"
Seed	42
Direction tags	>>tir<< (target=Tigrinya), >>eng<< (target=English), prepended to encoder input
Steps / wall time	106,905 steps / 13h21m
Final train loss (mean of all logged steps)	4.696

5. Evaluation

evaluate.py reports BLEU and chrF (via sacrebleu) and COMET (via unbabel-comet, Unbabel/wmt22-comet-da) on the held-out FLORES-200 devtest set (1,013 sentences, zero overlap with training data), in both directions, alongside qualitative before/after examples.

Results (real, FLORES-200 devtest, 1,013 sentences)

Direction	BLEU	chrF	COMET
en → ti	0.133	4.99	0.416
ti → en	0.899	15.91	0.374

These scores are poor, and this is reported honestly rather than omitted. Qualitative inspection of the FLORES-200 output shows severe repetition/degeneration under greedy decoding, especially on longer source sentences — e.g. a real en→ti output for a ~20-word input:

"ኣብ'ዚ እዋን'ዚ ኣብ'ዚ እዋን'ዚ ኣብ'ዚ እዋን'ዚ ኣብ'ዚ እዋን'ዚ ኣብ'ዚ እዋን'ዚ'ዚ'ዚ'ዚ ..."

and another where "ዶክተር" ("doctor") repeats over 100 times in place of an actual translation. This pattern — the model locking into a short repeated n-gram instead of producing a full translation — appears throughout the devtest set, not just on cherry-picked examples (see 05_evaluation/eval_report.json in the GitHub repo for unedited qualitative samples). Likely contributing factors, not yet isolated: greedy decoding with no repetition penalty, and/or insufficient training given the size of the newly-transplanted embedding matrix relative to available compute.

Do not use this checkpoint for anything beyond experimentation as-is. Candidate fixes not yet tried: beam search / repetition penalty at generation time, additional training epochs, or a smaller learning rate specifically for the embedding layer.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("Hailay/entimt-en-tigrinya-mt")
tokenizer = AutoTokenizer.from_pretrained("Hailay/entimt-en-tigrinya-mt")

text = ">>tir<< The weather is nice today."
inputs = tokenizer(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Given the repetition issue above, num_beams>1 and a repetition_penalty (e.g. no_repeat_ngram_size=3, repetition_penalty=1.3) are recommended over plain greedy decoding, though this hasn't been systematically re-evaluated.

Links

• Model: Hailay/entimt-en-tigrinya-mt
• Tokenizer: Hailay/geez-en-shared-tokenizer
• Code: github.com/hailaykidu/EnTiMT

Limitations

• The bulk of training volume comes from web-mined NLLB bitext, which is noisier than human-translated data even after cleaning -- quality is upper-bounded by that source's alignment accuracy.
• The FastText-based embedding initialization is a simplified, self-contained approximation of the FOCUS technique (trained on this project's own corpus rather than a large general-purpose auxiliary embedding space); it gives every subword piece a distributionally grounded starting point, not a semantically "solved" one.
• Tigrinya has real dialectal variation (Eritrean vs. Ethiopian) not explicitly modeled or balanced for in the training data.

Installation

pip install transformers torch sentencepiece

Citation

This artifact uses the MoVoC-Tok tokenizer introduced in Teklehaymanot et al. (2025). Please cite:

@inproceedings{teklehaymanot2025movoc,
  title     = {MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages},
  author    = {Teklehaymanot, Hailay Kidu and Fazlija, Dren and Nejdl, Wolfgang},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year      = {2025},
  url       = {https://arxiv.org/abs/2509.08812}
}