Hailay/VEXMLM-TiQuAD

VEXMLM — TiQuAD

Tigrinya question answering fine-tuned from Hailay/VEXMLM, the vocabulary-extended XLM-R for Ge'ez-script languages.

Official implementation: https://github.com/hailaykidu/VEXMLM

Task	question-answering
Dataset	TiQuAD
Language	Tigrinya
Architecture	XLMRobertaForQuestionAnswering
Base model	Hailay/VEXMLM
Vocabulary	280,002
Seeds published	42, 43, 44, 45, 46

TiQuAD is a supplementary task in the VEXMLM paper — a diagnostic evaluation, not a reported paper benchmark. Its 926 test questions make it a more stable measurement than TIGQA's 67, and it is the stronger Tigrinya QA checkpoint of the two.

Five-seed benchmark evaluation

Fine-tuned independently under seeds 42–46 with one configuration (hash ce27cc194946) on an A100-PCIE-40GB. Reported as mean ± standard deviation over the five runs, on the dataset's test split.

Metric	Score
Exact Match	50.24 ± 0.48
F1	58.90 ± 0.66

These are the paper's verified results. They come from the five-seed evaluation described above — not from interactive use.

Interactive inference vs. benchmark

Benchmark evaluation is the five-seed measurement on the held-out test split, shown in the table above.

Interactive inference is what the usage example below performs: Supply a Tigrinya context and question; the model returns an extracted span. Predictions on arbitrary user input are demonstrations only and do not produce or reproduce the benchmark score.

Repository layout

Five independently fine-tuned checkpoints, one per seed. The reported benchmark score is the mean ± standard deviation over all five; no single seed is the "five-seed model."

seed-42/  seed-43/  seed-44/  seed-45/  seed-46/

Load a specific seed with the subfolder argument, as in the example below.

Fine-tuning

Fine-tuned from Hailay/VEXMLM, a vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the SentencePiece model) after continued MLM pretraining.

Hyperparameter	Value
Max sequence length	256
Batch size	32
Epochs	4
Learning rate	2e-5
LR schedule	Linear decay, 10% warmup
Weight decay	0.01
Gradient clipping	1.0
Optimizer	AdamW (β₁ 0.9, β₂ 0.999, ε 1e-8)
Precision	bf16
Trainable parameters	All
Hardware	1× NVIDIA A100

Runs are bit-reproducible: enable_full_determinism, CUBLAS_WORKSPACE_CONFIG=:4096:8, dataloader_num_workers=0.

Usage

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

repo = "Hailay/VEXMLM-TiQuAD"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="seed-42")
model = AutoModelForQuestionAnswering.from_pretrained(repo, subfolder="seed-42")
model.eval()

question = "ኤርትራ ኣብ ኣየናይ ክፍለ ዓለም ትርከብ?"
context  = "ኤርትራ ኣብ አፍሪቃ ክፍለ ዓለም እትርከብ ሃገር እያ።"
enc = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    out = model(**enc)

start = out.start_logits.argmax()
end = out.end_logits.argmax()
print(tokenizer.decode(enc.input_ids[0][start:end + 1], skip_special_tokens=True))

Limitations

• Fine-tuned for Tigrinya on TiQuAD only; performance on other languages, domains or label schemes is not characterised.
• The base model covers Amharic and Tigrinya; other Ge'ez-script languages were not part of pretraining.
• Corpora are drawn largely from religious and news domains, and the model may reflect those distributions and any biases in them.
• Single-configuration study: no hyperparameter search was performed, and baseline comparisons in the paper are single-seed.

Reproducibility

The fine-tuning launcher, evaluation code and per-run result records are in the official repository: https://github.com/hailaykidu/VEXMLM

sbatch scripts/slurm_stage2_spm_seeds.sh    # 6 tasks × 5 seeds
python3 evaluation/export_spm_results.py    # regenerates the metrics table

Citation

@inproceedings{teklehaymanot2026vexmlm,
  title     = {Expanding the Lexicon of Ge'ez Based African Languages:
               A Comparative Study of Amharic and Tigrinya},
  author    = {Teklehaymanot, Hailay Kidu and Yadeta, Gebregziabihier and
               Nejdl, Wolfgang},
  booktitle = {Proceedings of the Workshop on Language Models for
               Underserved Communities (LM4UC) at IJCAI},
  year      = {2026}
}

Accepted at the LM4UC Workshop, IJCAI 2026.

License

Apache 2.0, following xlm-roberta-base.