language:
- ti
license: apache-2.0
library_name: transformers
pipeline_tag: question-answering
tags:
- question-answering
- extractive-qa
- xlm-roberta
- vexmlm
- geez
- low-resource
base_model: Hailay/VEXMLM
VEXMLM β TIGQA
Tigrinya question answering fine-tuned from
Hailay/VEXMLM, the vocabulary-extended
XLM-R for Ge'ez-script languages.
Official implementation: https://github.com/hailaykidu/VEXMLM
| Task | question-answering |
| Dataset | TIGQA |
| Language | Tigrinya |
| Architecture | XLMRobertaForQuestionAnswering |
| Base model | Hailay/VEXMLM |
| Vocabulary | 280,002 |
| Seeds published | 42, 43, 44, 45, 46 |
This checkpoint performs poorly in absolute terms. Exact Match of
2.39 is near zero. TIGQA's test split contains only 67 questions, so the metric is
both very hard and very high-variance. It is published for completeness and
reproducibility of the paper's evaluation, not as a usable Tigrinya QA system.
For Tigrinya extractive QA, see Hailay/VEXMLM-TiQuAD.
Five-seed benchmark evaluation
Fine-tuned independently under seeds 42β46 with one configuration (hash
ce27cc194946) on an A100-PCIE-40GB. Reported as mean Β± standard deviation over
the five runs, on the dataset's test split.
| Metric | Score |
|---|---|
| Exact Match | 2.39 Β± 0.82 |
| F1 | 9.76 Β± 0.97 |
These are the paper's verified results. They come from the five-seed evaluation described above β not from interactive use.
Interactive inference vs. benchmark
Benchmark evaluation is the five-seed measurement on the held-out test split, shown in the table above.
Interactive inference is what the usage example below performs: Supply a Tigrinya context and question; the model returns an extracted span. Predictions on arbitrary user input are demonstrations only and do not produce or reproduce the benchmark score.
Repository layout
Five independently fine-tuned checkpoints, one per seed. The reported benchmark score is the mean Β± standard deviation over all five; no single seed is the "five-seed model."
seed-42/ seed-43/ seed-44/ seed-45/ seed-46/
Load a specific seed with the subfolder argument, as in the example below.
Fine-tuning
Fine-tuned from Hailay/VEXMLM, a
vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the
SentencePiece model) after continued MLM pretraining.
| Hyperparameter | Value |
|---|---|
| Max sequence length | 256 |
| Batch size | 32 |
| Epochs | 4 |
| Learning rate | 2e-5 |
| LR schedule | Linear decay, 10% warmup |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Optimizer | AdamW (Ξ²β 0.9, Ξ²β 0.999, Ξ΅ 1e-8) |
| Precision | bf16 |
| Trainable parameters | All |
| Hardware | 1Γ NVIDIA A100 |
Runs are bit-reproducible: enable_full_determinism,
CUBLAS_WORKSPACE_CONFIG=:4096:8, dataloader_num_workers=0.
Usage
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch
repo = "Hailay/VEXMLM-TIGQA"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="seed-42")
model = AutoModelForQuestionAnswering.from_pretrained(repo, subfolder="seed-42")
model.eval()
question = "α€αα΅α« α£α₯ α£α¨αα ααα ααα α΅αα¨α₯?"
context = "α€αα΅α« α£α₯ α ααͺα ααα ααα α₯α΅αα¨α₯ ααα α₯α«α’"
enc = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
out = model(**enc)
start = out.start_logits.argmax()
end = out.end_logits.argmax()
print(tokenizer.decode(enc.input_ids[0][start:end + 1], skip_special_tokens=True))
Limitations
- Fine-tuned for Tigrinya on TIGQA only; performance on other languages, domains or label schemes is not characterised.
- The base model covers Amharic and Tigrinya; other Ge'ez-script languages were not part of pretraining.
- Corpora are drawn largely from religious and news domains, and the model may reflect those distributions and any biases in them.
- Single-configuration study: no hyperparameter search was performed, and baseline comparisons in the paper are single-seed.
Reproducibility
The fine-tuning launcher, evaluation code and per-run result records are in the official repository: https://github.com/hailaykidu/VEXMLM
sbatch scripts/slurm_stage2_spm_seeds.sh # 6 tasks Γ 5 seeds
python3 evaluation/export_spm_results.py # regenerates the metrics table
Citation
@inproceedings{teklehaymanot2026vexmlm,
title = {Expanding the Lexicon of Ge'ez Based African Languages:
A Comparative Study of Amharic and Tigrinya},
author = {Teklehaymanot, Hailay Kidu and Yadeta, Gebregziabihier and
Nejdl, Wolfgang},
booktitle = {Proceedings of the Workshop on Language Models for
Underserved Communities (LM4UC) at IJCAI},
year = {2026}
}
Accepted at the LM4UC Workshop, IJCAI 2026.
License
Apache 2.0, following xlm-roberta-base.