Add VEXMLM-AmQA: five verified fine-tuned checkpoints (seeds 42-46)

23579c2 verified 1 day ago

4.84 kB

language:
  - am
license: apache-2.0
library_name: transformers
pipeline_tag: question-answering
tags:
  - question-answering
  - extractive-qa
  - xlm-roberta
  - vexmlm
  - geez
  - low-resource
base_model: Hailay/VEXMLM

VEXMLM — AmQA

Amharic question answering fine-tuned from Hailay/VEXMLM, the vocabulary-extended XLM-R for Ge'ez-script languages.

Official implementation: https://github.com/hailaykidu/VEXMLM


Task	question-answering
Dataset	AmQA
Language	Amharic
Architecture	`XLMRobertaForQuestionAnswering`
Base model	`Hailay/VEXMLM`
Vocabulary	280,002
Seeds published	42, 43, 44, 45, 46

Extractive QA: the answer is always a span copied from the supplied context.

Five-seed benchmark evaluation

Fine-tuned independently under seeds 42–46 with one configuration (hash ce27cc194946) on an A100-PCIE-40GB. Reported as mean ± standard deviation over the five runs, on the dataset's test split.

Metric	Score
Exact Match	32.57 ± 0.77
F1	48.85 ± 0.96

These are the paper's verified results. They come from the five-seed evaluation described above — not from interactive use.

Interactive inference vs. benchmark

Benchmark evaluation is the five-seed measurement on the held-out test split, shown in the table above.

Interactive inference is what the usage example below performs: Supply an Amharic context and question; the model returns an extracted span. Predictions on arbitrary user input are demonstrations only and do not produce or reproduce the benchmark score.

Repository layout

Five independently fine-tuned checkpoints, one per seed. The reported benchmark score is the mean ± standard deviation over all five; no single seed is the "five-seed model."

seed-42/  seed-43/  seed-44/  seed-45/  seed-46/

Load a specific seed with the subfolder argument, as in the example below.

Fine-tuning

Fine-tuned from Hailay/VEXMLM, a vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the SentencePiece model) after continued MLM pretraining.

Hyperparameter	Value
Max sequence length	256
Batch size	32
Epochs	4
Learning rate	2e-5
LR schedule	Linear decay, 10% warmup
Weight decay	0.01
Gradient clipping	1.0
Optimizer	AdamW (β₁ 0.9, β₂ 0.999, ε 1e-8)
Precision	bf16
Trainable parameters	All
Hardware	1× NVIDIA A100

Runs are bit-reproducible: enable_full_determinism, CUBLAS_WORKSPACE_CONFIG=:4096:8, dataloader_num_workers=0.

Usage

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

repo = "Hailay/VEXMLM-AmQA"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="seed-42")
model = AutoModelForQuestionAnswering.from_pretrained(repo, subfolder="seed-42")
model.eval()

question = "ኢትዮጵያ የምትገኘው በየትኛው አህጉር ነው?"
context  = "ኢትዮጵያ በአፍሪካ አህጉር የምትገኝ ሀገር ናት።"
enc = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    out = model(**enc)

start = out.start_logits.argmax()
end = out.end_logits.argmax()
print(tokenizer.decode(enc.input_ids[0][start:end + 1], skip_special_tokens=True))

Limitations

Fine-tuned for Amharic on AmQA only; performance on other languages, domains or label schemes is not characterised.
The base model covers Amharic and Tigrinya; other Ge'ez-script languages were not part of pretraining.
Corpora are drawn largely from religious and news domains, and the model may reflect those distributions and any biases in them.
Single-configuration study: no hyperparameter search was performed, and baseline comparisons in the paper are single-seed.

Reproducibility

The fine-tuning launcher, evaluation code and per-run result records are in the official repository: https://github.com/hailaykidu/VEXMLM

sbatch scripts/slurm_stage2_spm_seeds.sh    # 6 tasks × 5 seeds
python3 evaluation/export_spm_results.py    # regenerates the metrics table

Citation

@inproceedings{teklehaymanot2026vexmlm,
  title     = {Expanding the Lexicon of Ge'ez Based African Languages:
               A Comparative Study of Amharic and Tigrinya},
  author    = {Teklehaymanot, Hailay Kidu and Yadeta, Gebregziabihier and
               Nejdl, Wolfgang},
  booktitle = {Proceedings of the Workshop on Language Models for
               Underserved Communities (LM4UC) at IJCAI},
  year      = {2026}
}

Accepted at the LM4UC Workshop, IJCAI 2026.

License

Apache 2.0, following xlm-roberta-base.