README.md

---
language:
- ti
license: apache-2.0
library_name: transformers
pipeline_tag: question-answering
tags:
- question-answering
- extractive-qa
- xlm-roberta
- vexmlm
- geez
- low-resource
base_model: Hailay/VEXMLM
---

# VEXMLM — TIGQA

Tigrinya **question answering** fine-tuned from
[`Hailay/VEXMLM`](https://huggingface.co/Hailay/VEXMLM), the vocabulary-extended
XLM-R for Ge'ez-script languages.

Official implementation: **https://github.com/hailaykidu/VEXMLM**

| | |
|---|---|
| Task | question-answering |
| Dataset | TIGQA |
| Language | Tigrinya |
| Architecture | `XLMRobertaForQuestionAnswering` |
| Base model | `Hailay/VEXMLM` |
| Vocabulary | 280,002 |
| Seeds published | 42, 43, 44, 45, 46 |

**This checkpoint performs poorly in absolute terms.** Exact Match of
2.39 is near zero. TIGQA's test split contains only 67 questions, so the metric is
both very hard and very high-variance. It is published for completeness and
reproducibility of the paper's evaluation, **not** as a usable Tigrinya QA system.
For Tigrinya extractive QA, see [`Hailay/VEXMLM-TiQuAD`](https://huggingface.co/Hailay/VEXMLM-TiQuAD).

## Five-seed benchmark evaluation

Fine-tuned independently under seeds 42–46 with one configuration (hash
`ce27cc194946`) on an A100-PCIE-40GB. Reported as mean ± standard deviation over
the five runs, on the dataset's **test** split.

| Metric | Score |
|---|---|
| Exact Match | **2.39 ± 0.82** |
| F1 | **9.76 ± 0.97** |

These are the paper's verified results. They come from the five-seed evaluation
described above — **not** from interactive use.

### Interactive inference vs. benchmark

**Benchmark evaluation** is the five-seed measurement on the held-out test split,
shown in the table above.

**Interactive inference** is what the usage example below performs: Supply a Tigrinya context and question; the model returns an extracted span.
Predictions on arbitrary user input are demonstrations only and do not produce or
reproduce the benchmark score.

## Repository layout

Five independently fine-tuned checkpoints, one per seed. The reported benchmark
score is the mean ± standard deviation over all five; **no single seed is the
"five-seed model."**

```
seed-42/  seed-43/  seed-44/  seed-45/  seed-46/
```

Load a specific seed with the `subfolder` argument, as in the example below.

## Fine-tuning

Fine-tuned from [`Hailay/VEXMLM`](https://huggingface.co/Hailay/VEXMLM), a
vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the
SentencePiece model) after continued MLM pretraining.

| Hyperparameter | Value |
|---|---|
| Max sequence length | 256 |
| Batch size | 32 |
| Epochs | 4 |
| Learning rate | 2e-5 |
| LR schedule | Linear decay, 10% warmup |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Optimizer | AdamW (β₁ 0.9, β₂ 0.999, ε 1e-8) |
| Precision | bf16 |
| Trainable parameters | All |
| Hardware | 1× NVIDIA A100 |

Runs are bit-reproducible: `enable_full_determinism`,
`CUBLAS_WORKSPACE_CONFIG=:4096:8`, `dataloader_num_workers=0`.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

repo = "Hailay/VEXMLM-TIGQA"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="seed-42")
model = AutoModelForQuestionAnswering.from_pretrained(repo, subfolder="seed-42")
model.eval()

question = "ኤርትራ ኣብ ኣየናይ ክፍለ ዓለም ትርከብ?"
context  = "ኤርትራ ኣብ አፍሪቃ ክፍለ ዓለም እትርከብ ሃገር እያ።"
enc = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    out = model(**enc)

start = out.start_logits.argmax()
end = out.end_logits.argmax()
print(tokenizer.decode(enc.input_ids[0][start:end + 1], skip_special_tokens=True))
```

## Limitations

- Fine-tuned for Tigrinya on TIGQA only; performance on other
  languages, domains or label schemes is not characterised.
- The base model covers Amharic and Tigrinya; other Ge'ez-script languages were
  not part of pretraining.
- Corpora are drawn largely from religious and news domains, and the model may
  reflect those distributions and any biases in them.
- Single-configuration study: no hyperparameter search was performed, and
  baseline comparisons in the paper are single-seed.

## Reproducibility

The fine-tuning launcher, evaluation code and per-run result records are in the
official repository: **https://github.com/hailaykidu/VEXMLM**

```bash
sbatch scripts/slurm_stage2_spm_seeds.sh    # 6 tasks × 5 seeds
python3 evaluation/export_spm_results.py    # regenerates the metrics table
```

## Citation

```bibtex
@inproceedings{teklehaymanot2026vexmlm,
  title     = {Expanding the Lexicon of Ge'ez Based African Languages:
               A Comparative Study of Amharic and Tigrinya},
  author    = {Teklehaymanot, Hailay Kidu and Yadeta, Gebregziabihier and
               Nejdl, Wolfgang},
  booktitle = {Proceedings of the Workshop on Language Models for
               Underserved Communities (LM4UC) at IJCAI},
  year      = {2026}
}
```

Accepted at the LM4UC Workshop, IJCAI 2026.

## License

Apache 2.0, following `xlm-roberta-base`.