Blame for README.md

---

language: ar
license: apache-2.0
tags:
  - arabic
  - ner
  - nested-ner
  - wojood
  - konooz
  - knowledgegrapheval-2026
  - arabicnlp-2026
  - adaptner
  - experiments
  - reproducibility
---


# NAMAA — KnowledgeGraphEval 2026 AdaptNER: Development-Phase Experiments

**KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026**

This is the **development-phase** experiment record of team NAMAA. It holds the training and
inference notebooks, the per-run metrics and configurations, the Konooz development-set
predictions and submissions for every candidate system, the evaluation scripts, and the
software environment.

> **How this relates to the paper.** These experiments come before the submitted systems
> described in the camera-ready paper. The submitted AdaptNER system **ranked 1st (test

> micro-F1 75.96)**. It is an exact-span majority vote over ARBERTv2 + CRF, XLM-R-large + CRF and
> ARBERTv2 + GlobalPointer. The submitted RE system **ranked 2nd (93.48)**. It is a majority
> vote over LoRA-fine-tuned Qwen3-8B, Qwen3.5-4B and Llama-3.1-8B-Instruct. Full test tables
> are in [§2](#2-official-test-results-camera-ready-paper).

The fine-tuned weights are in their own model repositories:

| Checkpoint | Formulation | Wojood val F1 | Wojood test F1 | Konooz dev F1 |
|---|---|---:|---:|---:|
| [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested) | nested, multi-label | 0.9245 | **0.9284** | **0.73** |
| [`KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested) | nested, multi-label | 0.9071 | 0.9092 | 0.67 |
| [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat) | flat, single-label | 0.9003 | not recorded | 0.69 |

All scores are entity-level strict micro-F1.

---

## Repository layout

```

├── experiments/                      per-run metrics + full config (repro_bundle.json)

│   ├── EXP-001_AraBERTv02_Flat/

│   ├── EXP-001_AraBERTv02_Nested/

│   ├── EXP-002_MARBERTv2_Flat/       weights not released; test predictions kept here

│   └── EXP-002_MARBERTv2_Nested/

├── notebooks/

│   ├── training/                     Wojood fine-tuning, one notebook per run

│   ├── konooz_dev_inference/         load weights from the Hub, predict Konooz dev, write submission

│   ├── blind_test_inference/         blind-test inference for the ARBERTv2 (unweighted) system

│   └── eda/                          Wojood + Konooz exploratory analysis

├── konooz_dev_predictions/

│   ├── namaa_wojood_runs/            EXP-001/002 predictions + submission zips

│   ├── backbone_sweep/               the nested backbone comparison (paper Table 4)

│   └── ensembles_and_llm/            top-3 / top-4 voting ensembles, LLM run

├── evaluation/                       per-entity-type and per-domain scorers + results

├── scripts/                          ensemble voting, Konooz EDA, hosted-LLM baseline

└── environment/pip_freeze.txt

```

---

## 1. Wojood experiments (in-domain)

Same recipe for all four runs, taken from `repro_bundle.json`: 8 epochs, learning rate 3e-5,
batch 32, warmup 0.1, weight decay 0.01, AdamW, bf16, max length 256, seed 42, early-stopping
patience 3. The nested runs use a 42-way sigmoid head (21 types × {B, I}) with a 0.5 threshold.
The flat runs use a softmax BIO head.

| Run | Encoder | Formulation | Val P | Val R | **Val F1** | Test P | Test R | **Test F1** |
|---|---|---|---:|---:|---:|---:|---:|---:|
| EXP-001 | AraBERTv02 | flat | 0.8908 | 0.9101 | 0.9003 | — | — | — |
| EXP-001 | AraBERTv02 | **nested** | 0.9178 | 0.9313 | **0.9245** | 0.9208 | 0.9362 | **0.9284** |
| EXP-002 | MARBERTv2 | flat | 0.8876 | 0.9087 | 0.8980 | 0.8979 | 0.9171 | 0.9074 |
| EXP-002 | MARBERTv2 | nested | 0.9042 | 0.9100 | 0.9071 | 0.9058 | 0.9127 | 0.9092 |

The EXP-001 flat run did not log test-split metrics. Flat and nested scores are measured on
different variants of Wojood, so compare them as indicative only.

**Per-entity-type macro-F1 on the Wojood test split** (`evaluation/pertype_results.json`):

| | AraBERTv02 nested | MARBERTv2 nested |
|---|---:|---:|
| micro-F1 | 0.9284 | 0.9092 |
| **macro-F1 (21 types)** | **0.8411** | **0.5877** |

MARBERTv2 nested makes no predictions at all for `PERCENT`, `PRODUCT`, `UNIT` and `QUANTITY`,
so their F1 is 0.0000. These types are 0.6% of mentions, which is why a 2-point micro-F1 gap
becomes a 25-point macro-F1 gap. AraBERTv02 scores higher on all 21 types and on all 13
Wojood sub-corpora. `evaluation/compute_pertype.py` and `compute_perdomain.py` regenerate these
numbers from each model's `test_predictions_conll.txt`. The recomputed precision matches the
original scorer exactly, and micro-F1 matches it to within 0.001.

---

## 2. Official test results (camera-ready paper)

Entity-level micro-averages under exact span-and-type matching, in percent.

**Subtask 1 — AdaptNER** (Konooz test). **Ranked 1st.**

| System | Recall | Precision | Micro-F1 |
|---|---:|---:|---:|
| Shared-task baseline | 63.87 | 62.30 | 63.00 |
| ARBERTv2 + CRF | 70.08 | 80.69 | 75.01 |
| XLM-R-large + CRF | 71.12 | 79.41 | 75.04 |
| ARBERTv2 + GlobalPointer | 66.17 | 84.02 | 74.03 |
| **Exact-span ensemble** | 69.44 | 83.83 | **75.96** |

**Subtask 2 — Relation Extraction** (WojoodRelations). **Ranked 2nd.**

| System | Recall | Precision | Micro-F1 |
|---|---:|---:|---:|
| Shared-task baseline | – | – | 93.09 |
| Qwen3-8B (LoRA) | 89.35 | 95.59 | 92.36 |
| Qwen3.5-4B (LoRA) | 89.58 | 96.20 | 92.77 |
| Llama-3.1-8B-Instruct (LoRA) | 89.03 | 96.21 | 92.48 |
| **Majority-vote ensemble** | 90.38 | 96.80 | **93.48** |

The checkpoints for these submitted systems are not in this repository.

## 3. Development-phase Konooz leaderboard

These are the team's scores on the Konooz **development** set (100 sentences) during
development. They are not reported in the camera-ready paper.

| System | Konooz dev F1 | Predictions in this repo |
|---|---:|---|
| Ensemble (top-3, weighted vote) | **0.74** | `ensembles_and_llm/submission_ensemble_top3.zip` |
| ARBERTv2 (unweighted) | **0.74** | `backbone_sweep/arbertv2-unweighted_*` |
| AraBERTv02, nested (EXP-001) | 0.73 | `namaa_wojood_runs/exp001_nested_konooz/` |
| ARBERTv2 (class-weighted) | 0.72 | `backbone_sweep/arbertv2-weighted_*` |
| XLM-R large | 0.71 | `backbone_sweep/xlm-roberta-large_*` |
| AraBERTv02, flat (EXP-001) | 0.69 | `namaa_wojood_runs/exp001_flat_konooz/` |
| MARBERT | 0.68 | `backbone_sweep/marbert_*` |
| mDeBERTa-v3 base | 0.67 | `backbone_sweep/mdeberta-v3-base_*` |
| MARBERTv2 (sweep run, not the released EXP-002 checkpoint) | 0.67 | `backbone_sweep/marbertv2_*` |
| ARBERTv2 + Konooz-MLM (last-3) | 0.65 | `backbone_sweep/arbertv2-konooz-adapted-last3_*` |
| AraBERTv02 baseline † | 0.61 | — |
| Qwen3-235B (zero-shot) | 0.26 | — |

† Organiser reference baseline. On the development set, the top-3 ensemble matches the best single system on
micro-F1 but raises macro-F1 from 0.49 to 0.55 (P = 0.75, R = 0.74).

`backbone_sweep/` also contains predictions for CAMeLBERT-mix and ARBERTv2 + Konooz-MLM (full).
These two runs were scored only on Wojood validation (paper Table 4) and were not submitted
to the Konooz leaderboard.

**Submission format:** one token per line, followed by 21 space-separated IOB2 columns in this
order:

```

CARDINAL CURR DATE EVENT FAC GPE LANGUAGE LAW LOC MONEY NORP

OCC ORDINAL ORG PERCENT PERS PRODUCT QUANTITY TIME UNIT WEBSITE

```

---

## 4. Reproducing

1. Get Wojood 1.1 and the Konooz development and blind-test data from
   [SinaLab, Birzeit University](https://sina.birzeit.edu/knowledgegraph_sharedtask2026/).
   They are not redistributed here.
2. Training: run `notebooks/training/<run>.ipynb`. They were written for Google Colab (A100 or T4).
   Each run's exact configuration and library versions are in `experiments/<run>/repro_bundle.json`.
3. Konooz inference: `notebooks/konooz_dev_inference/` loads the released checkpoints from the
   Hub by name.
4. Ensemble: `scripts/ensemble_predictions.py` does a weighted vote over per-system prediction files.

The notebooks read the Hub token from a Colab Secret or the `HF_TOKEN` environment variable.
No credentials are stored in this repository.

---

## Citation

```bibtex

@inproceedings{namaa-knowledgegrapheval-2026,

  title     = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained

               {LLM} Ensembles for {A}rabic Knowledge Graph Construction},

  author    = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and

               A.azim, Mona and Nacar, Omer},

  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference

               (ArabicNLP 2026)},

  year      = {2026},

  address   = {Budapest, Hungary},

  publisher = {Association for Computational Linguistics}

}

```

The corpus citations (Wojood, Konooz) and backbone citations are in the model cards linked
above.

## License

Apache-2.0 covers the code, notebooks and predictions in this repository. Wojood and Konooz
remain under SinaLab's original terms. Prediction files contain corpus tokens, so follow those
terms if you redistribute them.