Align card with camera-ready paper: title, authors, official test results; mark as development-phase checkpoint
c80bc5b verified | language: ar | |
| license: apache-2.0 | |
| tags: | |
| - arabic | |
| - ner | |
| - nested-ner | |
| - wojood | |
| - konooz | |
| - knowledgegrapheval-2026 | |
| - arabicnlp-2026 | |
| - adaptner | |
| - experiments | |
| - reproducibility | |
| # NAMAA — KnowledgeGraphEval 2026 AdaptNER: Development-Phase Experiments | |
| **KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026** | |
| This is the **development-phase** experiment record of team NAMAA. It holds the training and | |
| inference notebooks, the per-run metrics and configurations, the Konooz development-set | |
| predictions and submissions for every candidate system, the evaluation scripts, and the | |
| software environment. | |
| > **How this relates to the paper.** These experiments come before the submitted systems | |
| > described in the camera-ready paper. The submitted AdaptNER system **ranked 1st (test | |
| > micro-F1 75.96)**. It is an exact-span majority vote over ARBERTv2 + CRF, XLM-R-large + CRF and | |
| > ARBERTv2 + GlobalPointer. The submitted RE system **ranked 2nd (93.48)**. It is a majority | |
| > vote over LoRA-fine-tuned Qwen3-8B, Qwen3.5-4B and Llama-3.1-8B-Instruct. Full test tables | |
| > are in [§2](#2-official-test-results-camera-ready-paper). | |
| The fine-tuned weights are in their own model repositories: | |
| | Checkpoint | Formulation | Wojood val F1 | Wojood test F1 | Konooz dev F1 | | |
| |---|---|---:|---:|---:| | |
| | [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested) | nested, multi-label | 0.9245 | **0.9284** | **0.73** | | |
| | [`KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested) | nested, multi-label | 0.9071 | 0.9092 | 0.67 | | |
| | [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat) | flat, single-label | 0.9003 | not recorded | 0.69 | | |
| All scores are entity-level strict micro-F1. | |
| --- | |
| ## Repository layout | |
| ``` | |
| ├── experiments/ per-run metrics + full config (repro_bundle.json) | |
| │ ├── EXP-001_AraBERTv02_Flat/ | |
| │ ├── EXP-001_AraBERTv02_Nested/ | |
| │ ├── EXP-002_MARBERTv2_Flat/ weights not released; test predictions kept here | |
| │ └── EXP-002_MARBERTv2_Nested/ | |
| ├── notebooks/ | |
| │ ├── training/ Wojood fine-tuning, one notebook per run | |
| │ ├── konooz_dev_inference/ load weights from the Hub, predict Konooz dev, write submission | |
| │ ├── blind_test_inference/ blind-test inference for the ARBERTv2 (unweighted) system | |
| │ └── eda/ Wojood + Konooz exploratory analysis | |
| ├── konooz_dev_predictions/ | |
| │ ├── namaa_wojood_runs/ EXP-001/002 predictions + submission zips | |
| │ ├── backbone_sweep/ the nested backbone comparison (paper Table 4) | |
| │ └── ensembles_and_llm/ top-3 / top-4 voting ensembles, LLM run | |
| ├── evaluation/ per-entity-type and per-domain scorers + results | |
| ├── scripts/ ensemble voting, Konooz EDA, hosted-LLM baseline | |
| └── environment/pip_freeze.txt | |
| ``` | |
| --- | |
| ## 1. Wojood experiments (in-domain) | |
| Same recipe for all four runs, taken from `repro_bundle.json`: 8 epochs, learning rate 3e-5, | |
| batch 32, warmup 0.1, weight decay 0.01, AdamW, bf16, max length 256, seed 42, early-stopping | |
| patience 3. The nested runs use a 42-way sigmoid head (21 types × {B, I}) with a 0.5 threshold. | |
| The flat runs use a softmax BIO head. | |
| | Run | Encoder | Formulation | Val P | Val R | **Val F1** | Test P | Test R | **Test F1** | | |
| |---|---|---|---:|---:|---:|---:|---:|---:| | |
| | EXP-001 | AraBERTv02 | flat | 0.8908 | 0.9101 | 0.9003 | — | — | — | | |
| | EXP-001 | AraBERTv02 | **nested** | 0.9178 | 0.9313 | **0.9245** | 0.9208 | 0.9362 | **0.9284** | | |
| | EXP-002 | MARBERTv2 | flat | 0.8876 | 0.9087 | 0.8980 | 0.8979 | 0.9171 | 0.9074 | | |
| | EXP-002 | MARBERTv2 | nested | 0.9042 | 0.9100 | 0.9071 | 0.9058 | 0.9127 | 0.9092 | | |
| The EXP-001 flat run did not log test-split metrics. Flat and nested scores are measured on | |
| different variants of Wojood, so compare them as indicative only. | |
| **Per-entity-type macro-F1 on the Wojood test split** (`evaluation/pertype_results.json`): | |
| | | AraBERTv02 nested | MARBERTv2 nested | | |
| |---|---:|---:| | |
| | micro-F1 | 0.9284 | 0.9092 | | |
| | **macro-F1 (21 types)** | **0.8411** | **0.5877** | | |
| MARBERTv2 nested makes no predictions at all for `PERCENT`, `PRODUCT`, `UNIT` and `QUANTITY`, | |
| so their F1 is 0.0000. These types are 0.6% of mentions, which is why a 2-point micro-F1 gap | |
| becomes a 25-point macro-F1 gap. AraBERTv02 scores higher on all 21 types and on all 13 | |
| Wojood sub-corpora. `evaluation/compute_pertype.py` and `compute_perdomain.py` regenerate these | |
| numbers from each model's `test_predictions_conll.txt`. The recomputed precision matches the | |
| original scorer exactly, and micro-F1 matches it to within 0.001. | |
| --- | |
| ## 2. Official test results (camera-ready paper) | |
| Entity-level micro-averages under exact span-and-type matching, in percent. | |
| **Subtask 1 — AdaptNER** (Konooz test). **Ranked 1st.** | |
| | System | Recall | Precision | Micro-F1 | | |
| |---|---:|---:|---:| | |
| | Shared-task baseline | 63.87 | 62.30 | 63.00 | | |
| | ARBERTv2 + CRF | 70.08 | 80.69 | 75.01 | | |
| | XLM-R-large + CRF | 71.12 | 79.41 | 75.04 | | |
| | ARBERTv2 + GlobalPointer | 66.17 | 84.02 | 74.03 | | |
| | **Exact-span ensemble** | 69.44 | 83.83 | **75.96** | | |
| **Subtask 2 — Relation Extraction** (WojoodRelations). **Ranked 2nd.** | |
| | System | Recall | Precision | Micro-F1 | | |
| |---|---:|---:|---:| | |
| | Shared-task baseline | – | – | 93.09 | | |
| | Qwen3-8B (LoRA) | 89.35 | 95.59 | 92.36 | | |
| | Qwen3.5-4B (LoRA) | 89.58 | 96.20 | 92.77 | | |
| | Llama-3.1-8B-Instruct (LoRA) | 89.03 | 96.21 | 92.48 | | |
| | **Majority-vote ensemble** | 90.38 | 96.80 | **93.48** | | |
| The checkpoints for these submitted systems are not in this repository. | |
| ## 3. Development-phase Konooz leaderboard | |
| These are the team's scores on the Konooz **development** set (100 sentences) during | |
| development. They are not reported in the camera-ready paper. | |
| | System | Konooz dev F1 | Predictions in this repo | | |
| |---|---:|---| | |
| | Ensemble (top-3, weighted vote) | **0.74** | `ensembles_and_llm/submission_ensemble_top3.zip` | | |
| | ARBERTv2 (unweighted) | **0.74** | `backbone_sweep/arbertv2-unweighted_*` | | |
| | AraBERTv02, nested (EXP-001) | 0.73 | `namaa_wojood_runs/exp001_nested_konooz/` | | |
| | ARBERTv2 (class-weighted) | 0.72 | `backbone_sweep/arbertv2-weighted_*` | | |
| | XLM-R large | 0.71 | `backbone_sweep/xlm-roberta-large_*` | | |
| | AraBERTv02, flat (EXP-001) | 0.69 | `namaa_wojood_runs/exp001_flat_konooz/` | | |
| | MARBERT | 0.68 | `backbone_sweep/marbert_*` | | |
| | mDeBERTa-v3 base | 0.67 | `backbone_sweep/mdeberta-v3-base_*` | | |
| | MARBERTv2 (sweep run, not the released EXP-002 checkpoint) | 0.67 | `backbone_sweep/marbertv2_*` | | |
| | ARBERTv2 + Konooz-MLM (last-3) | 0.65 | `backbone_sweep/arbertv2-konooz-adapted-last3_*` | | |
| | AraBERTv02 baseline † | 0.61 | — | | |
| | Qwen3-235B (zero-shot) | 0.26 | — | | |
| † Organiser reference baseline. On the development set, the top-3 ensemble matches the best single system on | |
| micro-F1 but raises macro-F1 from 0.49 to 0.55 (P = 0.75, R = 0.74). | |
| `backbone_sweep/` also contains predictions for CAMeLBERT-mix and ARBERTv2 + Konooz-MLM (full). | |
| These two runs were scored only on Wojood validation (paper Table 4) and were not submitted | |
| to the Konooz leaderboard. | |
| **Submission format:** one token per line, followed by 21 space-separated IOB2 columns in this | |
| order: | |
| ``` | |
| CARDINAL CURR DATE EVENT FAC GPE LANGUAGE LAW LOC MONEY NORP | |
| OCC ORDINAL ORG PERCENT PERS PRODUCT QUANTITY TIME UNIT WEBSITE | |
| ``` | |
| --- | |
| ## 4. Reproducing | |
| 1. Get Wojood 1.1 and the Konooz development and blind-test data from | |
| [SinaLab, Birzeit University](https://sina.birzeit.edu/knowledgegraph_sharedtask2026/). | |
| They are not redistributed here. | |
| 2. Training: run `notebooks/training/<run>.ipynb`. They were written for Google Colab (A100 or T4). | |
| Each run's exact configuration and library versions are in `experiments/<run>/repro_bundle.json`. | |
| 3. Konooz inference: `notebooks/konooz_dev_inference/` loads the released checkpoints from the | |
| Hub by name. | |
| 4. Ensemble: `scripts/ensemble_predictions.py` does a weighted vote over per-system prediction files. | |
| The notebooks read the Hub token from a Colab Secret or the `HF_TOKEN` environment variable. | |
| No credentials are stored in this repository. | |
| --- | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{namaa-knowledgegrapheval-2026, | |
| title = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained | |
| {LLM} Ensembles for {A}rabic Knowledge Graph Construction}, | |
| author = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and | |
| A.azim, Mona and Nacar, Omer}, | |
| booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference | |
| (ArabicNLP 2026)}, | |
| year = {2026}, | |
| address = {Budapest, Hungary}, | |
| publisher = {Association for Computational Linguistics} | |
| } | |
| ``` | |
| The corpus citations (Wojood, Konooz) and backbone citations are in the model cards linked | |
| above. | |
| ## License | |
| Apache-2.0 covers the code, notebooks and predictions in this repository. Wojood and Konooz | |
| remain under SinaLab's original terms. Prediction files contain corpus tokens, so follow those | |
| terms if you redistribute them. | |