--- language: ar license: apache-2.0 tags: - arabic - ner - nested-ner - wojood - konooz - knowledgegrapheval-2026 - arabicnlp-2026 - adaptner - experiments - reproducibility --- # NAMAA — KnowledgeGraphEval 2026 AdaptNER: Development-Phase Experiments **KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026** This is the **development-phase** experiment record of team NAMAA. It holds the training and inference notebooks, the per-run metrics and configurations, the Konooz development-set predictions and submissions for every candidate system, the evaluation scripts, and the software environment. > **How this relates to the paper.** These experiments come before the submitted systems > described in the camera-ready paper. The submitted AdaptNER system **ranked 1st (test > micro-F1 75.96)**. It is an exact-span majority vote over ARBERTv2 + CRF, XLM-R-large + CRF and > ARBERTv2 + GlobalPointer. The submitted RE system **ranked 2nd (93.48)**. It is a majority > vote over LoRA-fine-tuned Qwen3-8B, Qwen3.5-4B and Llama-3.1-8B-Instruct. Full test tables > are in [§2](#2-official-test-results-camera-ready-paper). The fine-tuned weights are in their own model repositories: | Checkpoint | Formulation | Wojood val F1 | Wojood test F1 | Konooz dev F1 | |---|---|---:|---:|---:| | [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested) | nested, multi-label | 0.9245 | **0.9284** | **0.73** | | [`KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested) | nested, multi-label | 0.9071 | 0.9092 | 0.67 | | [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat) | flat, single-label | 0.9003 | not recorded | 0.69 | All scores are entity-level strict micro-F1. --- ## Repository layout ``` ├── experiments/ per-run metrics + full config (repro_bundle.json) │ ├── EXP-001_AraBERTv02_Flat/ │ ├── EXP-001_AraBERTv02_Nested/ │ ├── EXP-002_MARBERTv2_Flat/ weights not released; test predictions kept here │ └── EXP-002_MARBERTv2_Nested/ ├── notebooks/ │ ├── training/ Wojood fine-tuning, one notebook per run │ ├── konooz_dev_inference/ load weights from the Hub, predict Konooz dev, write submission │ ├── blind_test_inference/ blind-test inference for the ARBERTv2 (unweighted) system │ └── eda/ Wojood + Konooz exploratory analysis ├── konooz_dev_predictions/ │ ├── namaa_wojood_runs/ EXP-001/002 predictions + submission zips │ ├── backbone_sweep/ the nested backbone comparison (paper Table 4) │ └── ensembles_and_llm/ top-3 / top-4 voting ensembles, LLM run ├── evaluation/ per-entity-type and per-domain scorers + results ├── scripts/ ensemble voting, Konooz EDA, hosted-LLM baseline └── environment/pip_freeze.txt ``` --- ## 1. Wojood experiments (in-domain) Same recipe for all four runs, taken from `repro_bundle.json`: 8 epochs, learning rate 3e-5, batch 32, warmup 0.1, weight decay 0.01, AdamW, bf16, max length 256, seed 42, early-stopping patience 3. The nested runs use a 42-way sigmoid head (21 types × {B, I}) with a 0.5 threshold. The flat runs use a softmax BIO head. | Run | Encoder | Formulation | Val P | Val R | **Val F1** | Test P | Test R | **Test F1** | |---|---|---|---:|---:|---:|---:|---:|---:| | EXP-001 | AraBERTv02 | flat | 0.8908 | 0.9101 | 0.9003 | — | — | — | | EXP-001 | AraBERTv02 | **nested** | 0.9178 | 0.9313 | **0.9245** | 0.9208 | 0.9362 | **0.9284** | | EXP-002 | MARBERTv2 | flat | 0.8876 | 0.9087 | 0.8980 | 0.8979 | 0.9171 | 0.9074 | | EXP-002 | MARBERTv2 | nested | 0.9042 | 0.9100 | 0.9071 | 0.9058 | 0.9127 | 0.9092 | The EXP-001 flat run did not log test-split metrics. Flat and nested scores are measured on different variants of Wojood, so compare them as indicative only. **Per-entity-type macro-F1 on the Wojood test split** (`evaluation/pertype_results.json`): | | AraBERTv02 nested | MARBERTv2 nested | |---|---:|---:| | micro-F1 | 0.9284 | 0.9092 | | **macro-F1 (21 types)** | **0.8411** | **0.5877** | MARBERTv2 nested makes no predictions at all for `PERCENT`, `PRODUCT`, `UNIT` and `QUANTITY`, so their F1 is 0.0000. These types are 0.6% of mentions, which is why a 2-point micro-F1 gap becomes a 25-point macro-F1 gap. AraBERTv02 scores higher on all 21 types and on all 13 Wojood sub-corpora. `evaluation/compute_pertype.py` and `compute_perdomain.py` regenerate these numbers from each model's `test_predictions_conll.txt`. The recomputed precision matches the original scorer exactly, and micro-F1 matches it to within 0.001. --- ## 2. Official test results (camera-ready paper) Entity-level micro-averages under exact span-and-type matching, in percent. **Subtask 1 — AdaptNER** (Konooz test). **Ranked 1st.** | System | Recall | Precision | Micro-F1 | |---|---:|---:|---:| | Shared-task baseline | 63.87 | 62.30 | 63.00 | | ARBERTv2 + CRF | 70.08 | 80.69 | 75.01 | | XLM-R-large + CRF | 71.12 | 79.41 | 75.04 | | ARBERTv2 + GlobalPointer | 66.17 | 84.02 | 74.03 | | **Exact-span ensemble** | 69.44 | 83.83 | **75.96** | **Subtask 2 — Relation Extraction** (WojoodRelations). **Ranked 2nd.** | System | Recall | Precision | Micro-F1 | |---|---:|---:|---:| | Shared-task baseline | – | – | 93.09 | | Qwen3-8B (LoRA) | 89.35 | 95.59 | 92.36 | | Qwen3.5-4B (LoRA) | 89.58 | 96.20 | 92.77 | | Llama-3.1-8B-Instruct (LoRA) | 89.03 | 96.21 | 92.48 | | **Majority-vote ensemble** | 90.38 | 96.80 | **93.48** | The checkpoints for these submitted systems are not in this repository. ## 3. Development-phase Konooz leaderboard These are the team's scores on the Konooz **development** set (100 sentences) during development. They are not reported in the camera-ready paper. | System | Konooz dev F1 | Predictions in this repo | |---|---:|---| | Ensemble (top-3, weighted vote) | **0.74** | `ensembles_and_llm/submission_ensemble_top3.zip` | | ARBERTv2 (unweighted) | **0.74** | `backbone_sweep/arbertv2-unweighted_*` | | AraBERTv02, nested (EXP-001) | 0.73 | `namaa_wojood_runs/exp001_nested_konooz/` | | ARBERTv2 (class-weighted) | 0.72 | `backbone_sweep/arbertv2-weighted_*` | | XLM-R large | 0.71 | `backbone_sweep/xlm-roberta-large_*` | | AraBERTv02, flat (EXP-001) | 0.69 | `namaa_wojood_runs/exp001_flat_konooz/` | | MARBERT | 0.68 | `backbone_sweep/marbert_*` | | mDeBERTa-v3 base | 0.67 | `backbone_sweep/mdeberta-v3-base_*` | | MARBERTv2 (sweep run, not the released EXP-002 checkpoint) | 0.67 | `backbone_sweep/marbertv2_*` | | ARBERTv2 + Konooz-MLM (last-3) | 0.65 | `backbone_sweep/arbertv2-konooz-adapted-last3_*` | | AraBERTv02 baseline † | 0.61 | — | | Qwen3-235B (zero-shot) | 0.26 | — | † Organiser reference baseline. On the development set, the top-3 ensemble matches the best single system on micro-F1 but raises macro-F1 from 0.49 to 0.55 (P = 0.75, R = 0.74). `backbone_sweep/` also contains predictions for CAMeLBERT-mix and ARBERTv2 + Konooz-MLM (full). These two runs were scored only on Wojood validation (paper Table 4) and were not submitted to the Konooz leaderboard. **Submission format:** one token per line, followed by 21 space-separated IOB2 columns in this order: ``` CARDINAL CURR DATE EVENT FAC GPE LANGUAGE LAW LOC MONEY NORP OCC ORDINAL ORG PERCENT PERS PRODUCT QUANTITY TIME UNIT WEBSITE ``` --- ## 4. Reproducing 1. Get Wojood 1.1 and the Konooz development and blind-test data from [SinaLab, Birzeit University](https://sina.birzeit.edu/knowledgegraph_sharedtask2026/). They are not redistributed here. 2. Training: run `notebooks/training/.ipynb`. They were written for Google Colab (A100 or T4). Each run's exact configuration and library versions are in `experiments//repro_bundle.json`. 3. Konooz inference: `notebooks/konooz_dev_inference/` loads the released checkpoints from the Hub by name. 4. Ensemble: `scripts/ensemble_predictions.py` does a weighted vote over per-system prediction files. The notebooks read the Hub token from a Colab Secret or the `HF_TOKEN` environment variable. No credentials are stored in this repository. --- ## Citation ```bibtex @inproceedings{namaa-knowledgegrapheval-2026, title = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained {LLM} Ensembles for {A}rabic Knowledge Graph Construction}, author = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and A.azim, Mona and Nacar, Omer}, booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference (ArabicNLP 2026)}, year = {2026}, address = {Budapest, Hungary}, publisher = {Association for Computational Linguistics} } ``` The corpus citations (Wojood, Konooz) and backbone citations are in the model cards linked above. ## License Apache-2.0 covers the code, notebooks and predictions in this repository. Wojood and Konooz remain under SinaLab's original terms. Prediction files contain corpus tokens, so follow those terms if you redistribute them.