Align card with camera-ready paper: title, authors, official test results; mark as development-phase checkpoint

README.md

@@ -1,171 +1,204 @@
----
-language: ar
-license: apache-2.0
-tags:
-  - arabic
-  - ner
-  - nested-ner
-  - wojood
-  - konooz
-  - knowledgegrapheval-2026
-  - arabicnlp-2026
-  - adaptner
-  - experiments
-  - reproducibility
----
-
-# NAMAA Community — KnowledgeGraphEval 2026 AdaptNER: Experiments
-
-**KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026**
-
-This is the experiment record behind the NAMAA Community system paper. It holds the training
-and inference notebooks, the per-run metrics and configurations, the Konooz development-set
-predictions and submissions for every system in the paper, the evaluation scripts, and the
-software environment.
-
-The fine-tuned weights are in their own model repositories:
-
-| Checkpoint | Formulation | Wojood val F1 | Wojood test F1 | Konooz dev F1 |
-|---|---|---:|---:|---:|
-| [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested) | nested, multi-label | 0.9245 | **0.9284** | **0.73** |
-| [`KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested) | nested, multi-label | 0.9071 | 0.9092 | 0.67 |
-| [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat) | flat, single-label | 0.9003 | not recorded | 0.69 |
-
-All scores are entity-level strict micro-F1.
-
----
-
-## Repository layout
-
-```
-├── experiments/                      per-run metrics + full config (repro_bundle.json)
-│   ├── EXP-001_AraBERTv02_Flat/
-│   ├── EXP-001_AraBERTv02_Nested/
-│   ├── EXP-002_MARBERTv2_Flat/       weights not released; test predictions kept here
-│   └── EXP-002_MARBERTv2_Nested/
-├── notebooks/
-│   ├── training/                     Wojood fine-tuning, one notebook per run
-│   ├── konooz_dev_inference/         load weights from the Hub, predict Konooz dev, write submission
-│   ├── blind_test_inference/         blind-test inference for the ARBERTv2 (unweighted) system
-│   └── eda/                          Wojood + Konooz exploratory analysis
-├── konooz_dev_predictions/
-│   ├── namaa_wojood_runs/            EXP-001/002 predictions + submission zips
-│   ├── backbone_sweep/               the nested backbone comparison (paper Table 4)
-│   └── ensembles_and_llm/            top-3 / top-4 voting ensembles, LLM run
-├── evaluation/                       per-entity-type and per-domain scorers + results
-├── scripts/                          ensemble voting, Konooz EDA, hosted-LLM baseline
-└── environment/pip_freeze.txt
-```
-
----
-
-## 1. Wojood experiments (in-domain)
-
-Same recipe for all four runs, taken from `repro_bundle.json`: 8 epochs, learning rate 3e-5,
-batch 32, warmup 0.1, weight decay 0.01, AdamW, bf16, max length 256, seed 42, early-stopping
-patience 3. The nested runs use a 42-way sigmoid head (21 types × {B, I}) with a 0.5 threshold.
-The flat runs use a softmax BIO head.
-
-| Run | Encoder | Formulation | Val P | Val R | **Val F1** | Test P | Test R | **Test F1** |
-|---|---|---|---:|---:|---:|---:|---:|---:|
-| EXP-001 | AraBERTv02 | flat | 0.8908 | 0.9101 | 0.9003 | — | — | — |
-| EXP-001 | AraBERTv02 | **nested** | 0.9178 | 0.9313 | **0.9245** | 0.9208 | 0.9362 | **0.9284** |
-| EXP-002 | MARBERTv2 | flat | 0.8876 | 0.9087 | 0.8980 | 0.8979 | 0.9171 | 0.9074 |
-| EXP-002 | MARBERTv2 | nested | 0.9042 | 0.9100 | 0.9071 | 0.9058 | 0.9127 | 0.9092 |
-
-The EXP-001 flat run did not log test-split metrics. Flat and nested scores are measured on
-different variants of Wojood, so compare them as indicative only.
-
-**Per-entity-type macro-F1 on the Wojood test split** (`evaluation/pertype_results.json`):
-
-| | AraBERTv02 nested | MARBERTv2 nested |
-|---|---:|---:|
-| micro-F1 | 0.9284 | 0.9092 |
-| **macro-F1 (21 types)** | **0.8411** | **0.5877** |
-
-MARBERTv2 nested makes no predictions at all for `PERCENT`, `PRODUCT`, `UNIT` and `QUANTITY`,
-so their F1 is 0.0000. These types are 0.6% of mentions, which is why a 2-point micro-F1 gap
-becomes a 25-point macro-F1 gap. AraBERTv02 scores higher on all 21 types and on all 13
-Wojood sub-corpora. `evaluation/compute_pertype.py` and `compute_perdomain.py` regenerate these
-numbers from each model's `test_predictions_conll.txt`. The recomputed precision matches the
-original scorer exactly, and micro-F1 matches it to within 0.001.
-
----
-
-## 2. Konooz development leaderboard (cross-domain, official)
-
-Team results from the paper (Table 1). The best AdaptNER system ranked **1st**.
-
-| System | Konooz dev F1 | Predictions in this repo |
-|---|---:|---|
-| Ensemble (top-3, weighted vote) | **0.74** | `ensembles_and_llm/submission_ensemble_top3.zip` |
-| ARBERTv2 (unweighted) — *1st place* | **0.74** | `backbone_sweep/arbertv2-unweighted_*` |
-| AraBERTv02, nested (EXP-001) | 0.73 | `namaa_wojood_runs/exp001_nested_konooz/` |
-| ARBERTv2 (class-weighted) | 0.72 | `backbone_sweep/arbertv2-weighted_*` |
-| XLM-R large | 0.71 | `backbone_sweep/xlm-roberta-large_*` |
-| AraBERTv02, flat (EXP-001) | 0.69 | `namaa_wojood_runs/exp001_flat_konooz/` |
-| MARBERT | 0.68 | `backbone_sweep/marbert_*` |
-| mDeBERTa-v3 base | 0.67 | `backbone_sweep/mdeberta-v3-base_*` |
-| MARBERTv2 | 0.67 | `backbone_sweep/marbertv2_*` |
-| ARBERTv2 + Konooz-MLM (last-3) | 0.65 | `backbone_sweep/arbertv2-konooz-adapted-last3_*` |
-| AraBERTv02 baseline † | 0.61 | — |
-| Qwen3-235B (zero-shot) | 0.26 | — |
-
-† Organiser reference baseline. The top-3 ensemble matches the best single system on
-micro-F1 but raises macro-F1 from 0.49 to 0.55 (P = 0.75, R = 0.74).
-
-`backbone_sweep/` also contains predictions for CAMeLBERT-mix and ARBERTv2 + Konooz-MLM (full).
-These two runs were scored only on Wojood validation (paper Table 4) and were not submitted
-to the Konooz leaderboard.
-
-**Submission format:** one token per line, followed by 21 space-separated IOB2 columns in this
-order:
-
-```
-CARDINAL CURR DATE EVENT FAC GPE LANGUAGE LAW LOC MONEY NORP
-OCC ORDINAL ORG PERCENT PERS PRODUCT QUANTITY TIME UNIT WEBSITE
-```
-
----
-
-## 3. Reproducing
-
-1. Get Wojood 1.1 and the Konooz development and blind-test data from
-   [SinaLab, Birzeit University](https://sina.birzeit.edu/knowledgegraph_sharedtask2026/).
-   They are not redistributed here.
-2. Training: run `notebooks/training/<run>.ipynb`. They were written for Google Colab (A100 or T4).
-   Each run's exact configuration and library versions are in `experiments/<run>/repro_bundle.json`.
-3. Konooz inference: `notebooks/konooz_dev_inference/` loads the released checkpoints from the
-   Hub by name.
-4. Ensemble: `scripts/ensemble_predictions.py` does a weighted vote over per-system prediction files.
-
-The notebooks read the Hub token from a Colab Secret or the `HF_TOKEN` environment variable.
-No credentials are stored in this repository.
-
----
-
-## Citation
-
-```bibtex
-@inproceedings{namaa-knowledgegrapheval-2026,
-  title     = {{NAMAA} Community at {KnowledgeGraphEval} 2026: Multi-Label Nested {NER}
-               and Entity-Pair Relation Extraction for Cross-Domain Arabic
-               Knowledge Graph Construction},
-  author    = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and
-               Abdelazim, Mona and Nacer, Omar},
-  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
-               (ArabicNLP 2026)},
-  year      = {2026},
-  address   = {Budapest, Hungary},
-  publisher = {Association for Computational Linguistics}
-}
-```
-
-The corpus citations (Wojood, Konooz) and backbone citations are in the model cards linked
-above.
-
-## License
-
-Apache-2.0 covers the code, notebooks and predictions in this repository. Wojood and Konooz
-remain under SinaLab's original terms. Prediction files contain corpus tokens, so follow those
-terms if you redistribute them.
+---
+language: ar
+license: apache-2.0
+tags:
+  - arabic
+  - ner
+  - nested-ner
+  - wojood
+  - konooz
+  - knowledgegrapheval-2026
+  - arabicnlp-2026
+  - adaptner
+  - experiments
+  - reproducibility
+---
+
+# NAMAA — KnowledgeGraphEval 2026 AdaptNER: Development-Phase Experiments
+
+**KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026**
+
+This is the **development-phase** experiment record of team NAMAA. It holds the training and
+inference notebooks, the per-run metrics and configurations, the Konooz development-set
+predictions and submissions for every candidate system, the evaluation scripts, and the
+software environment.
+
+> **How this relates to the paper.** These experiments come before the submitted systems
+> described in the camera-ready paper. The submitted AdaptNER system **ranked 1st (test
+> micro-F1 75.96)**. It is an exact-span majority vote over ARBERTv2 + CRF, XLM-R-large + CRF and
+> ARBERTv2 + GlobalPointer. The submitted RE system **ranked 2nd (93.48)**. It is a majority
+> vote over LoRA-fine-tuned Qwen3-8B, Qwen3.5-4B and Llama-3.1-8B-Instruct. Full test tables
+> are in [§2](#2-official-test-results-camera-ready-paper).
+
+The fine-tuned weights are in their own model repositories:
+
+| Checkpoint | Formulation | Wojood val F1 | Wojood test F1 | Konooz dev F1 |
+|---|---|---:|---:|---:|
+| [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested) | nested, multi-label | 0.9245 | **0.9284** | **0.73** |
+| [`KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested) | nested, multi-label | 0.9071 | 0.9092 | 0.67 |
+| [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat) | flat, single-label | 0.9003 | not recorded | 0.69 |
+
+All scores are entity-level strict micro-F1.
+
+---
+
+## Repository layout
+
+```
+├── experiments/                      per-run metrics + full config (repro_bundle.json)
+│   ├── EXP-001_AraBERTv02_Flat/
+│   ├── EXP-001_AraBERTv02_Nested/
+│   ├── EXP-002_MARBERTv2_Flat/       weights not released; test predictions kept here
+│   └── EXP-002_MARBERTv2_Nested/
+├── notebooks/
+│   ├── training/                     Wojood fine-tuning, one notebook per run
+│   ├── konooz_dev_inference/         load weights from the Hub, predict Konooz dev, write submission
+│   ├── blind_test_inference/         blind-test inference for the ARBERTv2 (unweighted) system
+│   └── eda/                          Wojood + Konooz exploratory analysis
+├── konooz_dev_predictions/
+│   ├── namaa_wojood_runs/            EXP-001/002 predictions + submission zips
+│   ├── backbone_sweep/               the nested backbone comparison (paper Table 4)
+│   └── ensembles_and_llm/            top-3 / top-4 voting ensembles, LLM run
+├── evaluation/                       per-entity-type and per-domain scorers + results
+├── scripts/                          ensemble voting, Konooz EDA, hosted-LLM baseline
+└── environment/pip_freeze.txt
+```
+
+---
+
+## 1. Wojood experiments (in-domain)
+
+Same recipe for all four runs, taken from `repro_bundle.json`: 8 epochs, learning rate 3e-5,
+batch 32, warmup 0.1, weight decay 0.01, AdamW, bf16, max length 256, seed 42, early-stopping
+patience 3. The nested runs use a 42-way sigmoid head (21 types × {B, I}) with a 0.5 threshold.
+The flat runs use a softmax BIO head.
+
+| Run | Encoder | Formulation | Val P | Val R | **Val F1** | Test P | Test R | **Test F1** |
+|---|---|---|---:|---:|---:|---:|---:|---:|
+| EXP-001 | AraBERTv02 | flat | 0.8908 | 0.9101 | 0.9003 | — | — | — |
+| EXP-001 | AraBERTv02 | **nested** | 0.9178 | 0.9313 | **0.9245** | 0.9208 | 0.9362 | **0.9284** |
+| EXP-002 | MARBERTv2 | flat | 0.8876 | 0.9087 | 0.8980 | 0.8979 | 0.9171 | 0.9074 |
+| EXP-002 | MARBERTv2 | nested | 0.9042 | 0.9100 | 0.9071 | 0.9058 | 0.9127 | 0.9092 |
+
+The EXP-001 flat run did not log test-split metrics. Flat and nested scores are measured on
+different variants of Wojood, so compare them as indicative only.
+
+**Per-entity-type macro-F1 on the Wojood test split** (`evaluation/pertype_results.json`):
+
+| | AraBERTv02 nested | MARBERTv2 nested |
+|---|---:|---:|
+| micro-F1 | 0.9284 | 0.9092 |
+| **macro-F1 (21 types)** | **0.8411** | **0.5877** |
+
+MARBERTv2 nested makes no predictions at all for `PERCENT`, `PRODUCT`, `UNIT` and `QUANTITY`,
+so their F1 is 0.0000. These types are 0.6% of mentions, which is why a 2-point micro-F1 gap
+becomes a 25-point macro-F1 gap. AraBERTv02 scores higher on all 21 types and on all 13
+Wojood sub-corpora. `evaluation/compute_pertype.py` and `compute_perdomain.py` regenerate these
+numbers from each model's `test_predictions_conll.txt`. The recomputed precision matches the
+original scorer exactly, and micro-F1 matches it to within 0.001.
+
+---
+
+## 2. Official test results (camera-ready paper)
+
+Entity-level micro-averages under exact span-and-type matching, in percent.
+
+**Subtask 1 — AdaptNER** (Konooz test). **Ranked 1st.**
+
+| System | Recall | Precision | Micro-F1 |
+|---|---:|---:|---:|
+| Shared-task baseline | 63.87 | 62.30 | 63.00 |
+| ARBERTv2 + CRF | 70.08 | 80.69 | 75.01 |
+| XLM-R-large + CRF | 71.12 | 79.41 | 75.04 |
+| ARBERTv2 + GlobalPointer | 66.17 | 84.02 | 74.03 |
+| **Exact-span ensemble** | 69.44 | 83.83 | **75.96** |
+
+**Subtask 2 — Relation Extraction** (WojoodRelations). **Ranked 2nd.**
+
+| System | Recall | Precision | Micro-F1 |
+|---|---:|---:|---:|
+| Shared-task baseline | – | – | 93.09 |
+| Qwen3-8B (LoRA) | 89.35 | 95.59 | 92.36 |
+| Qwen3.5-4B (LoRA) | 89.58 | 96.20 | 92.77 |
+| Llama-3.1-8B-Instruct (LoRA) | 89.03 | 96.21 | 92.48 |
+| **Majority-vote ensemble** | 90.38 | 96.80 | **93.48** |
+
+The checkpoints for these submitted systems are not in this repository.
+
+## 3. Development-phase Konooz leaderboard
+
+These are the team's scores on the Konooz **development** set (100 sentences) during
+development. They are not reported in the camera-ready paper.
+
+| System | Konooz dev F1 | Predictions in this repo |
+|---|---:|---|
+| Ensemble (top-3, weighted vote) | **0.74** | `ensembles_and_llm/submission_ensemble_top3.zip` |
+| ARBERTv2 (unweighted) | **0.74** | `backbone_sweep/arbertv2-unweighted_*` |
+| AraBERTv02, nested (EXP-001) | 0.73 | `namaa_wojood_runs/exp001_nested_konooz/` |
+| ARBERTv2 (class-weighted) | 0.72 | `backbone_sweep/arbertv2-weighted_*` |
+| XLM-R large | 0.71 | `backbone_sweep/xlm-roberta-large_*` |
+| AraBERTv02, flat (EXP-001) | 0.69 | `namaa_wojood_runs/exp001_flat_konooz/` |
+| MARBERT | 0.68 | `backbone_sweep/marbert_*` |
+| mDeBERTa-v3 base | 0.67 | `backbone_sweep/mdeberta-v3-base_*` |
+| MARBERTv2 (sweep run, not the released EXP-002 checkpoint) | 0.67 | `backbone_sweep/marbertv2_*` |
+| ARBERTv2 + Konooz-MLM (last-3) | 0.65 | `backbone_sweep/arbertv2-konooz-adapted-last3_*` |
+| AraBERTv02 baseline † | 0.61 | — |
+| Qwen3-235B (zero-shot) | 0.26 | — |
+
+† Organiser reference baseline. On the development set, the top-3 ensemble matches the best single system on
+micro-F1 but raises macro-F1 from 0.49 to 0.55 (P = 0.75, R = 0.74).
+
+`backbone_sweep/` also contains predictions for CAMeLBERT-mix and ARBERTv2 + Konooz-MLM (full).
+These two runs were scored only on Wojood validation (paper Table 4) and were not submitted
+to the Konooz leaderboard.
+
+**Submission format:** one token per line, followed by 21 space-separated IOB2 columns in this
+order:
+
+```
+CARDINAL CURR DATE EVENT FAC GPE LANGUAGE LAW LOC MONEY NORP
+OCC ORDINAL ORG PERCENT PERS PRODUCT QUANTITY TIME UNIT WEBSITE
+```
+
+---
+
+## 4. Reproducing
+
+1. Get Wojood 1.1 and the Konooz development and blind-test data from
+   [SinaLab, Birzeit University](https://sina.birzeit.edu/knowledgegraph_sharedtask2026/).
+   They are not redistributed here.
+2. Training: run `notebooks/training/<run>.ipynb`. They were written for Google Colab (A100 or T4).
+   Each run's exact configuration and library versions are in `experiments/<run>/repro_bundle.json`.
+3. Konooz inference: `notebooks/konooz_dev_inference/` loads the released checkpoints from the
+   Hub by name.
+4. Ensemble: `scripts/ensemble_predictions.py` does a weighted vote over per-system prediction files.
+
+The notebooks read the Hub token from a Colab Secret or the `HF_TOKEN` environment variable.
+No credentials are stored in this repository.
+
+---
+
+## Citation
+
+```bibtex
+@inproceedings{namaa-knowledgegrapheval-2026,
+  title     = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained
+               {LLM} Ensembles for {A}rabic Knowledge Graph Construction},
+  author    = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and
+               A.azim, Mona and Nacar, Omer},
+  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
+               (ArabicNLP 2026)},
+  year      = {2026},
+  address   = {Budapest, Hungary},
+  publisher = {Association for Computational Linguistics}
+}
+```
+
+The corpus citations (Wojood, Konooz) and backbone citations are in the model cards linked
+above.
+
+## License
+
+Apache-2.0 covers the code, notebooks and predictions in this repository. Wojood and Konooz
+remain under SinaLab's original terms. Prediction files contain corpus tokens, so follow those
+terms if you redistribute them.