Waqf-AI/arabic-modernbert-unified

Arabic ModernBERT Unified

A compact Arabic text encoder with 32.79M parameters for masked-language modeling and 32.94M with all task heads. Its eight-layer ModernBERT backbone supports masked-word prediction, three-way natural language inference, extractive QA, and 384-dimensional sentence embeddings.

Model property	Value
Encoder	ModernBERT, 8 layers, hidden width 384, 6 attention heads
Tokenizer	Arabic-focused BPE, 48,000 entries
Configured input limit	4,096 tokens
HQ pretraining input length	Up to 1,024 tokens per block
Weights	FP32 safetensors, about 131 MB for encoder + MLM, plus 0.60 MB for auxiliary heads
Initialization	Random weights
License	Apache-2.0

Quick start

Use the standard Transformers fill-mask pipeline. Tested with Python 3.13, PyTorch 2.13.0 and Transformers 5.14.1.

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="Abdelkareem/arabic-modernbert-unified")
for candidate in fill_mask("تقع القاهرة على ضفاف نهر [MASK].", top_k=5):
    print(candidate["token_str"], candidate["score"])

Sentence embeddings and NLI

Download the release and run the included example.

hf download Abdelkareem/arabic-modernbert-unified --revision v1.0.0 --local-dir arabic-modernbert-unified
cd arabic-modernbert-unified
pip install -r requirements.txt
python example.py

from unified_encoder import UnifiedArabicEncoder

model = UnifiedArabicEncoder(".")
embeddings = model.encode(["يقرأ الطفل كتاباً.", "الطفل يقرأ كتاباً."])
print(float(embeddings[0] @ embeddings[1]))

scores = model.classify(
    ["يقرأ الطفل كتاباً في المكتبة."],
    ["الطفل يقرأ كتاباً."],
)[0]
print(dict(zip(("entailment", "neutral", "contradiction"), scores.tolist())))

The retrieval path mean-pools non-special tokens, applies the learned projection and L2-normalizes each vector. NLI uses the first-token hidden state and the released three-way classifier. The QA head is accessible as model.qa and maps each hidden state to start and end logits. For extractive QA, encode the question as the first sequence and the context as the second, and use only context offsets for answer spans.

Training

1. Arabic pretraining. A single pass over the retained Arabic arb_Arab subset of FineWeb2-HQ, pinned at c0c06e94fd3a44ae9e802b2b0fc533817601eb5e. After cleanup, overlap exclusions and exact deduplication, the run retained 3,098,999 documents and 18.30 GB of cleaned UTF‑8 text. The model saw 2.508B content tokens and 2.517B attended positions in 142,210 updates using 15% token masking. FineWeb2-HQ dataset card
2. Language and task continuation. The selected lineage used merged Arabic Wikipedia, HQ and Arabic XL-Sum language replay, Arabic NLI, TyDi QA and filtered domain examples. The reconstructed continuation budget is ~778M attended positions. Replay contributes to this count.
3. Unified adaptation. The final 1,000 updates combined MLM, NLI with teacher supervision, contrastive retrieval with teacher-vector alignment, and extractive QA replay. Domain MLM included Nawah and filtered Waqfeya text. Training manifest

Evaluation

Reviewed scores show Luna → Terra percentages on 100 shared Arabic cases per task. MLM credits exact answers and plausible completions accepted by the judge. NLI measures judge agreement. Retrieval measures whether the top results include a judged relevant sentence. All checkpoints used the same frozen probe protocol for NLI and retrieval.

Checkpoint	MLM parameters	Configured context	MLM top‑1 / top‑5	NLI agreement	Retrieval R@1 / R@5
Arabic ModernBERT Unified	32.79M	4,096	70/100 → 65/88	68 → 59	48/61 → 42/53
Waqf-AI ArabicBertHash v2	49.41M	2,048	65/93 → 48/76	37 → 41	31/39 → 27/35
AraBERTv2	135.26M	512	82/98 → 81/89	49 → 40	55/64 → 50/58
NAMAA AraModernBERT	149.59M	8,192	84/97 → 78/95	34 → 35	47/62 → 42/55
Arabic MiniBERT	11.58M	512	30/78 → 35/74	35 → 32	46/63 → 41/57

Unified uses 78% fewer MLM parameters than NAMAA. On these cases it comes within two points of NAMAA on sentence retrieval, while NAMAA leads by 13–14 points on MLM top‑1. Unified leads all three public baselines on NLI agreement after its own supervised task adaptation. The public checkpoints' encoder weights remained frozen.

Unified uses 34% fewer parameters for MLM than ArabicBertHash v2 and scores higher on all five reported metrics under both judges. ArabicBertHash v2 has a 16.25M-parameter encoder and a larger, untied MLM head. Its encoder is configured for 2,048 tokens, with a 1,024-token tokenizer limit in the release. Its NLI and retrieval scores come from the shared probes.

Exact-reference and native-head results

Protocol	Unified score
Whole-word MLM exact top‑1 / top‑5, 201 shared single-token-eligible cases	16.92% / 30.35%
Frozen NLI probe, 1,024 pairs	71.68% accuracy
Released native NLI head, same pairs	74.12% accuracy
Frozen retrieval probe, 512 queries	53.13% / 76.76% R@1 / R@5
Released native retrieval head, same queries	55.27% / 77.73% R@1 / R@5

Exact MLM scores require the original masked word. A blinded spot review found that each automated judge missed valid alternatives and accepted some invalid answers. The MLM scorer rejects blank and control-token candidates. Machine-readable evaluation

Scope and limitations

These diagnostics use short examples from previously used evaluation sets. Retrieval uses NLI-derived sentence pairs, and the NLI comparison includes the benefit of Unified’s prior supervised training. Long-document and extractive QA performance remain unmeasured here.

Files

File	Contents
model.safetensors	Encoder and tied MLM head
auxiliary_heads.safetensors	Trained NLI classifier, QA span head, and embedding projection
tokenizer.json, tokenizer_config.json	Released 48K tokenizer and special-token settings
unified_encoder.py, example.py	Optional local loader and runnable example
training_summary.json, evaluation/summary.json	Training and evaluation records
SHA256SUMS	Release file checksums

Architecture reference: ModernBERT. The Arabic checkpoint was initialized and trained from scratch.