arabert-openiti-911

Model Summary

arabert-openiti-911 is a domain-adapted Arabic language model for Classical Arabic historical prose. It was produced by continued pre-training of aubmindlab/bert-base-arabertv2 on a large corpus of primary Arabic texts from the OpenITI corpus (release 2025.1.9), filtered to authors with death dates up to AH 911 (approximately 1505 CE).

The model is intended to improve semantic similarity and text-reuse detection tasks on Classical Arabic chronicles and related genres, where standard Arabic NLP models trained on modern text perform poorly.

Work in progress

The model is a work in progress intended for testing and not for sharing beyond the project team.

---

Intended Uses

• Semantic alignment of Classical Arabic chronicle texts (e.g. paraphrase detection, text reuse, source detection)
• Embedding generation for Classical Arabic historical prose
• Drop-in replacement for AraBERT in any HuggingFace pipeline working with pre-modern Arabic texts
• Further fine-tuning on downstream Classical Arabic NLP tasks

Out-of-Scope Uses

• Modern Standard Arabic NLP tasks (use the original AraBERT for these)
• Non-Arabic text

---

Training Data

Source: OpenITI release 2025.1.9
Filter:

• Primary versions only (status = pri in OpenITI metadata)
• Arabic language only
• Author death date ≤ AH 911 (~1505 CE)
• Minimum 200 characters after cleaning

Result: 5,361 documents, ~993 million characters, ~287 million tokens

Normalisation (using the OpenITI Python library v0.1.6):

• OpenITI markup stripped (page references, milestone markers, structural headers, metadata header)
• deNoise: diacritics (harakat/tashkeel) removed
• normalize_ara_light: alif variants unified (أ إ آ ٱ → ا), alif maqsura normalised (ى → ي), Persian letters normalised; tā marbūṭa (ة) and hamzas preserved

---

Training Procedure

Parameter	Value
Base model	aubmindlab/bert-base-arabertv2
Task	Masked Language Modelling (MLM)
MLM probability	0.15
Sequence length	512 tokens
Epochs	2
Effective batch size	64 (16 per device × 4 gradient accumulation steps)
Learning rate	5e-5
LR scheduler	Linear with warmup (5% of steps)
Hardware	1× NVIDIA A100 SXM4 80GB
Training time	~2 hours
Final validation loss	1.6888
Final perplexity	~5.4

---

Evaluation

Validation perplexity of ~5.4 on a held-out sample of the same OpenITI corpus. For comparison, the base AraBERT model trained on modern Arabic would be expected to show substantially higher perplexity on Classical Arabic text.

Downstream evaluation on the Miskawayh–Ṭabarī text alignment task (comparing alignment quality against the base AraBERT baseline) is ongoing as an instance of the first of 15 alignment cases.

---

How to Use

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("SarahSav/arabert-openiti-911")
model = AutoModel.from_pretrained("SarahSav/arabert-openiti-911")

To use as a drop-in replacement for AraBERT in a HuggingFace pipeline, simply change the model name. The tokenizer and model architecture are identical to aubmindlab/bert-base-arabertv2.

---

Limitations

• Trained on written Classical Arabic prose — performance on poetry, documentary texts, or non-Arabic scripts may be limited
• The OpenITI corpus has uneven coverage across time periods and genres; some traditions are better represented than others
• Light normalisation was applied — texts with heavy dialectal or orthographic variation may not benefit as much

---

Citation

If you use this model, please cite the OpenITI corpus:

Nigst, Lorenz, Romanov, Maxim, Savant, Sarah Bowen, Seydi, Masoumeh, and Verkinderen, Peter. OpenITI: a Machine-Readable Corpus of Islamicate Texts (Version 2025.1.9). Zenodo, 2025.

---

Model Authors

Sarah Bowen Savant (Aga Khan University) with assistance from Claude Code (Anthropic).

License

Model weights inherit the license of the base model aubmindlab/bert-base-arabertv2. Training data is from OpenITI, which is openly licensed for research use.