MagharibiBERT

MagharibiBERT is a regionally adapted Transformer language model for Maghrebi Arabic.

It is initialized from UBC-NLP/MARBERT and further adapted using Masked Language Modeling (MLM) on a large multi-source corpus of Maghrebi text.

The model targets the linguistic and orthographic diversity of North African online communication, including Arabic-script dialect, Arabizi, spelling variation, and Arabic-French/English code-switching.

MagharibiBERT is developed by NumidiaNLP.

---

Model architecture

Property	Value
Architecture	BERT
Parent model	UBC-NLP/MARBERT
Objective	Masked Language Modeling
Parameters	162,942,880
Transformer layers	12
Hidden size	768
Attention heads	12
Vocabulary size	100,000
Maximum position embeddings	512

---

Regional continued pretraining

MagharibiBERT underwent additional regional continued pretraining on a curated mixture of 17 corpora covering Maghrebi Arabic and North African social-media language.

The training mixture includes material associated with Algeria, Tunisia, Morocco, Libya, and cross-regional Maghrebi sources, with explicit coverage of Arabic script, Arabizi, and code-mixed text.

Training configuration

• MLM sequence length: 128
• Optimizer steps: 120,000
• Effective batch size: 48
• Learning rate: 2e-5
• Warmup steps: 6,000
• Weight decay: 0.01
• Precision: BF16
• Random seed: 42

---

How to use

Fill-mask pipeline

from transformers import pipeline

pipe = pipeline(
    "fill-mask",
    model="NumidiaNLP/MagharibiBERT"
)

results = pipe("واش راك [MASK]", top_k=5)

for result in results:
    print(result["token_str"], result["score"])

Example output

Output produced by the released MagharibiBERT checkpoint:

{"token": "دير", "score": 0.277943}
{"token": "تهدر", "score": 0.071477}
{"token": "تدير", "score": 0.062739}
{"token": "تقول", "score": 0.047855}
{"token": "تحكي", "score": 0.038360}

Scores depend on the complete masked context. The example above is provided only to demonstrate model usage rather than as an evaluation result.

Load the model directly

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = "NumidiaNLP/MagharibiBERT"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)

---

Intended use

MagharibiBERT is intended primarily for research involving Maghrebi Arabic, including:

• hate and offensive language detection,
• sentiment analysis,
• dialect identification,
• Arabizi processing,
• code-switched text processing,
• social-media text classification,
• representation learning for Maghrebi Arabic.

---

Intrinsic evaluation

On the fixed held-out MLM test set used during development, the released checkpoint obtained:

Metric	MagharibiBERT
MLM loss	2.8099
Perplexity	16.6080
Masked-token accuracy	52.42%

Downstream evaluation against MARBERT and other Arabic language models is being conducted separately.

---

Limitations

Maghrebi Arabic is highly heterogeneous across countries, regions, writing systems, platforms, and communities. Performance may therefore vary across domains and linguistic varieties.

The model should not be interpreted as representing all North African languages or all Maghrebi speakers equally.

---

Citation

The citation for the accompanying MagharibiBERT paper will be added following publication.

MagharibiBERT is initialized from MARBERT. Users should also cite the original MARBERT work:

Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. ACL-IJCNLP 2021.

---

Organization

NumidiaNLP — Language Intelligence for North Africa