---
language:
- ar
library_name: transformers
pipeline_tag: fill-mask
base_model:
- UBC-NLP/MARBERT
tags:
- arabic
- maghrebi-arabic
- darija
- arabizi
- code-switching
- algeria
- morocco
- tunisia
- libya
- bert
- pytorch
- masked-language-modeling
---

<img src="https://i.imgur.com/cwgkvqe.png"
alt="MagharibiBERT" width="35%" align="right"/>

# MagharibiBERT

**MagharibiBERT** is a regionally adapted Transformer language model
for **Maghrebi Arabic**.

It is initialized from `UBC-NLP/MARBERT` and further adapted using
Masked Language Modeling (MLM) on a large multi-source corpus of
Maghrebi text.

The model targets the linguistic and orthographic diversity of
North African online communication, including Arabic-script dialect,
Arabizi, spelling variation, and Arabic-French/English code-switching.

MagharibiBERT is developed by **NumidiaNLP**.

---

## Model architecture

| Property | Value |
|---|---:|
| Architecture | BERT |
| Parent model | `UBC-NLP/MARBERT` |
| Objective | Masked Language Modeling |
| Parameters | 162,942,880 |
| Transformer layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 |
| Vocabulary size | 100,000 |
| Maximum position embeddings | 512 |

---

## Regional continued pretraining

MagharibiBERT underwent additional regional continued pretraining on
a curated mixture of **17 corpora** covering Maghrebi Arabic and
North African social-media language.

The training mixture includes material associated with Algeria, Tunisia,
Morocco, Libya, and cross-regional Maghrebi sources, with explicit
coverage of Arabic script, Arabizi, and code-mixed text.

### Training configuration

- MLM sequence length: 128
- Optimizer steps: 120,000
- Effective batch size: 48
- Learning rate: 2e-5
- Warmup steps: 6,000
- Weight decay: 0.01
- Precision: BF16
- Random seed: 42

---

## How to use

### Fill-mask pipeline

```python
from transformers import pipeline

pipe = pipeline(
    "fill-mask",
    model="NumidiaNLP/MagharibiBERT"
)

results = pipe("واش راك [MASK]", top_k=5)

for result in results:
    print(result["token_str"], result["score"])
```

### Example output

Output produced by the released MagharibiBERT checkpoint:

```text
{"token": "دير", "score": 0.277943}
{"token": "تهدر", "score": 0.071477}
{"token": "تدير", "score": 0.062739}
{"token": "تقول", "score": 0.047855}
{"token": "تحكي", "score": 0.038360}
```

> Scores depend on the complete masked context. The example above is
> provided only to demonstrate model usage rather than as an evaluation result.

### Load the model directly

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = "NumidiaNLP/MagharibiBERT"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
```

---

## Intended use

MagharibiBERT is intended primarily for research involving Maghrebi Arabic,
including:

- hate and offensive language detection,
- sentiment analysis,
- dialect identification,
- Arabizi processing,
- code-switched text processing,
- social-media text classification,
- representation learning for Maghrebi Arabic.

---

## Intrinsic evaluation

On the fixed held-out MLM test set used during development, the released
checkpoint obtained:

| Metric | MagharibiBERT |
|---|---:|
| MLM loss | 2.8099 |
| Perplexity | 16.6080 |
| Masked-token accuracy | 52.42% |

Downstream evaluation against MARBERT and other Arabic language models
is being conducted separately.

---

## Limitations

Maghrebi Arabic is highly heterogeneous across countries, regions,
writing systems, platforms, and communities. Performance may therefore
vary across domains and linguistic varieties.

The model should not be interpreted as representing all North African
languages or all Maghrebi speakers equally.

---

## Citation

The citation for the accompanying MagharibiBERT paper will be added
following publication.

MagharibiBERT is initialized from MARBERT. Users should also cite the
original MARBERT work:

> Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2021).
> ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.
> ACL-IJCNLP 2021.

---

## Organization

**NumidiaNLP — Language Intelligence for North Africa**