| ---
|
| language:
|
| - ar
|
| library_name: transformers
|
| pipeline_tag: fill-mask
|
| base_model:
|
| - UBC-NLP/MARBERT
|
| tags:
|
| - arabic
|
| - maghrebi-arabic
|
| - darija
|
| - arabizi
|
| - code-switching
|
| - algeria
|
| - morocco
|
| - tunisia
|
| - libya
|
| - bert
|
| - pytorch
|
| - masked-language-modeling
|
| ---
|
|
|
| <img src="https://i.imgur.com/cwgkvqe.png"
|
| alt="MagharibiBERT" width="35%" align="right"/>
|
|
|
| # MagharibiBERT
|
|
|
| **MagharibiBERT** is a regionally adapted Transformer language model
|
| for **Maghrebi Arabic**.
|
|
|
| It is initialized from `UBC-NLP/MARBERT` and further adapted using
|
| Masked Language Modeling (MLM) on a large multi-source corpus of
|
| Maghrebi text.
|
|
|
| The model targets the linguistic and orthographic diversity of
|
| North African online communication, including Arabic-script dialect,
|
| Arabizi, spelling variation, and Arabic-French/English code-switching.
|
|
|
| MagharibiBERT is developed by **NumidiaNLP**.
|
|
|
| ---
|
|
|
| ## Model architecture
|
|
|
| | Property | Value |
|
| |---|---:|
|
| | Architecture | BERT |
|
| | Parent model | `UBC-NLP/MARBERT` |
|
| | Objective | Masked Language Modeling |
|
| | Parameters | 162,942,880 |
|
| | Transformer layers | 12 |
|
| | Hidden size | 768 |
|
| | Attention heads | 12 |
|
| | Vocabulary size | 100,000 |
|
| | Maximum position embeddings | 512 |
|
|
|
| ---
|
|
|
| ## Regional continued pretraining
|
|
|
| MagharibiBERT underwent additional regional continued pretraining on
|
| a curated mixture of **17 corpora** covering Maghrebi Arabic and
|
| North African social-media language.
|
|
|
| The training mixture includes material associated with Algeria, Tunisia,
|
| Morocco, Libya, and cross-regional Maghrebi sources, with explicit
|
| coverage of Arabic script, Arabizi, and code-mixed text.
|
|
|
| ### Training configuration
|
|
|
| - MLM sequence length: 128
|
| - Optimizer steps: 120,000
|
| - Effective batch size: 48
|
| - Learning rate: 2e-5
|
| - Warmup steps: 6,000
|
| - Weight decay: 0.01
|
| - Precision: BF16
|
| - Random seed: 42
|
|
|
| ---
|
|
|
| ## How to use
|
|
|
| ### Fill-mask pipeline
|
|
|
| ```python
|
| from transformers import pipeline
|
|
|
| pipe = pipeline(
|
| "fill-mask",
|
| model="NumidiaNLP/MagharibiBERT"
|
| )
|
|
|
| results = pipe("واش راك [MASK]", top_k=5)
|
|
|
| for result in results:
|
| print(result["token_str"], result["score"])
|
| ```
|
|
|
| ### Example output
|
|
|
| Output produced by the released MagharibiBERT checkpoint:
|
|
|
| ```text
|
| {"token": "دير", "score": 0.277943}
|
| {"token": "تهدر", "score": 0.071477}
|
| {"token": "تدير", "score": 0.062739}
|
| {"token": "تقول", "score": 0.047855}
|
| {"token": "تحكي", "score": 0.038360}
|
| ```
|
|
|
| > Scores depend on the complete masked context. The example above is
|
| > provided only to demonstrate model usage rather than as an evaluation result.
|
|
|
| ### Load the model directly
|
|
|
| ```python
|
| from transformers import AutoTokenizer, AutoModelForMaskedLM
|
|
|
| model_name = "NumidiaNLP/MagharibiBERT"
|
|
|
| tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| model = AutoModelForMaskedLM.from_pretrained(model_name)
|
| ```
|
|
|
| ---
|
|
|
| ## Intended use
|
|
|
| MagharibiBERT is intended primarily for research involving Maghrebi Arabic,
|
| including:
|
|
|
| - hate and offensive language detection,
|
| - sentiment analysis,
|
| - dialect identification,
|
| - Arabizi processing,
|
| - code-switched text processing,
|
| - social-media text classification,
|
| - representation learning for Maghrebi Arabic.
|
|
|
| ---
|
|
|
| ## Intrinsic evaluation
|
|
|
| On the fixed held-out MLM test set used during development, the released
|
| checkpoint obtained:
|
|
|
| | Metric | MagharibiBERT |
|
| |---|---:|
|
| | MLM loss | 2.8099 |
|
| | Perplexity | 16.6080 |
|
| | Masked-token accuracy | 52.42% |
|
|
|
| Downstream evaluation against MARBERT and other Arabic language models
|
| is being conducted separately.
|
|
|
| ---
|
|
|
| ## Limitations
|
|
|
| Maghrebi Arabic is highly heterogeneous across countries, regions,
|
| writing systems, platforms, and communities. Performance may therefore
|
| vary across domains and linguistic varieties.
|
|
|
| The model should not be interpreted as representing all North African
|
| languages or all Maghrebi speakers equally.
|
|
|
| ---
|
|
|
| ## Citation
|
|
|
| The citation for the accompanying MagharibiBERT paper will be added
|
| following publication.
|
|
|
| MagharibiBERT is initialized from MARBERT. Users should also cite the
|
| original MARBERT work:
|
|
|
| > Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2021).
|
| > ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.
|
| > ACL-IJCNLP 2021.
|
|
|
| ---
|
|
|
| ## Organization
|
|
|
| **NumidiaNLP — Language Intelligence for North Africa**
|
|
|
|
|
