Create README.md

README.md

@@ -0,0 +1,159 @@
+---
+language:
+- ar
+license: unknown
+base_model:
+- T0KII/masribert
+- UBC-NLP/MARBERTv2
+tags:
+- arabic
+- egyptian-arabic
+- masked-language-modeling
+- bert
+- dialect
+- nlp
+pipeline_tag: fill-mask
+---
+
+# MasriBERT v2 — Egyptian Arabic Language Model
+
+MasriBERT v2 is a continued MLM pre-training of [MasriBERT v1](https://huggingface.co/T0KII/masribert) (itself built on [UBC-NLP/MARBERTv2](https://huggingface.co/UBC-NLP/MARBERTv2)) on a new, higher-quality Egyptian Arabic corpus emphasizing **conversational and dialogue register** — the primary register of customer-facing NLP applications.
+
+It is purpose-built as a backbone for downstream Egyptian Arabic NLP tasks including emotion classification, sarcasm detection, and sentiment analysis, with a specific focus on call-center and customer interaction language.
+
+## What Changed from v1
+
+| | MasriBERT v1 | MasriBERT v2 |
+|---|---|---|
+| Base model | UBC-NLP/MARBERTv2 | T0KII/masribert (v1) |
+| Training corpus | MASRISET (1.3M rows — tweets, reviews, news comments) | EFC + SFT Mixture (1.95M rows — forums, dialogue) |
+| Data register | Social media / news | Conversational / instructional dialogue |
+| Training steps | ~57,915 | ~21,500 (resumed from step 20,000) |
+| Final eval loss | 4.523 | **2.773** |
+| Final perplexity | 92.98 | **16.00** |
+| Training platform | Google Colab (A100) | Kaggle (T4 / P100) |
+
+The 5.8x perplexity improvement reflects both the richer training signal from conversational data and the cumulative MLM adaptation across all three training stages (MARBERTv2 → v1 → v2).
+
+## Training Corpus
+
+Two sources were used, targeting conversational Egyptian Arabic:
+
+**faisalq/EFC-mini — Egyptian Forums Corpus**
+Forum posts and comments from Egyptian Arabic internet forums. Long-form conversational text capturing how Egyptians write when explaining problems, complaining, and asking questions — closely mirroring customer behavior.
+
+**MBZUAI-Paris/Egyptian-SFT-Mixture — Egyptian Dialogue**
+Supervised fine-tuning dialogue data in Egyptian Arabic — instruction/response pairs curated specifically for Egyptian dialect LLM training. Chat formatting was stripped to raw text before training.
+
+Both sources were deduplicated (MD5 hash), shuffled with seed 42, and minimum 5-word samples enforced post-cleaning.
+
+After deduplication: **1,946,195 rows → 1,868,414 chunks of 64 tokens**
+
+## Text Cleaning Pipeline
+
+Same normalization as v1, applied uniformly:
+
+- Removed URLs, email addresses, @mentions, and hashtag symbols
+- Alef normalization: إأآا → ا
+- Alef maqsura: ى → ي
+- Hamza variants: ؤ, ئ → ء
+- Removed all Arabic tashkeel (diacritics)
+- Capped repeated characters at 2 (e.g. هههههه → هه)
+- Removed English characters
+- Preserved emojis (MARBERTv2 has native emoji embeddings from tweet pretraining)
+- Minimum 5 words per sample enforced post-cleaning
+
+## Training Configuration
+
+| Hyperparameter | Value |
+|---|---|
+| Block size | 64 tokens |
+| MLM probability | 0.20 (20%) |
+| Masking strategy | Token-level (whole word masking disabled — tokenizer incompatibility) |
+| Peak learning rate | 2e-5 |
+| Resume learning rate | 6.16e-6 (corrected for linear decay at step 20,000) |
+| LR schedule | Linear decay, no warmup on resume |
+| Batch size | 64 per device |
+| Gradient accumulation | 2 steps (effective batch = 128) |
+| Weight decay | 0.01 |
+| Precision | FP16 |
+| Eval / Save interval | Every 500 steps |
+| Early stopping patience | 3 evaluations |
+| Train blocks | 1,849,729 |
+| Eval blocks | 18,685 |
+
+Training was conducted on Kaggle (NVIDIA T4 / P100) across 2 epochs. Due to Kaggle's 12-hour session limit, training was split across two sessions with checkpoint resumption via HuggingFace Hub.
+
+## Eval Loss Curve
+
+| Step | Eval Loss |
+|---|---|
+| 500 | 3.830 |
+| 1,000 | 3.599 |
+| 2,000 | 3.336 |
+| 5,000 | 3.066 |
+| 8,500 | 2.945 |
+| 20,500 | 2.773 |
+| 21,000 | 2.783 |
+| **21,500** | **2.773 ← best** |
+
+## Usage
+
+```python
+from transformers import pipeline
+
+unmasker = pipeline("fill-mask", model="T0KII/MASRIBERTv2", top_k=3)
+
+results = unmasker("انا مش راضي عن الخدمة دي [MASK] بجد.")
+for r in results:
+    print(r['token_str'], round(r['score'], 4))
+```
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("T0KII/MASRIBERTv2")
+model = AutoModelForMaskedLM.from_pretrained("T0KII/MASRIBERTv2")
+```
+
+For downstream classification tasks (emotion, sentiment, sarcasm):
+
+```python
+from transformers import AutoModel
+
+encoder = AutoModel.from_pretrained("T0KII/MASRIBERTv2")
+# Attach your classification head on top of encoder.pooler_output or encoder.last_hidden_state
+```
+
+## Known Warnings
+
+**LayerNorm naming:** Loading this model produces warnings about missing/unexpected keys (`LayerNorm.weight` / `LayerNorm.bias` vs `LayerNorm.gamma` / `LayerNorm.beta`). This is a known naming compatibility issue between older MARBERTv2 checkpoint conventions and newer Transformers versions. The weights are correctly loaded — the warning is cosmetic and can be safely ignored.
+
+## Intended Downstream Tasks
+
+This model is the backbone for the following tasks in the **Kalamna** Egyptian Arabic AI call-center pipeline:
+
+- **Emotion Classification** — Multi-class emotion detection (anger, joy, sadness, fear, surprise, love, sympathy, neutral)
+- **Sarcasm Detection** — Egyptian Arabic sarcasm including culturally-specific patterns (religious phrase inversion, hyperbolic complaint, dialectal irony)
+- **Sentiment Analysis** — Positive / Negative / Neutral classification for customer interaction data
+
+## Model Lineage
+
+```
+UBC-NLP/MARBERTv2
+    └── T0KII/masribert  (v1 — MLM on MASRISET, 57K steps)
+            └── T0KII/MASRIBERTv2  (v2 — MLM on EFC + SFT, 21.5K steps)
+```
+
+## Citation
+
+If you use this model, please cite the original MARBERTv2 paper:
+
+```bibtex
+@inproceedings{abdul-mageed-etal-2021-arbert,
+    title = "{ARBERT} {\&} {MARBERT}: Deep Bidirectional Transformers for {A}rabic",
+    author = "Abdul-Mageed, Muhammad and Elmadany, AbdelRahim and Nagoudi, El Moatez Billah",
+    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics",
+    year = "2021"
+}
+```