--- title: Kalamna Egyptian Arabic Analyzer emoji: πŸ“Š colorFrom: green colorTo: indigo sdk: docker app_port: 7860 app_file: app.py pinned: false license: mit tags: - arabic - egyptian-arabic - nlp - sentiment-analysis - emotion-recognition - sarcasm-detection - franco-arabic - multitask - transformers - pytorch --- # Kalamna β€” Egyptian Arabic Analyzer (`masri-rf-space` v2) > **Multi-task Egyptian Arabic NLP** β€” emotion, sentiment, and sarcasm in a single forward pass. > Supports pure Egyptian Arabic, Franco Arabizi (with digit substitutions 2/3/5/7/9), mixed Arabic+English, and full English sentences. --- ## What's New in v2 | | v1 (old Space) | **v2 (this Space)** | |---|---|---| | **Architecture** | MASRIHEADS + BIHEADS β†’ XGBoost ensemble | **Single taMASRIBERT** deep-fusion model | | **BERT backbone** | MASRIBERTv2 (standard vocab) | **MASRIBERTv3** (expanded vocab with Franco tokens) | | **Inference passes** | 3Γ— BERT + 3Γ— RNN + XGBoost predict | **1Γ— BERT + 1Γ— RNN**, all three tasks at once | | **Franco handling** | Phonetic transliteration before encoding | **Native** β€” tokenizer vocab covers Franco directly | | **Models loaded** | 3 PyTorch models + 3 XGBoost `.json` files | **1 PyTorch model** | | **Cold-start size** | ~1.3 GB (MASRIHEADS) + 22 MB (BIHEADS) + XGB | **~1.1 GB** (taMASRIBERT fused weights) | --- ## Architecture β€” `UnifiedMASRIHead` (taMASRIBERT, BERT-only variant) ``` Input text β”‚ β–Ό dynamic_text_prep() β”œβ”€β”€ Pure English ──────────────► NAMAA ENβ†’EGY translation ─────┐ β”œβ”€β”€ Pure Arabic / Franco / Mixed ─────────────────────────────── β–Ό β”‚ clean_text() β”‚ (preserves Latin/Franco digits) β–Ό β”‚ MASRIBERTv3 Tokenizer β—„β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β–Ό MASRIBERTv3 Encoder [CLS] vector (768-dim) β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β–Ό β–Ό emotion_head sentiment_head (768β†’256β†’8) (768β†’256β†’3) β”‚ β”‚ └─────────────── sarcasm_head β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ (768β†’256β†’2) ``` **Why BERT-only (no FastText+RNN)?** - MASRIBERTv3 already captures sequence context and Franco patterns natively via its expanded tokenizer vocabulary - Removing the BiLSTM+BiGRU + FastText dependencies eliminates 3.3 GB cold-start and fasttext-wheel compilation issues on HF Spaces - BERT embeddings are sufficient for high performance on Egyptian Arabic **Latency improvement:** ~200–400 ms (was 200–600 ms with RNN path) ### Task labels | Task | Labels | |---|---| | Emotion (8) | `none`, `anger`, `joy`, `sadness`, `love`, `sympathy`, `surprise`, `fear` | | Sentiment (3) | `negative`, `neutral`, `positive` | | Sarcasm (2) | `not sarcastic`, `sarcastic` | > **Emotion threshold**: if `max(emotion_probs) < 0.45`, the emotion is reported as `none`. --- ## HuggingFace Models Used | Model | Role | |---|---| | [`T0KII/taMASRIBERT`](https://huggingface.co/T0KII/taMASRIBERT) | Main model β€” tokenizer + `UnifiedMASRIHead` weights (`pytorch_model.bin`) | | [`T0KII/MASRIBERTv3`](https://huggingface.co/T0KII/MASRIBERTv3) | BERT backbone inside taMASRIBERT (loaded by the model class) | | [`NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9`](https://huggingface.co/NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9) | ENβ†’EGY translation (used only for pure-English input) | > **Removed:** FastText arz embeddings. The BERT-only variant (v2.1+) relies on MASRIBERTv3's native handling of Franco + Arabic sequences via expanded tokenizer vocabulary. --- ## Text Preprocessing Pipeline `dynamic_text_prep()` routes text through one of two paths: ``` Input text β”‚ β”œβ”€β”€ has_arabic() OR has_franco_digits() OR mixed? β”‚ └──► clean_text() (diacritics removed, normalise alefs, β”‚ Latin/Franco digits preserved for tokenizer) β”‚ └── pure English (no Arabic, no Franco digits)? └──► NAMAA ENβ†’EGY translation ──► clean_text() ``` `clean_text()` in v2 **preserves Latin characters and Franco digits** (2, 3, 5, 7, 9). This is a breaking change from v1 which stripped all Latin β€” MASRIBERTv3's expanded vocabulary handles Franco natively without transliteration. --- ## Gradio Demo The interactive demo accepts text in any of the four input modes: | Input | Example | |---|---| | Pure Egyptian Arabic | `Ψ¨Ψ¬Ψ― ΨͺΨ³Ω„Ω… Ψ₯ΩŠΨ―ΩƒΩ…ΨŒ Ψ§Ω„Ψ΄ΨΊΩ„ Ω…ΩŠΨ© Ω…ΩŠΨ©` | | Franco Arabizi | `ana z3lan awy mn el service` | | Mixed Arabic+English | `Ψ£Ω†Ψ§ عايز Ψ£ΨΉΩ…Ω„ cancel Ω„Ω„Ψ±Ψ§ΩˆΨͺΨ± Ψ¨Ψͺاعي` | | Pure English | `how are you so bad at your job` | The **"Pre-processed"** box shows exactly what the model receives after `dynamic_text_prep()`, which is useful for debugging unexpected outputs. --- ## REST API The Space also exposes a FastAPI backend at the same URL, mounted at the root alongside Gradio. ### `GET /health` Returns server status and model version. **Response** ```json { "status": "ok", "model": "taMASRIBERT", "version": "2.0.0" } ``` --- ### `POST /detect` Runs full inference and returns structured results. **Headers** | Header | Required | Description | |---|---|---| | `X-API-Key` | Yes (if `EMOTION_API_KEY` secret is set) | API key for authentication | | `Content-Type` | Yes | `application/json` | **Request body** ```json { "text": "يا Ψ­Ω„Ψ§ΩˆΨ©.. Ω‡Ωˆ Ψ―Ω‡ Ψ§Ω„Ω„ΩŠ ΩΨ§Ω„Ψ­ΩŠΩ† ΩΩŠΩ‡ بس؟" } ``` **Response schema** ```json { "emotion": "string", // one of the 8 emotion labels, or "none" "confidence": 0.0, // float [0, 1] β€” probability of top emotion "sentiment": "string", // "negative" | "neutral" | "positive" "sarcasm": { "label": "string", // "sarcastic" | "not sarcastic" "score": 0.0 // float [0, 1] β€” probability of top sarcasm label }, "urgent": false, // true when emotion ∈ {anger, sadness, fear} AND sentiment == negative "latency_ms": 0.0, // end-to-end inference time in milliseconds "source": "string", // "model" | "fallback" | "error_fallback" "cleaned": "string" // pre-processed text that was fed to the model (nullable) } ``` **Example β€” cURL** ```bash curl -X POST "https:///detect" \ -H "Content-Type: application/json" \ -H "X-API-Key: your_secret_key" \ -d '{"text": "ana z3lan awy mn el service"}' ``` **Example β€” Python** ```python import requests resp = requests.post( "https:///detect", headers={"X-API-Key": "your_secret_key"}, json={"text": "يا Ψ­Ω„Ψ§ΩˆΨ©.. Ω‡Ωˆ Ψ―Ω‡ Ψ§Ω„Ω„ΩŠ ΩΨ§Ω„Ψ­ΩŠΩ† ΩΩŠΩ‡ بس؟"}, ) data = resp.json() print(data["emotion"], data["sarcasm"]["label"], data["urgent"]) # joy sarcastic False ``` **Error / fallback behaviour** | Condition | `source` value | Behaviour | |---|---|---| | Empty / blank text | `"fallback"` | Returns neutral defaults, `confidence: 0.0` | | Text cleans to empty | `"fallback"` | Same as above | | Model exception | `"error_fallback"` | Returns neutral defaults, logs exception server-side | | Wrong / missing API key | β€” | `HTTP 401 Unauthorized` | --- ## Spaces Secrets Set the following secrets in your Space settings (`Settings β†’ Repository secrets`): | Secret name | Required | Description | |---|---|---| | `HF_TOKEN` | **Yes** | HuggingFace token with read access to `T0KII/*` repos | | `EMOTION_API_KEY` | No | If set, all `/detect` requests must include `X-API-Key: `. Leave unset to disable auth (open API). | --- ## Integration with Kalamna Pipeline (Pipecat) The `/detect` endpoint is designed to slot directly into the Kalamna voice pipeline as the text-emotion signal. In `kalamna/ai/emotion/services.py`, the fusion logic delegates to this endpoint for text-based scores, while `masri-audioV2` (V2 Sprinter: CNN-BiLSTM-Transformer on Mel spectrograms) provides the acoustic signal. The two scores are fused at inference time; `hesitant` and `interested` labels are permanently delegated to the text branch. --- ## Performance Notes - **Cold start**: ~30–60 seconds on a free CPU Space (MASRIBERTv3 ~1.1 GB, NAMAA translator ~400 MB). - **Inference latency**: ~200–400 ms per request on CPU. Enable GPU hardware in Space settings to bring this below 50 ms. - **Memory**: ~2–3 GB RAM for all models loaded simultaneously (MASRIBERTv3 ~1.1 GB, NAMAA translator ~400 MB). **v2 improvements over v1:** - No 3.3 GB FastText download - No fasttext-wheel C++ compilation (fixes build errors on HF Spaces) - 2–3Γ— faster cold start - Slightly better latency per inference - Simpler deployment with fewer dependencies --- ## License MIT β€” model weights and Space code. Training datasets (MASRISET, EYASE) retain their own licenses. --- ## Citation / Credits - **MASRIBERTv3**: continued pre-training of [UBC-NLP/MARBERTv2](https://huggingface.co/UBC-NLP/MARBERTv2) on the MASRISET corpus (3.1M+ Egyptian Arabic rows), with expanded tokenizer vocabulary covering Franco Arabizi. - **taMASRIBERT**: `UnifiedMASRIHead` multi-task fine-tuning on Egyptian Arabic emotion, sentiment, and sarcasm datasets. - **NAMAA translator**: [`NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9`](https://huggingface.co/NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9) - **FastText embeddings**: [`facebook/fasttext-arz-vectors`](https://huggingface.co/facebook/fasttext-arz-vectors)