README.md

---
title: Kalamna Egyptian Arabic Analyzer
emoji: 📊
colorFrom: green
colorTo: indigo
sdk: docker
app_port: 7860
app_file: app.py
pinned: false
license: mit
tags:
- arabic
- egyptian-arabic
- nlp
- sentiment-analysis
- emotion-recognition
- sarcasm-detection
- franco-arabic
- multitask
- transformers
- pytorch
---

# Kalamna — Egyptian Arabic Analyzer (`masri-rf-space` v2)

> **Multi-task Egyptian Arabic NLP** — emotion, sentiment, and sarcasm in a single forward pass.
> Supports pure Egyptian Arabic, Franco Arabizi (with digit substitutions 2/3/5/7/9), mixed Arabic+English, and full English sentences.

---

## What's New in v2

| | v1 (old Space) | **v2 (this Space)** |
|---|---|---|
| **Architecture** | MASRIHEADS + BIHEADS → XGBoost ensemble | **Single taMASRIBERT** deep-fusion model |
| **BERT backbone** | MASRIBERTv2 (standard vocab) | **MASRIBERTv3** (expanded vocab with Franco tokens) |
| **Inference passes** | 3× BERT + 3× RNN + XGBoost predict | **1× BERT + 1× RNN**, all three tasks at once |
| **Franco handling** | Phonetic transliteration before encoding | **Native** — tokenizer vocab covers Franco directly |
| **Models loaded** | 3 PyTorch models + 3 XGBoost `.json` files | **1 PyTorch model** |
| **Cold-start size** | ~1.3 GB (MASRIHEADS) + 22 MB (BIHEADS) + XGB | **~1.1 GB** (taMASRIBERT fused weights) |

---

## Architecture — `UnifiedMASRIHead` (taMASRIBERT, BERT-only variant)

```
Input text
    │
    ▼
dynamic_text_prep()
    ├── Pure English  ──────────────► NAMAA EN→EGY translation ─────┐
    ├── Pure Arabic / Franco / Mixed ─────────────────────────────── ▼
    │                                                           clean_text()
    │                                                           (preserves Latin/Franco digits)
    ▼                                                                │
MASRIBERTv3 Tokenizer ◄──────────────────────────────────────────────┘
    │
    ▼
MASRIBERTv3 Encoder
[CLS] vector (768-dim)
    │
    ├─────────────────────────────────────────┐
    │                                         │
    ▼                                         ▼
emotion_head                            sentiment_head
(768→256→8)                             (768→256→3)
    │                                         │
    └─────────────── sarcasm_head ───────────┘
                 (768→256→2)
```

**Why BERT-only (no FastText+RNN)?**
- MASRIBERTv3 already captures sequence context and Franco patterns natively via its expanded tokenizer vocabulary
- Removing the BiLSTM+BiGRU + FastText dependencies eliminates 3.3 GB cold-start and fasttext-wheel compilation issues on HF Spaces
- BERT embeddings are sufficient for high performance on Egyptian Arabic

**Latency improvement:** ~200–400 ms (was 200–600 ms with RNN path)

### Task labels

| Task | Labels |
|---|---|
| Emotion (8) | `none`, `anger`, `joy`, `sadness`, `love`, `sympathy`, `surprise`, `fear` |
| Sentiment (3) | `negative`, `neutral`, `positive` |
| Sarcasm (2) | `not sarcastic`, `sarcastic` |

> **Emotion threshold**: if `max(emotion_probs) < 0.45`, the emotion is reported as `none`.

---

## HuggingFace Models Used

| Model | Role |
|---|---|
| [`T0KII/taMASRIBERT`](https://huggingface.co/T0KII/taMASRIBERT) | Main model — tokenizer + `UnifiedMASRIHead` weights (`pytorch_model.bin`) |
| [`T0KII/MASRIBERTv3`](https://huggingface.co/T0KII/MASRIBERTv3) | BERT backbone inside taMASRIBERT (loaded by the model class) |
| [`NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9`](https://huggingface.co/NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9) | EN→EGY translation (used only for pure-English input) |

> **Removed:** FastText arz embeddings. The BERT-only variant (v2.1+) relies on MASRIBERTv3's native handling of Franco + Arabic sequences via expanded tokenizer vocabulary.

---

## Text Preprocessing Pipeline

`dynamic_text_prep()` routes text through one of two paths:

```
Input text
    │
    ├── has_arabic() OR has_franco_digits() OR mixed?
    │       └──► clean_text()  (diacritics removed, normalise alefs,
    │                           Latin/Franco digits preserved for tokenizer)
    │
    └── pure English (no Arabic, no Franco digits)?
            └──► NAMAA EN→EGY translation ──► clean_text()
```

`clean_text()` in v2 **preserves Latin characters and Franco digits** (2, 3, 5, 7, 9).
This is a breaking change from v1 which stripped all Latin — MASRIBERTv3's expanded vocabulary handles Franco natively without transliteration.

---

## Gradio Demo

The interactive demo accepts text in any of the four input modes:

| Input | Example |
|---|---|
| Pure Egyptian Arabic | `بجد تسلم إيدكم، الشغل مية مية` |
| Franco Arabizi | `ana z3lan awy mn el service` |
| Mixed Arabic+English | `أنا عايز أعمل cancel للراوتر بتاعي` |
| Pure English | `how are you so bad at your job` |

The **"Pre-processed"** box shows exactly what the model receives after `dynamic_text_prep()`, which is useful for debugging unexpected outputs.

---

## REST API

The Space also exposes a FastAPI backend at the same URL, mounted at the root alongside Gradio.

### `GET /health`

Returns server status and model version.

**Response**
```json
{
  "status": "ok",
  "model": "taMASRIBERT",
  "version": "2.0.0"
}
```

---

### `POST /detect`

Runs full inference and returns structured results.

**Headers**

| Header | Required | Description |
|---|---|---|
| `X-API-Key` | Yes (if `EMOTION_API_KEY` secret is set) | API key for authentication |
| `Content-Type` | Yes | `application/json` |

**Request body**
```json
{
  "text": "يا حلاوة.. هو ده اللي فالحين فيه بس؟"
}
```

**Response schema**
```json
{
  "emotion":    "string",   // one of the 8 emotion labels, or "none"
  "confidence": 0.0,        // float [0, 1] — probability of top emotion
  "sentiment":  "string",   // "negative" | "neutral" | "positive"
  "sarcasm": {
    "label": "string",      // "sarcastic" | "not sarcastic"
    "score": 0.0            // float [0, 1] — probability of top sarcasm label
  },
  "urgent":     false,      // true when emotion ∈ {anger, sadness, fear} AND sentiment == negative
  "latency_ms": 0.0,        // end-to-end inference time in milliseconds
  "source":     "string",   // "model" | "fallback" | "error_fallback"
  "cleaned":    "string"    // pre-processed text that was fed to the model (nullable)
}
```

**Example — cURL**
```bash
curl -X POST "https://<your-space-url>/detect" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your_secret_key" \
  -d '{"text": "ana z3lan awy mn el service"}'
```

**Example — Python**
```python
import requests

resp = requests.post(
    "https://<your-space-url>/detect",
    headers={"X-API-Key": "your_secret_key"},
    json={"text": "يا حلاوة.. هو ده اللي فالحين فيه بس؟"},
)
data = resp.json()
print(data["emotion"], data["sarcasm"]["label"], data["urgent"])
# joy  sarcastic  False
```

**Error / fallback behaviour**

| Condition | `source` value | Behaviour |
|---|---|---|
| Empty / blank text | `"fallback"` | Returns neutral defaults, `confidence: 0.0` |
| Text cleans to empty | `"fallback"` | Same as above |
| Model exception | `"error_fallback"` | Returns neutral defaults, logs exception server-side |
| Wrong / missing API key | — | `HTTP 401 Unauthorized` |

---

## Spaces Secrets

Set the following secrets in your Space settings (`Settings → Repository secrets`):

| Secret name | Required | Description |
|---|---|---|
| `HF_TOKEN` | **Yes** | HuggingFace token with read access to `T0KII/*` repos |
| `EMOTION_API_KEY` | No | If set, all `/detect` requests must include `X-API-Key: <value>`. Leave unset to disable auth (open API). |

---

## Integration with Kalamna Pipeline (Pipecat)

The `/detect` endpoint is designed to slot directly into the Kalamna voice pipeline as the text-emotion signal. In `kalamna/ai/emotion/services.py`, the fusion logic delegates to this endpoint for text-based scores, while `masri-audioV2` (V2 Sprinter: CNN-BiLSTM-Transformer on Mel spectrograms) provides the acoustic signal. The two scores are fused at inference time; `hesitant` and `interested` labels are permanently delegated to the text branch.

---

## Performance Notes

- **Cold start**: ~30–60 seconds on a free CPU Space (MASRIBERTv3 ~1.1 GB, NAMAA translator ~400 MB).
- **Inference latency**: ~200–400 ms per request on CPU. Enable GPU hardware in Space settings to bring this below 50 ms.
- **Memory**: ~2–3 GB RAM for all models loaded simultaneously (MASRIBERTv3 ~1.1 GB, NAMAA translator ~400 MB).

**v2 improvements over v1:**
- No 3.3 GB FastText download
- No fasttext-wheel C++ compilation (fixes build errors on HF Spaces)
- 2–3× faster cold start
- Slightly better latency per inference
- Simpler deployment with fewer dependencies

---

## License

MIT — model weights and Space code. Training datasets (MASRISET, EYASE) retain their own licenses.

---

## Citation / Credits

- **MASRIBERTv3**: continued pre-training of [UBC-NLP/MARBERTv2](https://huggingface.co/UBC-NLP/MARBERTv2) on the MASRISET corpus (3.1M+ Egyptian Arabic rows), with expanded tokenizer vocabulary covering Franco Arabizi.
- **taMASRIBERT**: `UnifiedMASRIHead` multi-task fine-tuning on Egyptian Arabic emotion, sentiment, and sarcasm datasets.
- **NAMAA translator**: [`NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9`](https://huggingface.co/NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9)
- **FastText embeddings**: [`facebook/fasttext-arz-vectors`](https://huggingface.co/facebook/fasttext-arz-vectors)