Spaces:

T0KII
/

taMASRIBERTs

Running

App Files Files Community

taMASRIBERTs / README.md

T0KII

Update README.md

5000ca0 verified about 1 month ago

preview code

Raw

History Blame Contribute Delete

10.2 kB

metadata

title: Kalamna Egyptian Arabic Analyzer
emoji: 📊
colorFrom: green
colorTo: indigo
sdk: docker
app_port: 7860
app_file: app.py
pinned: false
license: mit
tags:
  - arabic
  - egyptian-arabic
  - nlp
  - sentiment-analysis
  - emotion-recognition
  - sarcasm-detection
  - franco-arabic
  - multitask
  - transformers
  - pytorch

Kalamna — Egyptian Arabic Analyzer (`masri-rf-space` v2)

Multi-task Egyptian Arabic NLP — emotion, sentiment, and sarcasm in a single forward pass. Supports pure Egyptian Arabic, Franco Arabizi (with digit substitutions 2/3/5/7/9), mixed Arabic+English, and full English sentences.

What's New in v2

	v1 (old Space)	v2 (this Space)
Architecture	MASRIHEADS + BIHEADS → XGBoost ensemble	Single taMASRIBERT deep-fusion model
BERT backbone	MASRIBERTv2 (standard vocab)	MASRIBERTv3 (expanded vocab with Franco tokens)
Inference passes	3× BERT + 3× RNN + XGBoost predict	1× BERT + 1× RNN, all three tasks at once
Franco handling	Phonetic transliteration before encoding	Native — tokenizer vocab covers Franco directly
Models loaded	3 PyTorch models + 3 XGBoost `.json` files	1 PyTorch model
Cold-start size	~1.3 GB (MASRIHEADS) + 22 MB (BIHEADS) + XGB	~1.1 GB (taMASRIBERT fused weights)

Architecture — `UnifiedMASRIHead` (taMASRIBERT, BERT-only variant)

Input text
    │
    ▼
dynamic_text_prep()
    ├── Pure English  ──────────────► NAMAA EN→EGY translation ─────┐
    ├── Pure Arabic / Franco / Mixed ─────────────────────────────── ▼
    │                                                           clean_text()
    │                                                           (preserves Latin/Franco digits)
    ▼                                                                │
MASRIBERTv3 Tokenizer ◄──────────────────────────────────────────────┘
    │
    ▼
MASRIBERTv3 Encoder
[CLS] vector (768-dim)
    │
    ├─────────────────────────────────────────┐
    │                                         │
    ▼                                         ▼
emotion_head                            sentiment_head
(768→256→8)                             (768→256→3)
    │                                         │
    └─────────────── sarcasm_head ───────────┘
                 (768→256→2)

Why BERT-only (no FastText+RNN)?

MASRIBERTv3 already captures sequence context and Franco patterns natively via its expanded tokenizer vocabulary
Removing the BiLSTM+BiGRU + FastText dependencies eliminates 3.3 GB cold-start and fasttext-wheel compilation issues on HF Spaces
BERT embeddings are sufficient for high performance on Egyptian Arabic

Latency improvement: ~200–400 ms (was 200–600 ms with RNN path)

Task labels

Task	Labels
Emotion (8)	`none`, `anger`, `joy`, `sadness`, `love`, `sympathy`, `surprise`, `fear`
Sentiment (3)	`negative`, `neutral`, `positive`
Sarcasm (2)	`not sarcastic`, `sarcastic`

Emotion threshold: if max(emotion_probs) < 0.45, the emotion is reported as none.

HuggingFace Models Used

Model	Role
`T0KII/taMASRIBERT`	Main model — tokenizer + `UnifiedMASRIHead` weights (`pytorch_model.bin`)
`T0KII/MASRIBERTv3`	BERT backbone inside taMASRIBERT (loaded by the model class)
`NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9`	EN→EGY translation (used only for pure-English input)

Removed: FastText arz embeddings. The BERT-only variant (v2.1+) relies on MASRIBERTv3's native handling of Franco + Arabic sequences via expanded tokenizer vocabulary.

Text Preprocessing Pipeline

dynamic_text_prep() routes text through one of two paths:

Input text
    │
    ├── has_arabic() OR has_franco_digits() OR mixed?
    │       └──► clean_text()  (diacritics removed, normalise alefs,
    │                           Latin/Franco digits preserved for tokenizer)
    │
    └── pure English (no Arabic, no Franco digits)?
            └──► NAMAA EN→EGY translation ──► clean_text()

clean_text() in v2 preserves Latin characters and Franco digits (2, 3, 5, 7, 9). This is a breaking change from v1 which stripped all Latin — MASRIBERTv3's expanded vocabulary handles Franco natively without transliteration.

Gradio Demo

The interactive demo accepts text in any of the four input modes:

Input	Example
Pure Egyptian Arabic	`بجد تسلم إيدكم، الشغل مية مية`
Franco Arabizi	`ana z3lan awy mn el service`
Mixed Arabic+English	`أنا عايز أعمل cancel للراوتر بتاعي`
Pure English	`how are you so bad at your job`

The "Pre-processed" box shows exactly what the model receives after dynamic_text_prep(), which is useful for debugging unexpected outputs.

REST API

The Space also exposes a FastAPI backend at the same URL, mounted at the root alongside Gradio.

`GET /health`

Returns server status and model version.

Response

{
  "status": "ok",
  "model": "taMASRIBERT",
  "version": "2.0.0"
}

`POST /detect`

Runs full inference and returns structured results.

Headers

Header	Required	Description
`X-API-Key`	Yes (if `EMOTION_API_KEY` secret is set)	API key for authentication
`Content-Type`	Yes	`application/json`

Request body

{
  "text": "يا حلاوة.. هو ده اللي فالحين فيه بس؟"
}

Response schema

{
  "emotion":    "string",   // one of the 8 emotion labels, or "none"
  "confidence": 0.0,        // float [0, 1] — probability of top emotion
  "sentiment":  "string",   // "negative" | "neutral" | "positive"
  "sarcasm": {
    "label": "string",      // "sarcastic" | "not sarcastic"
    "score": 0.0            // float [0, 1] — probability of top sarcasm label
  },
  "urgent":     false,      // true when emotion ∈ {anger, sadness, fear} AND sentiment == negative
  "latency_ms": 0.0,        // end-to-end inference time in milliseconds
  "source":     "string",   // "model" | "fallback" | "error_fallback"
  "cleaned":    "string"    // pre-processed text that was fed to the model (nullable)
}

Example — cURL

curl -X POST "https://<your-space-url>/detect" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your_secret_key" \
  -d '{"text": "ana z3lan awy mn el service"}'

Example — Python

import requests

resp = requests.post(
    "https://<your-space-url>/detect",
    headers={"X-API-Key": "your_secret_key"},
    json={"text": "يا حلاوة.. هو ده اللي فالحين فيه بس؟"},
)
data = resp.json()
print(data["emotion"], data["sarcasm"]["label"], data["urgent"])
# joy  sarcastic  False

Error / fallback behaviour

Condition	`source` value	Behaviour
Empty / blank text	`"fallback"`	Returns neutral defaults, `confidence: 0.0`
Text cleans to empty	`"fallback"`	Same as above
Model exception	`"error_fallback"`	Returns neutral defaults, logs exception server-side
Wrong / missing API key	—	`HTTP 401 Unauthorized`

Spaces Secrets

Set the following secrets in your Space settings (Settings → Repository secrets):

Secret name	Required	Description
`HF_TOKEN`	Yes	HuggingFace token with read access to `T0KII/*` repos
`EMOTION_API_KEY`	No	If set, all `/detect` requests must include `X-API-Key: <value>`. Leave unset to disable auth (open API).

Integration with Kalamna Pipeline (Pipecat)

The /detect endpoint is designed to slot directly into the Kalamna voice pipeline as the text-emotion signal. In kalamna/ai/emotion/services.py, the fusion logic delegates to this endpoint for text-based scores, while masri-audioV2 (V2 Sprinter: CNN-BiLSTM-Transformer on Mel spectrograms) provides the acoustic signal. The two scores are fused at inference time; hesitant and interested labels are permanently delegated to the text branch.

Performance Notes

Cold start: ~30–60 seconds on a free CPU Space (MASRIBERTv3 ~1.1 GB, NAMAA translator ~400 MB).
Inference latency: ~200–400 ms per request on CPU. Enable GPU hardware in Space settings to bring this below 50 ms.
Memory: ~2–3 GB RAM for all models loaded simultaneously (MASRIBERTv3 ~1.1 GB, NAMAA translator ~400 MB).

v2 improvements over v1:

No 3.3 GB FastText download
No fasttext-wheel C++ compilation (fixes build errors on HF Spaces)
2–3× faster cold start
Slightly better latency per inference
Simpler deployment with fewer dependencies

License

MIT — model weights and Space code. Training datasets (MASRISET, EYASE) retain their own licenses.

Citation / Credits

MASRIBERTv3: continued pre-training of UBC-NLP/MARBERTv2 on the MASRISET corpus (3.1M+ Egyptian Arabic rows), with expanded tokenizer vocabulary covering Franco Arabizi.
taMASRIBERT: UnifiedMASRIHead multi-task fine-tuning on Egyptian Arabic emotion, sentiment, and sarcasm datasets.
NAMAA translator: NAMAA-Space/masrawy-english-to-egyptian-arabic-translator-v2.9
FastText embeddings: facebook/fasttext-arz-vectors

Kalamna — Egyptian Arabic Analyzer (masri-rf-space v2)