README.md

---
language:
- ar
- en
- fr
license: mit
tags:
- aspect-based-sentiment-analysis
- absa
- arabic-nlp
- arabic-dialects
- franco-arabic
- ensemble
- marbert
- arabert
- camelbert
- xlm-roberta
- text-classification
- multi-label-classification
pipeline_tag: text-classification
---

# DeepX-AI-Hackathon-ABSA

**A multi-dialect, multilingual Aspect-Based Sentiment Analysis (ABSA) system for Arabic reviews (Modern Standard Arabic, dialectal Arabic, Franco-Arabic, English, and French).**

An **ensemble** of 4 independently fine-tuned transformer backbones, combined with a language-aware routing system, purpose-built for real-world user reviews (e.g. Google Maps, Play Store, food delivery apps) written in mixed scripts and dialects — including Egyptian, Gulf, and Levantine Arabic, and Franco-Arabic (Arabic written with Latin letters and digits).

---

## 📌 Overview

This project is a complete **two-stage pipeline** that extracts, for every review:

1. **Aspects** mentioned in the review (e.g. food, service, price, cleanliness…).
2. **Sentiment** associated with each individual aspect (positive / negative / neutral).

The final output for each review is a set of **(aspect, sentiment) tuples**:

```json
{
  "review_id": 12345,
  "aspects": [
    {"aspect": "food", "sentiment": "positive"},
    {"aspect": "service", "sentiment": "negative"}
  ]
}
```

### Why an ensemble of 4 models?

Real-world Arabic reviews are far from uniform — the same user might write in Modern Standard Arabic, a regional dialect, Franco-Arabic (Latin letters + digits standing in for Arabic sounds), or plain English/French. To handle this diversity, four different backbones were fine-tuned, each with a different strength, and their outputs are combined via a **language-routed, weighted soft-voting ensemble**.

---

## 🧠 Architecture

### Stage 1 — Aspect Detection (Multi-Label Classification)

For every review text, the model predicts which of 9 aspect categories are mentioned:

| Aspect | Description |
|---|---|
| `food` | Food and taste |
| `service` | Service and staff |
| `price` | Price and cost |
| `cleanliness` | Cleanliness and hygiene |
| `delivery` | Delivery and shipping |
| `ambiance` | Ambiance and atmosphere |
| `app_experience` | App or website experience |
| `general` | Overall experience |
| `none` | No clear aspect (rating-only comment) |

**Model architecture:**
```
Review text → Backbone (BERT / RoBERTa) → [CLS] embedding
                                                ↓
                Metadata Fusion (concatenated with text embedding):
                - Star rating       → Embedding
                - Business category → Embedding
                - Platform (Google Maps / Play Store) → Embedding
                                                ↓
                        MLP + LayerNorm + Dropout
                                                ↓
                Linear layer → 9 logits (multi-label)
                                                ↓
                    Sigmoid + per-class decision threshold
```

### Stage 2 — Sentiment Classification (per aspect)

Once the aspects are extracted, for every (review, aspect) pair an **aspect-aware question** is built:

- Arabic backbones: `[نجوم=X] <text> [SEP] ما رايك فى <aspect> ؟`
- Latin-script backbone (XLM-R): `[stars=X] <text> [SEP] what about the <aspect> ?`

The pair is then classified into one of 3 classes: `positive` / `negative` / `neutral`.

```
(Text + aspect question) → Backbone → [CLS]
                                          ↓
                Metadata Fusion + Aspect Embedding
                                          ↓
                    MLP (128) → 3 logits
                                          ↓
                                    Softmax
```

---

## 🔀 Language Router

Before any modeling, every text is automatically routed into one of the following categories using regex heuristics and marker-word dictionaries:

| Route | Description |
|---|---|
| `arabic` | Predominantly Arabic script (Arabic-character ratio > 70%) |
| `mixed` | A mix of Arabic and Latin script |
| `franco` | Franco-Arabic (e.g. `momtaz awy`, `7elw giddan`) |
| `latin` | Fully English or French |
| `other_script` | Other scripts (Chinese, Korean, Russian, etc.) |
| `empty_rating_only` | Empty / too short to classify → falls back to a star-rating rule |

> **Important fix (V2):** Very short texts (1–2 words) without a clear sentiment word (e.g. "Up" or "Shady") are *not* routed to Franco — they're routed to `empty_rating_only` instead, since model predictions on such short strings are less reliable than a simple star-based rule.

### Which backbones handle which route?

```python
ROUTE_BACKBONES = {
    'arabic':  ['marbert', 'arabert', 'camelbert', 'xlmr'],
    'mixed':   ['marbert', 'camelbert', 'xlmr'],
    'franco':  ['marbert', 'arabert', 'camelbert', 'xlmr'],
    'latin':   ['xlmr'],
    'other_script': ['xlmr'],
    'empty_rating_only': [],   # handled entirely by a hand-written star-based rule
}
```

---

## 🏗️ Backbones

| # | Name | Hugging Face ID | Strength | Aspect F1 (val) | Sentiment F1 (val) |
|---|---|---|---|---|---|
| 1 | **MARBERT v2** | `UBC-NLP/MARBERTv2` | Arabic dialects / Twitter-style text | 0.8707 | 0.7811 |
| 2 | **AraBERT (Twitter)** | `aubmindlab/bert-base-arabertv02-twitter` | Best overall performance on MSA/dialectal text | **0.9371** | **0.8144** |
| 3 | **CAMeLBERT-DA** | `CAMeL-Lab/bert-base-arabic-camelbert-da` | Specialized in dialectal Arabic | 0.9371 | 0.8092 |
| 4 | **XLM-RoBERTa base** | `FacebookAI/xlm-roberta-base` | Multilingual — English, French, Franco-Arabic | 0.8025 | 0.7533 |

**Training data notes:**
- `MARBERT` and `XLM-R` were trained on an **expanded dataset**: real Arabic reviews + **synthetically generated Franco-Arabic** variants (see below).
- `AraBERT` and `CAMeLBERT` were trained on real Arabic data only.

---

## 🎲 Synthetic Franco-Arabic Data Augmentation

Since real Franco-Arabic examples are scarce in the training set, an **automatic Arabic → Franco-Arabic converter** was built, based on:

1. A **common-word dictionary** (60+ frequent Arabic words/phrases mapped to their typical Franco-Arabic spellings, e.g. `ممتاز → mumtaz/momtaz`, `مش → mesh/mish`).
2. A **character-level transliteration map** for the remaining words (e.g. `ح → 7`, `ع → 3`, `غ → 8`).
3. Controlled randomness (75% chance of using the common-word dictionary) to mimic natural spelling variation.

This expanded the `MARBERT`/`XLM-R` training set from **1,971** to **3,809** samples.

---

## ⚖️ Ensemble & Weight Tuning

After training the four backbones, a **weighted soft-voting ensemble** combines their probability outputs, weighted per model, and thresholded per aspect class.

### Weights (from a grid search over 625 combinations on the validation set):

| Model | Weight |
|---|---|
| MARBERT | 0.5 |
| AraBERT | 1.5 |
| CAMeLBERT | 1.2 |
| XLM-R | 0.5 |

> The fixed weights actually used for final test-set inference were: `{'marbert': 1.2, 'arabert': 1.2, 'camelbert': 1.0, 'xlmr': 0.7}` — see [`ensemble_weights.json`](./ensemble_weights.json).

### Per-aspect decision thresholds

Saved in [`thresholds.npy`](./thresholds.npy):

| Aspect | Threshold |
|---|---|
| food | 0.50 |
| service | 0.46 |
| price | 0.36 |
| cleanliness | 0.46 |
| delivery | 0.32 |
| ambiance | 0.46 |
| app_experience | 0.36 |
| general | 0.46 |
| none | 0.52 |

---

## 📏 Post-Processing Rules

1. **Empty / rating-only reviews** (`empty_rating_only`) skip the models entirely and are classified directly from the star rating:
   - ⭐ ≥ 4 → `general: positive`
   - ⭐ ≤ 2 → `general: negative`
   - ⭐ = 3 → `none: neutral`
2. **Max 6 aspects** per review (kept by highest predicted probability).
3. If `none` co-occurs with other aspects, `none` is dropped (a specific aspect and "no aspect" together are contradictory).
4. If no aspect crosses its threshold, `none: neutral` is used as a default.
5. If a sentiment prediction is unavailable for a given aspect, the star rating is used as a fallback.

---

## 📊 Results (Validation Set — 1,971 reviews)

### Main metric: Tuple F1 (aspect + sentiment must both match)

| Metric | Value |
|---|---|
| **Tuple F1** | **0.9056** (90.56%) |
| Precision | 0.9062 |
| Recall | 0.9049 |
| Aspect F1 (aspect detection only) | 0.9860 |
| Sentiment accuracy (given correct aspect) | 0.9184 |
| Review exact match (all tuples correct per review) | 0.8772 |

### Performance by route

| Route | # Samples | F1 |
|---|---|---|
| Arabic | 1,814 | 0.9182 |
| Latin | 4 | 1.0000 |
| Mixed | 24 | 0.8667 |
| Empty / rating-only | 129 | 0.6124 |

### Performance by aspect

| Aspect | F1 |
|---|---|
| food | 0.9989 |
| service | 0.9980 |
| cleanliness | 0.9946 |
| delivery | 0.9938 |
| price | 0.9929 |
| ambiance | 0.9973 |
| app_experience | 0.9956 |
| general | 0.9367 |
| none | 0.6796 |

### Sentiment confusion matrix (when the aspect is correctly detected)

| Gold \ Pred | negative | neutral | positive |
|---|---|---|---|
| **negative** | 1405 | 84 | 38 |
| **neutral** | 10 | 97 | 17 |
| **positive** | 43 | 76 | 1514 |

---

## 📁 Repository Contents

```
DeepX-AI-Hackathon-ABSA/
├── models/                         # Weights for all 4 backbones (Stage 1 + Stage 2)
├── ensemble_weights.json           # Final ensemble weights
├── thresholds.npy                  # Per-aspect decision thresholds (9 values)
├── submission.json                 # Predictions on the unlabeled set
├── submission_test.json            # Predictions on the hidden test set
├── __huggingface_repos__.json      # Repository metadata
└── README.md                       # This file
```

---

## 🚀 Usage

> ⚠️ This is **not** a single model loadable with `AutoModel` and a standard `pipeline()`. It is an **ensemble of 4 backbones plus custom language-routing, preprocessing, and post-processing logic**. Running inference requires the full inference code (from the original training notebook), not just the saved weights.

### Inference outline:

```python
from transformers import AutoTokenizer, AutoModel
import torch, json, numpy as np

BACKBONES = {
    'marbert':   'UBC-NLP/MARBERTv2',
    'arabert':   'aubmindlab/bert-base-arabertv02-twitter',
    'camelbert': 'CAMeL-Lab/bert-base-arabic-camelbert-da',
    'xlmr':      'FacebookAI/xlm-roberta-base',
}

# 1. Load the Stage-1 (aspect) and Stage-2 (sentiment) checkpoints from models/
# 2. Route each text via detect_language()
# 3. Preprocess it according to its route via preprocess_by_route()
# 4. Run the backbones listed in ROUTE_BACKBONES[route]
# 5. Combine outputs with the weights in ensemble_weights.json
# 6. Apply the per-aspect thresholds in thresholds.npy
# 7. Apply the post-processing rules to build the final prediction
```

For the complete code (model definitions, helper functions, preprocessing, and ensembling logic), see the original training notebook shipped alongside this project.

---

## 🎯 Intended Use

- Multi-dialect customer review analysis (Google Maps / Play Store / food-delivery platforms).
- Extracting per-aspect strengths and weaknesses (food, service, price, cleanliness, …) for business owners.
- Sentiment dashboards for restaurants, hotels, delivery apps, clinics, and e-commerce.

## ⚠️ Limitations

- Performance on very short / empty reviews (`empty_rating_only`) is comparatively weaker (F1 = 0.61) since it relies purely on a star-rating rule rather than the model.
- The `none` aspect has weaker performance (F1 = 0.68), reflecting the difficulty of distinguishing "no clear aspect" from a generic "general" comment.
- Trained on only 1,971 labeled reviews — a relatively small dataset, which may limit generalization to domains not well represented in training (e.g. medical or real-estate reviews).
- Franco-Arabic training examples are synthetically generated rather than fully authentic, which may reduce accuracy on unusual real-world Franco-Arabic spelling patterns.

## 🏆 Context

This model was developed as part of the **DeepX AI Hackathon**, addressing an Aspect-Based Sentiment Analysis (ABSA) task on multilingual, multi-dialect Arabic reviews.

## 📄 License

MIT

---

*For questions about this model, please open a Discussion on the Hugging Face repository page.*