Ysfxjo/marbert-saudi-complaint-action

marbert-saudi-complaint-topic

Fine-tuned UBC-NLP/MARBERTv2 for 4-class topic routing on a gold-standard Arabic complaint subset from GLARE.
(Any auto line saying “None dataset” from Trainer is superseded by this description.)

Evaluation set (held-out):

• Loss: 0.1378
• Accuracy: 0.9905
• Precision: 0.9905
• Recall: 0.9905
• F1: 0.9905

Model description

• Task: Topic of complaint: POLICY_SECURITY (0), FINANCIAL (1), TECHNICAL (2), CONTENT (3).
• Base model: MARBERTv2; cite Abdul-Mageed et al. (ACL 2020).
• Companion paper & code: GitHub YOUSEF-ysfxjo/complaint-xai-fl-research.

Intended uses & limitations

Uses: Topic triage for Arabic e-commerce / app-review complaints (policy vs payment vs technical vs content).

Limitations: Class boundaries can blur (e.g. financial vs technical); max length 128 in training; Gulf-heavy style—other dialects may degrade.

Training and evaluation data

• Source: GLARE-derived gold topic task — 10,000 samples per class (40,000 total), balanced. Details in companion repository.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 300
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1
0.1155	1.0	1125	0.1520	0.9752	0.9753	0.9752	0.9752
0.1433	2.0	2250	0.1883	0.9848	0.9848	0.9848	0.9848
0.086	3.0	3375	0.1708	0.9872	0.9873	0.9872	0.9872
0.0468	4.0	4500	0.1367	0.9902	0.9903	0.9902	0.9902
0.0105	5.0	5625	0.1378	0.9905	0.9905	0.9905	0.9905

Framework versions

• Transformers 4.53.3
• Pytorch 2.6.0+cu124
• Datasets 4.4.1
• Tokenizers 0.21.2

Inference example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "Ysfxjo/marbert-saudi-complaint-topic"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
text = "السحب من المحفظة لا يعمل من أمس"
inputs = tok(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    pred = model(**inputs).logits.argmax(-1).item()
print(model.config.id2label[pred])