Model Card for albert-base-v2-persian-sentiment-binary

A binary Persian sentiment classifier (negative / positive), fine-tuned from shekar-ai/albert-base-v2-persian-zwnj-naab-mlm on a combination of DeepSentiPers (binary-collapsed) and the Snappfood Persian sentiment dataset.

Model Details

Model Description

This model classifies Persian-language text into two sentiment classes: negative and positive. It was built by attaching a sequence classification head to a pretrained ALBERT-base Persian MLM checkpoint and fine-tuning it on a combined dataset drawn from two sources:

• DeepSentiPers (Digikala digital product reviews): original 5-point polarity labels (-2 to +2) reduced to binary by mapping -2/-1 to negative, +1/+2 to positive, and dropping neutral (0) rows entirely.
• Snappfood (food delivery reviews): used as-is, already labeled HAPPY/SAD, mapped directly to positive/negative.

This model card has been automatically generated.

• Developed by: MahdiAlikhah
• Model type: ALBERT-base for sequence classification, binary Persian sentiment analysis
• Language(s) (NLP): Persian (fa)
• License: apache-2.0 (inherited from base model — confirm against shekar-ai/albert-base-v2-persian-zwnj-naab-mlm's own license before relying on this)
• Finetuned from model: shekar-ai/albert-base-v2-persian-zwnj-naab-mlm

Model Sources

• Repository: https://huggingface.co/MahdiAlikhah/albert-base-v2-persian-sentiment-binary

Uses

Direct Use

Sentence- or short-passage-level binary sentiment classification of Persian text (negative / positive), particularly for text similar to its training domains: e-commerce product reviews and food delivery reviews.

Out-of-Scope Use

Not evaluated on domains far from product/food-delivery reviews (e.g. social media, news, formal writing); not designed for languages other than Persian or mixed-language text; not suitable as a sole signal for high-stakes decisions without human review. Also not designed to detect neutral or mixed sentiment — the training data excludes neutral examples, so genuinely neutral or ambivalent input will still be forced into one of the two classes.

Bias, Risks, and Limitations

The two source datasets differ substantially in size (Snappfood ~52k training rows vs. DeepSentiPers ~2.5k), so the model's behavior is disproportionately shaped by Snappfood's food-delivery domain and review style. Evaluated separately, the model reached 87.27% accuracy / 0.873 macro-F1 on the Snappfood test set (9,033 examples) and 93.99% accuracy / 0.890 macro-F1 on the DeepSentiPers test set — however the DeepSentiPers test set is small (316 examples, only 54 negative), so that number carries more sampling variance and should be treated as a rough estimate rather than a precise figure. On Snappfood, the model shows a mild bias toward under-predicting positive sentiment (more false negatives than false positives), consistent with mixed or lukewarm reviews being genuinely ambiguous under a strict binary label.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Validate performance on your own domain/data before deploying, especially outside product-review or food-delivery-review Persian text, and treat the DeepSentiPers-specific accuracy figure as approximate given its small test set.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "MahdiAlikhah/albert-base-v2-persian-sentiment-binary"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "این محصول عالی بود"
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
pred = logits.argmax(-1).item()
print(model.config.id2label[pred])

Training Details

Training Data

Combined training set built from:

• DeepSentiPers (https://github.com/JoyeBright/DeepSentiPers), original.csv split, neutral (0) rows dropped, remaining labels mapped to negative (-2/-1) and positive (+1/+2). Stratified 80/10/10 train/val/test split.
• Snappfood (PNLPhub/snappfood-sentiment-analysis on the Hugging Face Hub), used with its own provided train/validation/test splits, labels mapped from HAPPY/SAD strings to positive/negative.

Both sources were combined for training and validation. Test evaluation was kept separate per source (see Results) rather than merged, since the two domains differ meaningfully. Training rows overlapping verbatim with either test set were removed prior to training.

Training Procedure

Preprocessing

Text normalized with hazm's Normalizer (Persian half-space/ZWNJ normalization), then tokenized with the base model's tokenizer (max_length=128, truncation and padding to max length).

Training Hyperparameters

• Training regime: fp16 mixed precision
• Epochs: up to 5, with early stopping (patience 2) on validation macro-F1; best checkpoint was epoch 2
• Batch size: 16 (train), 32 (eval)
• Learning rate: 2e-5
• Weight decay: 0.01
• Loss: class-weighted cross-entropy (balanced class weights) to counteract residual class imbalance in the combined training set

Speeds, Sizes, Times

Train runtime: approximately 34 minutes (2,036s) for 4 epochs (stopped early) on a single T4 GPU (Google Colab).

Evaluation

Testing Data, Factors & Metrics

Testing Data

Two held-out test sets, evaluated separately:

• DeepSentiPers test split (316 examples: 54 negative, 262 positive)
• Snappfood test split (9,033 examples: 4,547 negative, 4,486 positive)

Neither test set was used in training or validation.

Metrics

Accuracy, and macro F1 (chosen as the primary metric because of class imbalance, particularly in the small DeepSentiPers negative class).

Results

DeepSentiPers test set: accuracy 0.940, macro F1 0.890. Negative class: precision 0.86, recall 0.78, F1 0.82 (support 54). Positive class: precision 0.96, recall 0.97, F1 0.96 (support 262).

Snappfood test set: accuracy 0.873, macro F1 0.873. Negative class: precision 0.85, recall 0.91, F1 0.88 (support 4,547). Positive class: precision 0.90, recall 0.84, F1 0.87 (support 4,486).

Summary

The model performs solidly on both domains, with Snappfood being the more statistically reliable estimate given its much larger test set. On Snappfood, the model shows a mild bias toward missing positive sentiment (more false negatives than false positives), which is consistent with mixed or lukewarm reviews being inherently hard to resolve under a strict binary label. For context, a dedicated ParsBERT model trained only on a clean, balanced 70k-row Snappfood set reports 87.98% F1 — this model's 87.27% F1 on Snappfood is close to that despite being a smaller architecture trained on mixed-domain data.

Technical Specifications

Model Architecture and Objective

ALBERT-base architecture with a sequence classification head (linear layer over the pooled [CLS] output), binary softmax output, fine-tuned with class-weighted cross-entropy loss.

Compute Infrastructure

Hardware

Single NVIDIA T4 GPU (Google Colab)