yeshpanovrustem/rembert-sentiment-analysis-polarity-classification

Polarity Classification Model for Russian, Kazakh, and Code-Switched Movie Reviews

This is a RemBERT model fine-tuned for sentiment analysis on movie reviews in Russian and/or Kazakh. It predicts the polarity of a review as positive, negative, or neutral. The model was fine-tuned on 100,000+ Movie Reviews from Kazakhstan.

Results on the test set

Model	Accuracy	Precision	Recall	F1	Kappa
RemBERT	0.95	0.84	0.81	0.82	0.88

How to use

You can use this model with the Transformers pipeline for text classification.

from transformers import AutoModelForSequenceClassification
from transformers import AutoTokenizer
from transformers import TextClassificationPipeline

model = AutoModelForSequenceClassification.from_pretrained("yeshpanovrustem/rembert-sentiment-analysis-polarity-classification")
tokenizer = AutoTokenizer.from_pretrained("yeshpanovrustem/rembert-sentiment-analysis-polarity-classification")

pipe = TextClassificationPipeline(model = model, tokenizer = tokenizer)

reviews = ["Довольно интересное кино.", "Бұл фильмнің мағынасын түсінбедім."]

for review in reviews:
    print(pipe(review))

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 20
• eval_batch_size: 20
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 2
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1 Macro	Kappa
0.1789	1.0	4001	0.2144	0.9340	0.7629	0.8529
0.1307	2.0	8002	0.1944	0.9414	0.8051	0.8699

Framework versions

• Transformers 5.2.0
• Pytorch 2.10.0+cu126
• Datasets 4.6.0
• Tokenizers 0.22.2