rubert-tiny-emotion-russian-cedr-m7

Emotion detection in Russian text over seven classes, fine-tuned from cointegrated/rubert-tiny on Aniemore/cedr-m7 (12M parameters).

The head is multi-label: seven independent sigmoids, not a softmax over seven classes, so a sentence may carry more than one emotion or none at all. That is why the headline number below is ROC AUC, which needs no threshold.

Results on CEDR-M7 test

Metric	Value
ROC AUC, macro	0.9008
ROC AUC, micro	0.9532
ROC AUC, weighted	0.9367
Weighted accuracy (top-1)	0.7965
Unweighted accuracy (macro recall)	0.6121
macro-F1 @ 0.5	0.6036
micro-F1 @ 0.5	0.7955
Subset accuracy (exact set)	0.7625

How to read these

ROC AUC is threshold-free: it asks whether the model ranks the sentences carrying an emotion above the ones that do not. macro-F1 is taken at a fixed threshold of 0.5; a different operating point moves it, so tune the threshold on your own data before reading it as a ceiling. Weighted accuracy is top-1: the highest-scoring label counted against the gold set, which is a fair reading here because 1865 of the 1882 test rows carry exactly one label.

disgust has 3 examples in the test split. Its per-class numbers are noise, and they drag both macro averages. The macro figures are reported as measured rather than quietly dropping the class, but weigh them with that in mind.

Evaluated on the official test split, 1882 rows, sources: twitter, LiveJournal and Lenta. Scores come from the model's own id2label order — it differs across this family, and reading the labels in the wrong order silently mislabels everything.

First - you should prepare few functions to talk to model

import torch
from transformers import BertForSequenceClassification, AutoTokenizer

LABELS = ['anger', 'disgust', 'enthusiasm', 'fear', 'happiness', 'neutral', 'sadness']
tokenizer = AutoTokenizer.from_pretrained('Aniemore/rubert-tiny-emotion-russian-cedr-m7')
model = BertForSequenceClassification.from_pretrained('Aniemore/rubert-tiny-emotion-russian-cedr-m7')

@torch.no_grad()
def predict_emotion(text: str) -> str:
    """
        We take the input text, tokenize it, pass it through the model, and then return the predicted label
        :param text: The text to be classified
        :type text: str
        :return: The predicted emotion
    """
    inputs = tokenizer(text, max_length=512, padding=True, truncation=True, return_tensors='pt')
    outputs = model(**inputs)
    predicted = torch.nn.functional.softmax(outputs.logits, dim=1)
    predicted = torch.argmax(predicted, dim=1).numpy()

    return LABELS[predicted[0]]

@torch.no_grad()
def predict_emotions(text: str) -> list:
    """
        It takes a string of text, tokenizes it, feeds it to the model, and returns a dictionary of emotions and their
        probabilities
        :param text: The text you want to classify
        :type text: str
        :return: A dictionary of emotions and their probabilities.
    """
    inputs = tokenizer(text, max_length=512, padding=True, truncation=True, return_tensors='pt')
    outputs = model(**inputs)
    predicted = torch.nn.functional.softmax(outputs.logits, dim=1)
    emotions_list = {}
    for i in range(len(predicted.numpy()[0].tolist())):
        emotions_list[LABELS[i]] = predicted.numpy()[0].tolist()[i]
    return emotions_list

And then - just gently ask a model to predict your emotion

simple_prediction = predict_emotion("Какой же сегодня прекрасный день, братья")
not_simple_prediction = predict_emotions("Какой же сегодня прекрасный день, братья")

print(simple_prediction)
print(not_simple_prediction)

Or, just simply use our package (GitHub), that can do whatever you want (or maybe not)

🤗

Citations

@misc{Aniemore,
  author = {Артем Аментес, Илья Лубенец, Никита Давидчук},
  title = {Открытая библиотека искусственного интеллекта для анализа и выявления эмоциональных оттенков речи человека},
  year = {2022},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.com/aniemore/Aniemore}},
  email = {hello@socialcode.ru}
}