Initial commit

README.md

@@ -0,0 +1,137 @@
+---
+library_name: pytorch
+tags:
+- chimera-ml
+- lefsa
+- pytorch
+- audio
+- video
+- text
+- multimodal
+- emotion-recognition
+- sentiment-analysis
+- affective-computing
+- emoaffectnet
+- wav2vec2
+- whisper
+- jina-embeddings
+datasets:
+- RAMAS
+- MELD
+- CMU-MOSEI
+base_model:
+- openai/whisper-base
+- FacebookAI/xlm-roberta-base
+- jinaai/jina-embeddings-v3
+---
+
+# LEFSA Models
+
+This repository contains LEFSA model weights for multimodal affective state recognition.
+LEFSA stands for **Label Encoder Fusion Strategy with Averaging** and is designed for joint **emotion recognition** and **sentiment recognition** from audio, video, and text modalities.
+
+## Files
+
+- `lefsa.pt` — LEFSA checkpoint for joint audio-video-text emotion and sentiment recognition.
+- `emoaffectnet.pt` — EmoAffectNet model for visual feature extraction. The original EmoAffectNet repository is available here: https://github.com/ElenaRyumina/EMO-AffectNetModel.
+
+## What the Model Predicts
+
+The model has two classification heads.
+
+| Task | Number of classes | Class order |
+|---|---:|---|
+| Emotion recognition | 7 | `neutral`, `happy`, `sad`, `anger`, `surprise`, `disgust`, `fear` |
+| Sentiment recognition | 3 | `negative`, `neutral`, `positive` |
+
+Use this exact class order when converting logits or probabilities to labels.
+
+## Model Overview
+- Acoustic, visual, and linguistic features are downsampled to a common temporal representation.
+- The model applies cross-modal transformer blocks to model interactions between modalities.
+- A label encoder produces unimodal emotion and sentiment predictions and injects this label-level context back into the fusion module.
+- In LEFSA, unimodal predictions are additionally averaged with multimodal predictions to improve robustness.
+
+## Research Corpora
+
+The model family was evaluated in a multilingual and multicorpus setting.
+
+| Corpus | Language / domain | Modalities | Tasks |
+|---|---|---|---|
+| RAMAS | Russian, dyadic semi-spontaneous interactions | Audio, video, text | Emotion, sentiment |
+| MELD | English, scripted TV-series dialogues | Audio, video, text | Emotion, sentiment |
+| CMU-MOSEI | English, in-the-wild YouTube monologues | Audio, video, text | Emotion, sentiment |
+
+Emotion labels are mapped to seven classes: neutral, happiness, sadness, anger, surprise, disgust, and fear.
+Sentiment labels are mapped to three classes: negative, neutral, and positive.
+
+### Feature and Fusion Strategy Comparison
+
+`MF` is macro F1-score. For CMU-MOSEI emotion recognition, `mMF` is the mean macro F1-score over binary positive/negative emotion classes. The average value is the integral score reported in the paper.
+
+| ID | Audio | Video | Text | Fusion | RAMAS Emotion MF7 | RAMAS Sentiment MF3 | MELD Emotion MF7 | MELD Sentiment MF3 | CMU-MOSEI Emotion mMF6 | CMU-MOSEI Sentiment MF3 | Average |
+|---:|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|
+| 1 | Wav2Vec2 | EmoAffectNet | JINA | BFS | 60.57 | 65.02 | 38.56 | 65.94 | 62.46 | 62.56 | 59.18 |
+| 2 | ExHuBERT | EmoAffectNet | JINA | BFS | 62.35 | 64.02 | 38.40 | 63.93 | 62.10 | 62.32 | 58.85 |
+| 3 | Wav2Vec2 | EmoAffectNet | RoBERTa | BFS | 57.64 | 67.14 | 35.48 | 65.03 | 61.58 | 60.19 | 57.84 |
+| 4 | ExHuBERT | EmoAffectNet | RoBERTa | BFS | 60.54 | 64.70 | 35.05 | 63.78 | 60.78 | 59.25 | 57.35 |
+| 5 | Wav2Vec2 | ResEmoteNet | JINA | BFS | 55.88 | 59.93 | 37.75 | 63.81 | 62.22 | 63.69 | 57.21 |
+| 6 | ExHuBERT | ResEmoteNet | JINA | BFS | 57.21 | 61.40 | 38.25 | 64.06 | 61.11 | 60.81 | 57.14 |
+| 7 | ExHuBERT | ResEmoteNet | RoBERTa | BFS | 49.62 | 54.44 | 32.56 | 62.15 | 59.88 | 60.47 | 53.19 |
+| 8 | Wav2Vec2 | ResEmoteNet | RoBERTa | BFS | 52.29 | 54.87 | 34.08 | 61.65 | 58.79 | 57.42 | 53.18 |
+| 9 | Wav2Vec2 | EmoAffectNet | JINA | LEFS | 61.38 | 66.57 | 39.79 | 65.70 | 62.79 | 61.59 | 59.64 |
+| 10 | Wav2Vec2 | EmoAffectNet | JINA | LEFSA | **62.52** | 64.96 | **40.09** | **67.02** | 62.30 | 62.00 | **59.81** |
+
+The best LEFSA configuration uses **Wav2Vec2 + EmoAffectNet + JINA** features with Label Encoder Fusion Strategy with Averaging.
+
+### Comparison with Prior Multimodal Approaches
+
+`ST` means single-task recognition and `MT` means multitask recognition.
+
+| Approach | Corpus | Setup | Emotion A7 | Emotion WF7 | Sentiment A3 | Sentiment WF3 |
+|---|---|---|---:|---:|---:|---:|
+| Ours | RAMAS | MT | 68.99 | 67.79 | 84.11 | 84.02 |
+| Zhang et al. | MELD | MT | 41.17 | 41.22 | 67.33 | 67.21 |
+| Van et al. | MELD | ST | 66.28 | 65.69 | — | — |
+| Hwang et al. | MELD | ST | 66.70 | 65.93 | — | — |
+| Tu et al. | MELD | ST | 67.85 | 67.02 | — | — |
+| Ours | MELD | MT | 62.30 | 59.79 | 69.20 | 69.02 |
+
+| Approach | Corpus | Setup | Emotion mWA6 | Emotion mWF6 | Sentiment A2 | Sentiment WF2 |
+|---|---|---|---:|---:|---:|---:|
+| Chauhan et al. | CMU-MOSEI | MT | 62.97 | 79.02 | 80.37 | 78.23 |
+| Sangwan et al. | CMU-MOSEI | MT | 63.16 | 79.06 | 80.15 | 78.30 |
+| Hwang et al. | CMU-MOSEI | ST | — | — | 87.40 | 87.30 |
+| Zheng et al. | CMU-MOSEI | ST | �� | — | 85.90 | 86.00 |
+| Ours | CMU-MOSEI | MT | 64.78 | 79.06 | 84.83 | 84.90 |
+
+## Related Publications
+
+Markitantov M., Ryumina E., Kaya H., Karpov A. Multi-Modal Multi-Task Affective States Recognition Based on Label Encoder Fusion // In Proc. Interspeech 2025, pp. 3010–3014. https://doi.org/10.21437/Interspeech.2025-2060
+
+```bibtex
+@inproceedings{markitantov25_interspeech,
+  title     = {{Multi-Modal Multi-Task Affective States Recognition Based on Label Encoder Fusion}},
+  author    = {Maxim Markitantov and Elena Ryumina and Heysem Kaya and Alexey Karpov},
+  year      = {2025},
+  booktitle = {{Interspeech 2025}},
+  pages     = {3010--3014},
+  doi       = {10.21437/Interspeech.2025-2060},
+  issn      = {2958-1796}
+}
+```
+
+Markitantov M., Ryumina E., Dvoynikova A., Karpov A. Multi-lingual approach for multi-modal emotion and sentiment recognition based on triple fusion // Information Fusion, 2026, vol. 132, article 104207. https://doi.org/10.1016/j.inffus.2026.104207
+
+```bibtex
+@article{markitantov2026triplefusion,
+  title   = {Multi-lingual approach for multi-modal emotion and sentiment recognition based on triple fusion},
+  author  = {Markitantov, Maxim and Ryumina, Elena and Dvoynikova, Anastasia and Karpov, Alexey},
+  journal = {Information Fusion},
+  volume  = {132},
+  pages   = {104207},
+  year    = {2026},
+  doi     = {10.1016/j.inffus.2026.104207},
+  url     = {https://doi.org/10.1016/j.inffus.2026.104207}
+}
+```

emoaffectnet.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8274190b5be4355bd2f07b59f593fcdb294f9d7c563bfa9ac9e5ea06c10692d2
+size 98562934

lefsa.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:247acfa8fb55c528917d6535428c64d8769e2e1ba396f8499112f94535271f2b
+size 28791882