| --- |
| library_name: pytorch |
| tags: |
| - chimera-ml |
| - lefsa |
| - pytorch |
| - audio |
| - video |
| - text |
| - multimodal |
| - emotion-recognition |
| - sentiment-analysis |
| - affective-computing |
| - emoaffectnet |
| - wav2vec2 |
| - whisper |
| - jina-embeddings |
| datasets: |
| - RAMAS |
| - MELD |
| - CMU-MOSEI |
| --- |
| |
| # LEFSA Models |
|
|
| This repository contains LEFSA model weights for multimodal affective state recognition. |
| LEFSA stands for **Label Encoder Fusion Strategy with Averaging** and is designed for joint **emotion recognition** and **sentiment recognition** from audio, video, and text modalities. |
|
|
| ## Files |
|
|
| - `lefsa.pt` — LEFSA checkpoint for joint audio-video-text emotion and sentiment recognition. |
| - `emoaffectnet.pt` — EmoAffectNet model for visual feature extraction. The original EmoAffectNet repository is available here: https://github.com/ElenaRyumina/EMO-AffectNetModel. |
|
|
| ## What the Model Predicts |
|
|
| The model has two classification heads. |
|
|
| | Task | Number of classes | Class order | |
| |---|---:|---| |
| | Emotion recognition | 7 | `neutral`, `happy`, `sad`, `anger`, `surprise`, `disgust`, `fear` | |
| | Sentiment recognition | 3 | `negative`, `neutral`, `positive` | |
|
|
| Use this exact class order when converting logits or probabilities to labels. |
|
|
| ## Model Overview |
| - Acoustic, visual, and linguistic features are downsampled to a common temporal representation. |
| - The model applies cross-modal transformer blocks to model interactions between modalities. |
| - A label encoder produces unimodal emotion and sentiment predictions and injects this label-level context back into the fusion module. |
| - In LEFSA, unimodal predictions are additionally averaged with multimodal predictions to improve robustness. |
|
|
| ## Research Corpora |
|
|
| The model family was evaluated in a multilingual and multicorpus setting. |
|
|
| | Corpus | Language / domain | Modalities | Tasks | |
| |---|---|---|---| |
| | RAMAS | Russian, dyadic semi-spontaneous interactions | Audio, video, text | Emotion, sentiment | |
| | MELD | English, scripted TV-series dialogues | Audio, video, text | Emotion, sentiment | |
| | CMU-MOSEI | English, in-the-wild YouTube monologues | Audio, video, text | Emotion, sentiment | |
|
|
| Emotion labels are mapped to seven classes: neutral, happiness, sadness, anger, surprise, disgust, and fear. |
| Sentiment labels are mapped to three classes: negative, neutral, and positive. |
|
|
| ### Feature and Fusion Strategy Comparison |
|
|
| `MF` is macro F1-score. For CMU-MOSEI emotion recognition, `mMF` is the mean macro F1-score over binary positive/negative emotion classes. The average value is the integral score reported in the paper. |
|
|
| | ID | Audio | Video | Text | Fusion | RAMAS Emotion MF7 | RAMAS Sentiment MF3 | MELD Emotion MF7 | MELD Sentiment MF3 | CMU-MOSEI Emotion mMF6 | CMU-MOSEI Sentiment MF3 | Average | |
| |---:|---|---|---|---|---:|---:|---:|---:|---:|---:|---:| |
| | 1 | Wav2Vec2 | EmoAffectNet | JINA | BFS | 60.57 | 65.02 | 38.56 | 65.94 | 62.46 | 62.56 | 59.18 | |
| | 2 | ExHuBERT | EmoAffectNet | JINA | BFS | 62.35 | 64.02 | 38.40 | 63.93 | 62.10 | 62.32 | 58.85 | |
| | 3 | Wav2Vec2 | EmoAffectNet | RoBERTa | BFS | 57.64 | 67.14 | 35.48 | 65.03 | 61.58 | 60.19 | 57.84 | |
| | 4 | ExHuBERT | EmoAffectNet | RoBERTa | BFS | 60.54 | 64.70 | 35.05 | 63.78 | 60.78 | 59.25 | 57.35 | |
| | 5 | Wav2Vec2 | ResEmoteNet | JINA | BFS | 55.88 | 59.93 | 37.75 | 63.81 | 62.22 | 63.69 | 57.21 | |
| | 6 | ExHuBERT | ResEmoteNet | JINA | BFS | 57.21 | 61.40 | 38.25 | 64.06 | 61.11 | 60.81 | 57.14 | |
| | 7 | ExHuBERT | ResEmoteNet | RoBERTa | BFS | 49.62 | 54.44 | 32.56 | 62.15 | 59.88 | 60.47 | 53.19 | |
| | 8 | Wav2Vec2 | ResEmoteNet | RoBERTa | BFS | 52.29 | 54.87 | 34.08 | 61.65 | 58.79 | 57.42 | 53.18 | |
| | 9 | Wav2Vec2 | EmoAffectNet | JINA | LEFS | 61.38 | 66.57 | 39.79 | 65.70 | 62.79 | 61.59 | 59.64 | |
| | 10 | Wav2Vec2 | EmoAffectNet | JINA | LEFSA | **62.52** | 64.96 | **40.09** | **67.02** | 62.30 | 62.00 | **59.81** | |
|
|
| The best LEFSA configuration uses **Wav2Vec2 + EmoAffectNet + JINA** features with Label Encoder Fusion Strategy with Averaging. |
|
|
| ### Comparison with Prior Multimodal Approaches |
|
|
| `ST` means single-task recognition and `MT` means multitask recognition. |
|
|
| | Approach | Corpus | Setup | Emotion A7 | Emotion WF7 | Sentiment A3 | Sentiment WF3 | |
| |---|---|---|---:|---:|---:|---:| |
| | Ours | RAMAS | MT | 68.99 | 67.79 | 84.11 | 84.02 | |
| | Zhang et al. | MELD | MT | 41.17 | 41.22 | 67.33 | 67.21 | |
| | Van et al. | MELD | ST | 66.28 | 65.69 | — | — | |
| | Hwang et al. | MELD | ST | 66.70 | 65.93 | — | — | |
| | Tu et al. | MELD | ST | 67.85 | 67.02 | — | — | |
| | Ours | MELD | MT | 62.30 | 59.79 | 69.20 | 69.02 | |
|
|
| | Approach | Corpus | Setup | Emotion mWA6 | Emotion mWF6 | Sentiment A2 | Sentiment WF2 | |
| |---|---|---|---:|---:|---:|---:| |
| | Chauhan et al. | CMU-MOSEI | MT | 62.97 | 79.02 | 80.37 | 78.23 | |
| | Sangwan et al. | CMU-MOSEI | MT | 63.16 | 79.06 | 80.15 | 78.30 | |
| | Hwang et al. | CMU-MOSEI | ST | — | — | 87.40 | 87.30 | |
| | Zheng et al. | CMU-MOSEI | ST | — | — | 85.90 | 86.00 | |
| | Ours | CMU-MOSEI | MT | 64.78 | 79.06 | 84.83 | 84.90 | |
|
|
| ## Related Publications |
|
|
| Markitantov M., Ryumina E., Kaya H., Karpov A. Multi-Modal Multi-Task Affective States Recognition Based on Label Encoder Fusion // In Proc. Interspeech 2025, pp. 3010–3014. https://doi.org/10.21437/Interspeech.2025-2060 |
|
|
| ```bibtex |
| @inproceedings{markitantov25_interspeech, |
| title = {{Multi-Modal Multi-Task Affective States Recognition Based on Label Encoder Fusion}}, |
| author = {Maxim Markitantov and Elena Ryumina and Heysem Kaya and Alexey Karpov}, |
| year = {2025}, |
| booktitle = {{Interspeech 2025}}, |
| pages = {3010--3014}, |
| doi = {10.21437/Interspeech.2025-2060}, |
| issn = {2958-1796} |
| } |
| ``` |
|
|
| Markitantov M., Ryumina E., Dvoynikova A., Karpov A. Multi-lingual approach for multi-modal emotion and sentiment recognition based on triple fusion // Information Fusion, 2026, vol. 132, article 104207. https://doi.org/10.1016/j.inffus.2026.104207 |
|
|
| ```bibtex |
| @article{markitantov2026triplefusion, |
| title = {Multi-lingual approach for multi-modal emotion and sentiment recognition based on triple fusion}, |
| author = {Markitantov, Maxim and Ryumina, Elena and Dvoynikova, Anastasia and Karpov, Alexey}, |
| journal = {Information Fusion}, |
| volume = {132}, |
| pages = {104207}, |
| year = {2026}, |
| doi = {10.1016/j.inffus.2026.104207}, |
| url = {https://doi.org/10.1016/j.inffus.2026.104207} |
| } |
| ``` |
