UEmo-D1: An Efficient and Calibrated Decision Model for Unified Multimodal Emotion Recognition
Abstract
A unified decision-model framework for multimodal emotion recognition maps instruction-guided multimodal inputs to probability distributions over candidate options, yielding strong recognition, low inference latency, and calibrated confidence across multiple tasks.
Multimodal Emotion Recognition (MER) seeks to identify emotions conveyed through or elicited by media comprising diverse modality compositions. Research in this field has largely evolved along two paradigms: task-specific models and approaches based on Multimodal Large Language Models (MLLMs). Despite their distinct strengths, neither fully reconciles three capabilities essential to human-like emotion understanding: cross-task generalization, efficient decision-making, and receptiveness to multiple plausible interpretations. Inspired by recent advances in decision models, we explore a unified framework that maps multimodal inputs, guided by natural language instructions, directly to distributions over candidate options, addressing these requirements through direct probabilistic decisions. We instantiate this framework in UEmo-D1, the first decision model for MER to our knowledge. To support joint training and systematic evaluation, we also curate UEmo-Data, comprising 977,270 training samples and 41 evaluation subsets across six MER tasks. Extensive experiments demonstrate UEmo-D1's combination of strong recognition performance, low inference latency, and well-calibrated confidence estimates. Complementary analyses further reveal how training strategies and data composition influence these capabilities, providing practical insights into unified MER modeling. Together, our findings establish decision models as a feasible direction toward human-level emotion recognition. Project page: https://github.com/wdqqdw/UEmo-D1.
Get this paper in your agent:
hf papers read 2611.24481 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 1
wudq/UEmo-Data-v1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper