README.md

---
base_model:
- Qwen/Qwen3.5-4B
datasets:
- Jiaha0Hu4ng/EmoWorld-130K
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: video-text-to-text
---

# Model Card

<!-- Provide a quick summary of what the model is/does. -->
<p align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/66953e2f417bbfcd51933cd0/0XVrXOZhxIMCYT_wecN_a.png" width="280"/>
</p>
OneEmo-Base is a unified multimodal reasoning model for emotion perception, understanding, and interaction in video scenarios. It simultaneously supports eight tasks, ranging from sentiment analysis and emotion/intention recognition to empathetic response generation and emotional support conversation within a single frameworks.
<!-- Provide a longer summary of what this model is. -->

## Overview
This is a SFT stage 1 training version of OneEmo which using Curriculum SFT strategy.
### Support task

- **MER** – Multimodal emotion recognition (single label)
- **MSA** – Multimodal sentiment analysis (positive/neutral/negative)
- **OVMER** – Open-vocabulary multimodal emotion recognition
- **MIR** – Multimodal intention recognition (requires candidate labels)
- **MHD** – Multimodal humour understanding (yes/no)
- **MSD** – Multimodal sarcasm understanding (yes/no)
- **ERG** – Empathetic response generation
- **ESC** – Emotional support conversation

### Software

The model was trained and evaluated in the following environment:

- **Python libraries:**
  - `torch`: 2.10.0+cu128
  - `torchaudio`: 2.10.0+cu128
  - `torchvision`: 0.25.0+cu128
  - `transformers`: 5.2.0
  - `vllm`: 0.19.0 
  - `ms-swift`: 4.1.3 (optional, for alternative inference engine)
- **Environment management:** Conda (environment name: `oneemo`)

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** [https://github.com/waHAHJIAHAO/OneEmo](https://github.com/waHAHJIAHAO/OneEmo)
- **Paper:** [OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction](https://huggingface.co/papers/2608.06013)

## How to Get Started with the Model

Use the code below to get started with the model.

### VLLM
```bash
vllm serve Jiaha0Hu4ng/OneEmo-Base \
    --host 0.0.0.0 \
    --port 8000 \
    --served-model-name OneEmo-Base \
    --max-model-len 32768 \
    --mm-encoder-tp-mode data \
    --media-io-kwargs '{"video": {"num_frames": 16}}' \
    --default-chat-template-kwargs '{"enable_thinking": true}' \
    --reasoning-parser qwen3
```

### Swift
```python
import torch
from swift import get_model_processor, get_template
from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig

model_path = "Jiaha0Hu4ng/OneEmo-Base"

model, processor = get_model_processor(
    model_path,
    model_type="qwen3_5",
    torch_dtype=torch.bfloat16,
    attn_impl="flash_attn",
)
template = get_template(processor, enable_thinking=True)
engine = TransformersEngine(model, template=template)
request_config = RequestConfig(max_tokens=2048, temperature=0.7)

infer_request = InferRequest(
    messages=[{
        "role": "user",
        "content": (
            "Based on the video, describe the range of emotions the character "
            "may be feeling."
        )
    }],
    videos=["path/to/your/video.mp4"]
)
resp_list = engine.infer([infer_request], request_config=request_config)
print(resp_list[0].choices[0].message.content)
```

### Transformers

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
from qwen_vl_utils import process_vision_info

model_path = "Jiaha0Hu4ng/OneEmo-Base"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # fallback to "sdpa" if flash-attn is not installed
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_path)

# Example: open-vocabulary multimodal emotion recognition (OVMER)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "path/to/your/video.mp4",
                "max_pixels": 360 * 420,
                "fps": 2.0,
            },
            {
                "type": "text",
                "text": "Based on the video, describe the range of emotions the character may be feeling.",
            },
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=2048,
        temperature=0.7,
        top_p=0.9,
        top_k=50,
    )
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])
```

For other tasks, replace the prompt text with the corresponding template (see the full model card on Hugging Face for the complete prompt table). The model also supports deployment with **vLLM** (OpenAI-compatible API) and inference via **ms-swift**; refer to the repository for those examples.

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

We use Emo-World-130K to train OneEmo.


## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[To Be continued]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[To Be continued]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[To Be continued]

### Results

[To Be continued]

## Technical Specifications [optional]

[To Be continued]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

<!-- **BibTeX:**

```bibtex
@misc{oneemo2026,
  title={OneEmo: A Unified Multimodal Model for Emotion, Intention, and Mental-Health Understanding in Videos},
  author={Huang, Jiahao and others},
  year={2026},
  howpublished={\url{https://huggingface.co/Jiaha0Hu4ng/OneEmo}}
}
``` -->
[To Be continued]

<!-- **APA:**

Huang, J. et al. (2026). *OneEmo: A Unified Multimodal Model for Emotion, Intention, and Mental-Health Understanding in Videos*. Hugging Face. https://huggingface.co/Jiaha0Hu4ng/OneEmo
 -->

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

OneEmo can be used directly for video-based emotion analysis, intention recognition, sentiment classification, humour/sarcasm detection, and for generating empathetic or supportive textual responses. It is suitable for researchers and developers building affective computing applications, mental-health chatbots, or social signal processing tools.

### Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

The model can be fine-tuned on domain-specific datasets (e.g., clinical interviews, customer service videos) or integrated into larger dialogue systems for emotionally aware conversation. When fine-tuning, users should follow the same prompt format and task prefix convention.

Note: The model is **not** intended for:
- Clinical diagnosis of mental disorders without professional oversight.
- Making high-stakes decisions (e.g., hiring, legal judgements) based on emotion predictions.
- Use in surveillance or manipulative applications.
It may underperform on languages or cultures not well represented in training data, and on videos longer than the maximum frame budget (16 frames by default).

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

- The model inherits biases from its pre-training and fine-tuning data; emotion labels may reflect cultural stereotypes.
- Generated empathetic or counselling responses can occasionally be unsafe, inappropriate, or not clinically sound.
- Performance degrades with noisy audio transcripts, low-resolution video, or out-of-domain scenarios.
- The chain-of-thought reasoning is not guaranteed to be factually correct or logically consistent.

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. We recommend:
- Not treating model outputs as professional mental-health advice.
- Employing human review when used in sensitive contexts.
- Evaluating the model on representative data before deployment.
- Considering additional fairness and safety guardrails, especially for vulnerable populations.